S.,
There are babies that have no problem with shorter nursing session. The best indicator is growth, as in "Is he growing" not what his chart ranking is. Also, is he having plenty of wet/dirty diapers? When you change wet diapers, does his pee seem to be pretty colorless? If so, you guys should be good to go.
As far as the nursing goes, it sounds like simple distraction. Both my daughters did this, but a bit later in their case. For about a month I had to try to nurse them without other distraction. We would go into a darkened room. Sometimes quiet worked better, sometimes white noise helped. A nursing necklace helped with my first - it helped her focus her attention on me instead of looking around. A brightly colored scarf might do the same thing. Also, playing games with her kept her attention on me. Talking, singing, gentle tickles, touching and naming body parts, making funny faces... anything that keeps his attention on you. Woe to the person who walked into the nursery when they were nursing!! Anyway, it doesn't last long. Never fear, he will grow out of it if it is simple distraction.
Other things it could be are teething. Sometimes babies don't want to nurse when their teeth hurt. He could have an ear infection, and it hurts for him to be laying on his side. I assume you haven't had any traumatic experiences while nursing lately, like shouting after he bit you? Sometimes this can put baby off nursing. It's definitely worth exploring the health aspects, just to make sure everything is alright. Hylands makes some fabulous homeopathic teething tablets. If he is in pain they might help. You can also give Tylenol and see if it helps. Maybe a half hour or so before nursing?
If he doesn't want to latch back on, just let him go. Make sure to offer nursing more often, though. You might have good luck offering the same breast the second time, to make sure that he is draining the breast fully. Also, the richest, fattiest milk is at the end of the feed, so you definitely want him to take that hindmilk. I would offer my daughter one breast for one to two hours, so no matter how often she nursed in that time period, it was from the same breast, then I switched off to the other. Now, if he seems cranky and unsatisfied when you offer the same breast, he may have drained it and be ready for the other. Play it by ear and you guys will figure it out!
If you keep having problems, you can get wonderful help from a local La Leche League leader. They have seen it all while working with nursing women. Their training far exceeds what most doctors receive, and they also all have the advantage of having nursed their own babies. I don't think I would have been able to make it in nursing my children without the help of LLL.
I hope you get everything figured out. You have already made it through what is typically the hardest part of nursing. Way to go mama!
Best of luck,
S.