Please don not worry. I haven't read any other responses yet,. so I might be re-stating something that others have said.
If baby is peeing and pooping properly, they are FINE! I had the same concerns with my 2nd child, who was born at almost 10 pounds, and it took her a full MONTH to surpass her weight gain. The docs pushed and pushed me to supplement, which I never did do. Turns out that my daughter was just a slow gainer- at 16 months old, she's still only 22 pounds. She's hung out below the 30th percentile for weight since she was born.
There is nothing you are doing "wrong". As long as those wet diapers are actually wet- not just damp, then baby is eating as much as they need. Of course keep on the lookout for signs of dehydration- lethargy, getting weaker, sunken soft spot, etc.
Also, there is no need to intentionally add calories to your diet- your body will let you know if you need to eat more.
If it's taking him a full hour to finish a meal, I would say that he perhaps doesn't have an efficent suck. You might be able to work with him on it if he isn't latching properly, or it might just be something he grows out of as he gets older and stronger.
And my last tip- don't feed him on a schedule, or "by the clock" Nurse whenever baby starts rooting or chewing hands.
I'm sure by 3 children, you know all of what I said, but maybe my suggestions will be helpful to some other mama reading. ;)
I'm fairly sure your weight loss has nothing to do with milk content. I gained 24 pounds during last pregnancy. When I weighed myself a few days ago at 2 weeks PP, I had lost 28 pounds total, 4 pounds below pre-preg weight. And my daughter seems to be gaining just fine.