E.E.
I don't think you have anything to worry about at all! My son is 13 months old and not quite 20 lbs, and about 29 in tall. He has always been between the 5 and 15th percentiles for height and weight. Only recently his height went up to 50th, but his weight is still around the 5th. But he is very healthy and does not look skinny at all. And I have always been small- I'm only 5 feet.
It's normal for all kids to not grow as quickly after 6 months. The first few months babies grow at such a rapid weight and then it slows down because of a combination of things- a natural slowing of growth, an increase of movement (crawling, etc) which burns more calories and starting solids because solid foods have less calories than breast milk or formula.
Since you say you and your husband's families are also small, I think this is probably a genetic thing. I think you are wayyyyy over-reacting. Don't bring him to an endocrinologist. That is totally unnecessary at this age. As long as he is hitting all his milestones and the doctor says he is ok, then he is absolutely fine.
You also have to remember that many kids today (even babies) are overweight, so the growth charts are different than in the past. It's actually better to have a child that is a bit lower on the charts than at the top. If you look on an old growth chart from the 70's, when we were kids, I bet your son is in the middle range. My mother-in-law still had my husband's old baby growth chart and our son is growing exactly the same way my husband did, and my husband is 6 feet tall.
But honestly, what's the big deal if he does turn out to be short? I'm short and the biggest problem I have is finding pants that fit- I just get them hemmed. I'm not worried at all, and you should not be either.