Hi J.,
First of all, let me say that you are NOT failing. Since there are certain development milestones that are basically the same among children, when our kid diverts from the norm, we think there's something wrong with them or we're failing as a parent when really, kids are drastically different in many ways, even when they are young... temperment, abilities, social skills, tastes etc. So don't be so hard on yourself! As parents we do the best we can and that's all we can do.
Having said that, I haven't had first-hand experience with this but have a very close friend who has so I thought I would share that with you. Tommy (not his real name) was two and was barely saying anything (less than your child I think). She and her husband both talked at earlier ages. His Mom was told by the pediatrician that this was normal in some children and not to worry about it but her instincts were that she should press it further. Turns out there was some brain "connection" that was not working that he needed to get therapy for so that it could activate. I'm not explaining this very well but there was no surgery or physical cure needed, just this speech therapy. Bottom Line: Tommy is a wonderful 13 year old (friends with my middle-schooler) who is kind, smart, athletic, well-spoken and much loved by his friends.
Another friend's child was held back a school year on advice from a physician due to speech issues. That child (now 5th grade) reads at a near high-school level, was put back into his normal grade level, and has astounded a group of native Chinese with his ability to speak Mandarin.
Try not to worry about something being wrong until it happens. Just take this thing one step at a time. Do you have to have a referral from the pediatrician to see a specialist? If you think it's necessary, push it, nobody knows your child better than you. I had a very young doctor poo poo my concerns about my young child's hearing and speech. I finally sent her a letter stating I disagreed and if she would not give me a referral, I would go to the specialist independently, asking for her written decision. I never heard back from her. My son had ear tubes 3 times, his adenoids out, and it was clear from the blockage in his ears that he couldn't hear certain sounds and it was affecting his speech. He is fine now.
If the situation with your son turns out to be nothing--FABULOUS! But your family needs to know now, right? Better to be safe than sorry.
GOOD LUCK to you.
T. from San Diego