I have three boys. It has been worrisome, for me, to learn the two oldest have Aspergers Syndrome and the third is autistic (though high functioning). It has also been a blessing that I have had help through the ECI program and then at the public school for the speech therapy my boys needed/need.
I remember hiding my tears when I saw my nephews (my sister-in-law has three boys all near the same age as mine!) speaking full sentences. Even if the pronunciation wasn't fully developed, they were talking and mine weren't. I could only console myself that every child is different.
Please ask these folks telling you their opinion that you thank them for their opinion and ask them to mind their attitude when they are around your toddler because you don't want him treated any differently (because of their opinion). These people are quite focused on comparing children. Be cautious that praise for a child's achievements doesn't imply they are better then another child.
If people continue to be nosy, say that you find their opinion inaccurate (they don't have the training, right?) and that you have your son's best interests at heart and are doing all you can to see that he can be the best he can be.
If I were in your shoes, I would seek the (free) assessment of a certified person in the ECI program. You may find the phone number for your local ECI office in the phone book or do a search on-line. Regardless of the recommendations of the assessment, you can gain some self-assurance for yourself and for being assertive towards those people who keep saying that your toddler might be autistic.
From your description, I bet that your son is growing at his own pace (the beat of his own drum) and is just fine. It would be better to compare his own progress over the months and years instead of looking for the contrast between him and other children.
Best wishes,
D.