Your ped is absolutely correct -- it is probably too soon to label your ds as a late talker. By 23 months, I think children typically say 8 to 10 words. Frankly, it sounds like your sister was not a late talker since 2-3 yr olds speak as least 40 - 50 words if I remember correctly. Unless there are other developmental delays (and it sounds like you're son is on track otherwise), peds typically do not refer children until about 2 1/2.
Also, a delay is just that....a delay. It's different from a speech disorder. So calling a person speech delayed is the same as saying they're a late talker.
My suggestions would be to do things you're probably already doing such as read to him. Talk and describe what you're doing, what he's doing, what you see, etc. Offer him choices (do you want the ball or car). See if you can have him point things out, etc. Consider teaching him signs. Signing does not delay language development -- just the opposite. We found that signing decreased my son's frustration and really opened the door to speech for him.
My son was diagnosed with a delay at 26 months. He actually went in before 2 1/2. My ped and I agreed that if he didn't talk like a chatter box by a certain date, we'd have him evaluated. Like your son he wss on track or ahead in other areas. Long story short, he's been in therapy for a year. Fortunately, insurance does cover it. I will say that it is not cheap and he has had close to 15k in therapy.
I think your son is just too young to label as speech delayed considering he's not even 2 yet. He may surprise you yet. Start keeping a list of all the words he says (even if only once) and phrases he uses so that you can better evaluate his situation. This will also be helpful is you do decide to go the therapy route later on.
Good luck!