You put it very aptly: you're driving yourself nuts! Put on the brakes and rest. :^)
I'm laughing at my mental picture of your boy saying a thing once or twice and then thinking to himself, "Well, I know THAT one," and shelving it.
What does your child's doctor say? If the doctor isn't worried about any sort of hearing problem (or other problem), and if your boy is doing well in other things, it might be a matter of patience and timing. The "schedules" given for child development aren't written in stone.
Just treat him like the smart boy you know he is... work with him on attentiveness and obedience... and wait to see what happens next. One day he might open his mouth and say, "Sis, would you be quiet for a few minutes? Nobody else in this house can get a word in edgewise!"