An introvert myself, I think one way to explain it to your husband is this: for introverts, having that much forced interaction is upsetting, not enjoyable. You want your son to start a sport not when he's 'supposed to', but when you think it would be something he would enjoy. Socially, for some of us introverts, team sports with large groups aren't enjoyable. This requires the introvert to pay constant attention to other people and everything they are doing, reading them all the time. For introverts, tracking groups of others can be stressful. It may not feel 'safe' for them, because people are in each other's space during sports.
It's just plain unpleasant.
All of this said, our son is now five and a half and just now starting to express interest in martial arts. His cousins take karate lessons and we've walked by the local tae kwon do place a few times. As your son gets older, he's going to feel more comfortable socially and become interested in something. Not being interested at four is super typical in any case; I hope your husband takes heart in this. From my observation, it's really often the parent who signs their kid up or thinks they 'need' the classes or lessons. If the child enjoys it, chances are they will continue. Think of all that children need to learn in sports, too. Sportsmanship in and of itself is a huge personality trait that is real work to attain for many. It does sting to lose. Non-competitive sports would be the way to go at four. "Losing" seems like the stakes are too high, and that many people running around may just be too much for your little guy.
Take heart... you have years, an entire life in front of you for sports and other fun activities. Enjoy this time with your little guy in his comfort zone~ the more you respect it, the easier it will be for him to venture out of it. Your acceptance of his needs now will give him more security in the long run. Good luck to you... and please, tell your husband that your son is like many kids, not just introverted ones. I've worked with little ones for a long time... they are certainly their own creatures, in any case! :)
ETA: and Odd1 wrote:
More than you asked: I will say I am a stickler for politeness. My daughter does not have to be friends with everyone, but a greeting, a smile, a response is required, even if it is simply to say "maybe we can play later..." There are skills/tools to be learned and the choice to use them WHILE honoring one's innate being.
Beautiful expression of expectations! Thanks for writing that.