i would hope not. and i think you on-the-spot explanation was a good one especially since it was impromptu.
my daughter and i used to change at my gym after a private swim lesson around the same time as the zumba ladies were getting clean and clothed. all those naked ladies of different colors, sizes, nationalities and age. i was just waiting for it. a comment about someone's appearance. and then it came. she started mimicking someone with a lisp. i was so unprepared.
there was an article recently either in the washington post or ny times (i think) about kids noticing skin color at a young age and about how parent's were so hesitant to bring up race at all with their children. whenever some one complains about the effort to be careful when it comes to race i want remind them that if we were really in a color-blind world then questions like this wouldn't get asked. we would be beyond that.
i think you did great. my daughter goes around trying to guess what languages everyone is speaking. i'm sure she has offended someone but she's 4. her curiosity is natural and so is your son's. and as for the person who implied that got his ignorance from i'd say that most 4 yos aren't even aware that there is a continent called africa and he has part of the story right. many of the black people in my neighborhood are from africa. and my daughter tells me that they speak french. ; )