Might be nothing...might be something. But when it comes to things like cognition I prefer to err on the side of caution. Either way when you continue to expose him (to learn) you are only helping him. For all you know he may be a gifted child who is simply uninterested in the methods of which he is being taught.
I have a child with developmental delays. When I voiced my concern when she was 18 mos only 1 person said to have her evaluated. Everyone else implied that I was overacting. I followed my gut and had her checked. I figured if there is nothing wrong ... no harm done. In her case there was something.
My child is also 4. She took on to animals and colors fairly quick but has difficulty with letters. I had some sand paper letters. I have her trace the letters with her fingers and only introduce 3 letters at the most during any given time. When she has mastered them I introduce 3 more and reintroduce the 3 she had mastered previously. She likes to feel them so it holds her interest a little longer.
Flashcards can sometimes be such a drag so just incorporate his exposure in everything you do. "Grab me a yellow shirt from your closet." "Pick up your green socks." This way it does not feel like work.
Colors should be pretty spontaneous at this age. Typically colors are learned at 2. I would wonder about color blindness. Try exposing him to the same color things and ask him to name it. Like "...this is a yellow crayon can you pick out all the other things that are also yellow?"
Bottom line is you are the best advocate for your child. The first 5 years are the best times for learning. You never know all this concern is for naught and soon enough he will be spitting out all that knowledge everyone had been feeding him.
Best to you,
D.