When my boys hit 3 1/2 months old, they had growth spurt and nursed and nursed and nursed every hour, and never seemed full. They would nurse, then come off and cry (because I was out of milk) I was trying so hard to follow doctor's recommendations (which 5 and 3 years ago was to start solids at 4 months, not 6) but my first son actually lost weight in those two weeks.
And it's not like I didn't have a plentiful milk supply up until then. I had so much I'd wake up in the morning rock-hard and have to pump before the baby woke up. I had a freezer full of milk, and was so excited that I actually called up the breastmilk bank to find out about donating. But I suddenly had to use that milk to feed my ravenous son. He used it all up in the two weeks before I started solids.
Then with my second son, I started on the minipill. A few weeks later, he was crying all the time. I took him to his 4 month check up. He's a BIG kid, so the doctor didn't seem concerned that he hadn't gained any weight in the last month. That's stupid. Babies ALWAYS gain weight. I didn't listen to my doctor about not worrying. I stopped taking the minipill and my son stopped crying so much, slept more, and SUDDENLY shot right out of the clothing size he'd been wearing.
See, I was unintentionally starving my children, and my doctors weren't worried.
So do what your gut tells you to do. If you think your baby needs more food to be full, happy, and GROWING, he's definitely not too young to start on solids. In fact, don't they say that babies that wait until 6 months to start solids are less open to new flavors? I've kept up on the recent recommendations, and they say BETWEEN 4 and 6 months, not WAIT UNTIL 6 months.
From my experiences, I'd tell you start solids right away. But that's a decision only you can make.