I have a similar experience as you describe, only I'm the sibling who has been in school for 28 years of my life, and my brother has learned everything he knows "hands-on". I have two college degrees in Computer Science, and will have a PhD in Psychology. He works, very successfully, as a Software Engineer and has never stepped foot on a college campus. He taught himself computer programming nights and weekends while raising two small kids and working well over 40 hours a week doing mostly manual labor. I have the highest regard for him, if you can't tell ;-)
If you want me to get into the definition of intelligence, I had several weeks of graduate level instruction on just what that definition is, how it has evolved over the years, how it is measured, and could have a long (and maybe boring) discussion about the reliability and validity of the various tests and measurements of intelligence... That's "book smart" even though I do have great work experience and real life experience to draw from, also. My brother is "street smart" as in, he is self-taught, self-confident in any situation, and better working with a wider variety of people than I am.
My final analysis - I'm inclined to say he's much smarter than me, no matter how we'd score on any test or measurement of intelligence.
He and I both have amazing and sometimes annoying brains, as do my other two siblings, because we were lucky to come from two people who passed down both their genetics and mostly their work ethic. My oldest brother can fix any machine he lays his hands-on (really, he has been known to diagnose engine problems by having you hold your phone up to the car while it's running...) and my younger sister is amazingly creative and social and runs her own hair stylist business. Our life experience varies, and our preferred method of learning does, also. We are all "smart" in our own ways.
And, I love to read books - all kinds - those that give me knowledge and those that entertain.