Time to drop all the testing and diagnosing. He doesn't have a diagnosis except the vague one for "intellectual disability" -- so it's time to stop hunting for one and turn that same energy to something more productive. A lot of what you describe, especially now that you have ruled out ADHD, ADD, autism, etc. etc., sounds like fundamental immaturity, and that's not a condition that can be diagnosed or treated. The fact he is, as you put it, "THE youngest in his class," combined with the fact that generally boys often mature more slowly than girls, indicates that he might benefit from time, time and more time. It's too bad he didn't get an extra year off before K, and he could still be held back, but the fact he's doing decently in school probably means they won't hold him back though he would benefit from it.
It is interesting to me that you finally found this diagnosis of "intellectual disability" yet you note he also, your words, "Has never been behind academically...never really ahead, but never behind either." That does not sound like an intellectually disabled child but kid who's doing OK -- unfocused, yes, but not crashing and burning academically. In your place I would be skeptical that possibly whoever gave that diagnosis was...well, just seeking a diagnosis to give. Didn't the person doing the diagnosing give you next steps to take, specialists to see, classes he could take, etc? If not , have you asked them why not? Did you just get handed a diagnosis and sent on your way with no next steps outlined for you? If so -- if no one said, "Now here's what you do next and here are numbers to call, etc." -- that is a red flag that maybe they were seeking just to placate you with some, any, diagnosis since they realized you were seeking one so hard.
Your gut knows better, mom. You already say that this diagnosis "doesn't make sense to me" and "he is coming along in other areas of his life for sure." Please trust yourself more -- you know he's doing OK, just not as focused or advanced as some other kids (and probably doing better than other peers, too). Normal, so far. What have his classroom teachers, who spend far, far more time with him than test-givers or doctors, said? How detailed have your talks with them been? What do THEY think of IQ testing at this age? Do they reassure you that though he's not very focused he's doing OK in school and grasping things and learning? They see kids all the time, all day long, and often have the best read on a child-- better than some clinicians who just administer tests.
Take the energy that has gone into getting diagnoses and instead use it to find an occupational therapist to work with him on his low muscle tone (I've seen OTs do wonders and kids be done with OT by the time they're done with elementary or earlier). Keep up the swimming and karate and don't put him into team sports if he finds them stressful. Ask the OT about other things to do to help with coordination. Keep up the summer school if that is working well for him and he enjoys it, and also consider taking him to a lot more things in summer such as library story times (some include crafts tied in to the week's story), lots of play dates, etc. Ask the teachers at the summer school what they think, especially as they work with kids with learning issues.
Get set up with the school counselors, principals and teachers before next year begins -- call now, before they're all gone for summer, and make an appointment to meet the week before school starts in fall and set up an IEP (if one is needed -- I wonder if it really is) or otherwise set up frequent communication among you, the teachers, the counselors and the administration so his focus issues are handled well and swiftly and you know what goes on each day pretty closely.