Here are my concerns about the scenario where he goes to the junior high school building for math:
1. He's still young (elementary school). He's still growing, and still a kid (presumably he likes to play with his friends). Adding on 90 minutes to an already full day may make him tired, and detracts from a certain amount of free time - time to just be a kid. He'll be the only one of his friends who has to get up in time to get to junior high, and he may be very tired from such a demanding schedule.
2. Being in a class with older students is entirely different than being in a different school. Sure, being a 3rd grader in a 4th grade math class is probably not a big deal. But a junior high school is so different from elementary school. The kids are older, they're pre-teens and young teens, the pace is different, and he might be overwhelmed. He won't be there long enough to make friends (he'll just zip in, go to math class, and get back on the bus) or to figure out how junior high works.
Since he's still in elementary school, unless you plan on accelerating everything, where he'll be one of those kids who loves learning and will graduate from high school at age 15 and already have some college credits accumulated, I'd forego the junior high/bus plan.
Instead, I would request that he be taken to the library or resource room during math class and take an online math class geared for his learning style and math levels. There are many to choose from. Or the math department can provide a tutor for him during that time and give him one-on-one math lessons at his appropriate level.
I think the school needs to accommodate him, not the other way around. Provide a tutor, provide an online learning situation, or adapt somehow.