Aww. I know how you feel - been there, too. Then I went back into the school (I drive my kids) to tell him I loved him and have a good day, and I saw him laughing and playing with his friends without a care in the world. So I left again, and just talked about it when he got home. It's always good to tell our children we're sorry we didn't handle things better. They like to know that we make mistakes, too. And it's a good example of how to take responsibility for our mistakes, and make amends.
Blessings to you and your son.
P.S. BTW, my son was the same way at that age - hated wearing shorts, only wanted to wear sweat pants, because they were the most comfortable. Fleece sweat pants in summer!! Drove me nuts, but I let him do it. He grew out of that phase, thank heavens! Then he went through a phase where he would ONLY wear shorts!!! Yep, even in the winter! Thank goodness for snow pants, which I would insist he wore over his shorts if he went outside the house. sigh........