If the fabric or something about the shorts is defective then by all means take them back to get a refund.
But if they were stretch denim or something like that then you kind of bought them knowing they would stretch out.
The store basically has to throw them away and take the loss. It really annoys me when I have some new garment and something is wrong with it.
I bought a really nice red dress with a huge white collar one time. I wore it a few times and then took it to the dry cleaners to get it cleaned. They tested the colorfastness of it and refused to take it. The fabric color rubbed off on to the rag with cleaning fluid on it. If they cleaned it every other garment in the drum would be dyed red. They could not clean it.
I took it back to the store and they let me exchange it for something else. I still have that dress and have had it cleaned many many times. It is one of my favorites.
On the other hand, I had a friend who would go to Target every day or every other day and buy a garment, wear it out that night to the clubs, leave the tags in it, then take it back the next day or two to exchange it saying it didn't fit when she tried it on at home. That was so wrong...so many people do this and stores are getting more and more where they won't take back anything that is worn. I hope they will work with you on this though.