No, you're not being petty. I believe very firmly in vacations! You might need to adjust your idea of "vacation," though. Instead of a week, see if you can get away for one night.
Camp out! Kids love camping. Don't worry about tents, sleeping bags and STUFF for camping. Just take your bedding, or sleep in the car. Or sleep in the back yard. Pack food (fruit, peanut butter) that doesn't need cooked or refrigerated. Find somewhere to swim. Take a walk with a scavenger hunt list - who can find a pinecone? A seashell? An animal? Find out about any county or state parks - they usually have free programs led by rangers. Often kids get "Junior Ranger" buttons or certificates.
Make a big deal out of the things you CAN do! Don't say, "I'd love to go somewhere nice, but we have to settle for this." Say, "We are so lucky! We're going on an adventure today!" Go to the library - it's free! They often show movies once or twice a month, or more during summer - find out when. Bring your own snacks. Find a park near you, and spend the afternoon there. Then buy a cheap box of popsicles at the grocery store. What would you do in your own town if you had out of town guests? Go do that!
We never had much money when I was growing up, but my parents took us on a lot of picnics, hikes, trips to the lake, cookouts, camping and anything else free, cheap or just the cost of gas (or a bus ticket). When we went out of town, we stayed at a relative's house or camped. And we did a LOT of day trips and weekend trips, because it's hard to do a week or more! We all have great memories of those times, and we didn't feel deprived a bit.
If you make a big deal about the stuff you do, when your son goes back to school and is asked what he did this summer, instead of saying, "Nothing," he'll be excited to tell about the water fight you let him and his friends have at your house, about video night in the back yard instead of on the couch, about getting goodies from the library reading program, about popsicles in the park or the weekend camping in the mountains and how he's sure he heard bizarre animal calls. If you're excited, the kids will be excited. And hey, free and cheap are now almost status symbols ;p - everyone's looking for those things!