Ask your son what HE might want to do and get him involved in the planning and what is doable and what is not (and explain why not).
He may be interested in having a family day, going to a museum, for example, maybe with one or two of his friends along, or just as a family. Make sandwiches and have a picnic. Even if it is cold it can be fun to sit at a picnic shelter on the museum grounds, have a picnic lunch, and maybe toss a ball around or lay tag to warm up again and to get some of that energy released.
At home: you do have a table that you eat family meals at? How many does it seat? Invite two - six (depending on the total number of seats available, to include him)of his friends (and yes, at 6 years they can be dropped off and picked up again by their parents) and have a fund afternoon of playing board and/or card games. Have some cupcakes prepared, and the last activity can be that they each frost and decorate their own cup cake, then they all are put together to form a 'cake', you all sing Happy Birthday, and then they can eat their own creations.
If the timing is so that you would need to feed them lunch or dinner: get a bag of small pita breads, tomato sauce, cheese and a few topping selections. Again, they can make their own mini pizzas, play a game while they're on the oven, and then eat them. It is cheaper than ordered out pizza, probably healthier, definitely more fun, and it is a party activity at the same time.
My boys at that age liked treasure hunts or scavenger hunts (actually, they still do, as teenagers! LOL). That could be done in a park, but it can also be done in your own neighborhood. Get the older son and Dad involved, they can think of a neat trail through the neighborhood, mark it (with sidewalk chalk), leave little notes that give additional clues as to where to go (and they could involve puzzles, riddles, math problems, etc. There can be a treasure hidden at the end (e.g. a blow pop for every kid, or a mini notebook for their future detective adventures, or whatever is small, cheap and neat, just find an imaginative name or description for it) - or it can lead right back to your home (Dad and brother will accompany them, to supervise and to make sure they don't get lost, and to help if they're having problems with any clues) and you will be ready with the cup cake activity for them and will have had a breather.
There is one game that is played around a table that has been played at pretty much every birthday party my kids have ever had, they enjoy it so much.
You will need a chocolate bar (big enough so that there will be at least a couple of pieces for each player), well and tightly wrapped in multiple layers of newspaper; a die; a pair of mittens, a hat and a scarf; a (sturdy) plastic fork and knife (i like to use some from a play picnic set, those are blunt and very hard to break).
All players sit around the table, all items are stacked in the middle, the birthday kid gets to start. He rolls the die and passes it on immediately to the player to his left. If it was any of the numbers 1 - 5, nothing happens, if he had trolled a six, he shouts "six" and gets to put on the hat, wrap the scarf around his neck, put on the mittens, pick up the for and knife and then may start trying to unwrap the chocolate. Meanwhile the die keeps going around the table, every kid rolling it and passing it on. As soon as the next player rolls a six, the kid who was working on the chocolate (who was getting ready to do so) needs to pass all the items to the player who rolled the six, who in turn gets to gear up and work on the chocolate, until the next six is rolled. As you can imagine they will get very exciting (and loud), there will be a lot of back and forth, and it will take quite a while until they get to the actual chocolate, get through the original wrapping and actually touch chocolate. Now, while you could let the game continue and have the objective it continue until all the chocolate is eaten (with fork and knife, of course) I usually call a stop once they hit the chocolate. I usually break the chocolate to pieces then and distribute it to them evenly (or put their pieces into little plastic snack bags, which hey then can take home). Playing it to the end often gets messy, chocolate crumbs fall to the floor, and usually it gets shared very unevenly with some kid not getting any, and lots of it that needs to be thrown away. A waste in my eyes.
The better wrapped the chocolate is, with many, many individual layers of newspaper wrappings, the more fun it is and the longer it can go.
I'm not a big fan of them just watching a movie, I'd rather they interact with each other. For that same reason I always had a rule that they get to invite as many friends as they are turning years old. When my kids ere at parties with many kids they often came home disappointed because they felt they did not really get to spend time with their friend whom they were going to visit for his birthday.
But there are so many games that you could play around a kitchen table. If you don't own any suitable board games, check out the library at your sons school as well as the public library, they often have some games, and you could also check at your church if you could borrow a couple of board games, they may have some, too. Or you can mention on the invitation that the guest may bring a favorite board game to share at the party.
Most of all, get your son involved in the planning, find out what he is dreaming of and then work with him to try to make it a reality within your constraints.