For Father's Day every year, we go to the Pirate Festival in Vallejo (it's free!!! and you can bring your own food/beverages if you want). Google "Nor Cal Pirate Festival" for the website and details!
I know something my husband would probably like is some time to play one of his console games (Call of Duty, Homefront, etc)... especially the ones which shouldn't be played in front of young children.
And since Dad's the cook in our house, we'll probably eat out somewhere (if we don't eat at the Pirate Festival, or his parents don't invite us over for bbq)
I agree about it not being special until YOU make it special.