First off, what they said was out of exasperation and anger, obviously, so don't take it at face value. But don't ignore it, either - it sounds like there's some serious miscommunication going on if they feel as though you are keeping your kids from them. They want to be a part of their grandchildren's lives. (I know, I know, my in-laws drive me a little crazy, too. But I make sure the kids get to see them.)
I'm a little confused. On the one hand, you say you think "the kids are doing fine being here at home with us this summer" then you say "My hubby and I are workin 5 12 to 15 hour days." Are the kids home alone all that time? Because if that's the case, I don't think summer camp is really *such* a bad idea. (Expensive perhaps.) My son started going to summer day camps when he was almost four, and he LOVED it! Learned to climb a tree, ride a pony, and swim underwater - lots better then vegging in front of a TV.
Maybe the grandparents can watch the kids while you're are work? Win-win situation, no? Grandparents and grandkids get time together, you have someone watching the children while you work. That seems too obvious, though - is there something else going on you haven't mentioned?
And even if they don't legally have a lot of "rights" regarding your children, in our hearts we know that children have a right to see their grandparents and vice versa, so try to find a way to make it happen. It doesn't have to be when you're enjoying the little time you have away from work.