E.,
I think it's wonderful that you feel the way you do about your little one! Now days, it's all too common for women to have a baby and go back to work in 6 weeks, leaving the infant with a sitter. Unlike some others, I did not get the impression that you needed a break from her. Why don't you make friends with other moms, so that you can take your daughter with you? Eventually you will want to leave her! :) But don't rush it or force it if you are fortunate to be able to stay home with her. The MOPS, mothers of preschoolers, group is national, and you can find it at many churches.
You said you get out with your husband and leave the baby with the grandparents, so that's all you need, isn't it? Is your husband reasonably satisfied with the amount of time you have together?
It's a cliche, but children grow up fast and they are precious. Many people cannot conceive or lose a child to illness or accident. Please, please, don't feel guilty about wanting to be with your child! You will eventually want more time away when the time is right.