I know that feeling. But I have the benefit of hindsight, as my children are all grown.
I always wanted four children, my husband wanted two. We compromised with three. I felt if I pushed for the fourth, my husband would resent me for it. I no longer believe that would have happened. I think he would have eventually embraced the idea. I still often wish I had had another one, but I also often think he was right. His reasons were all financial reasons. Mine were purely emotional. I didn't look past when they were babies (financially speaking). He envisioned putting them all through college. We have one graduated from college and two currently in college. It IS incredibly expensive! I can't imagine putting one more through college. Even if we had expected them to put themselves through college, the older they got, the more expensive they got.
After my youngest started school, I abandoned the idea of having another. At that time, I had some time to myself (SAHM) and all the kids were, what I like to call, voice activated and I didn't want to go backwards. But when I was really, really, really sure we made the right choice was when they were teenagers. Driving lessons just about did me in! I will take diapers, potty training, terrible twos... all of it over driving lessons. And worse than the driving lessons is when you have to let them drive away by themselves! OMG! What stress! Also, my kids were exceptionally good teenagers, but the WERE teenagers. It was not my favorite age.
What I'm trying to say is, for me, as much as I really wanted and still sometimes wish I had had one more child, we have three healthy, happy, wonderful grown children and I feel so fortunate I could burst.
My husband contends that I would have ALWAYS wanted "just one more" no matter how many kids we had. Can't say I disagree -- I do love babies and children! =) Could you possibly be the same? Part of parenting is making responsible decisions. Bringing only the number of children in the world that you can care for, emotionally, physically and financially is one of the most important decisions you'll have to make.
I hope you and your husband can come to an agreement that makes you both happy.