If you barely have enough energy to get through the day, then your qualms are probably worth paying attention to. Adding another child to your happy and blessed trio may enhance your lives in some significant ways (babies are so amazing!), but detract from your overall happiness and sense of security, at least for the first few years. And if the baby is high-needs, or has any health problems, or even if the pregnancy affects your health for the worse, it is indeed possible to go over an edge from which there is no easy return.
I'm now in my 60's and have been close to many families over many decades. Most of the women I've known have not had all the babies they potentially wanted; life simply provides limits for most of us that we wisely realize we should not exceed, and those limits may become narrower over time for any number of reasons.
I have also known women who bravely forged ahead (or proceeded with accidental pregnancies) who, in their most honest moments, wished for an "undo" button so they could go back to an earlier, simpler time. Of course they dearly love all their children, and they somehow get by, whether coping with birth defects or other expensive health issues, a particularly demanding, non-sleeping baby, loss of a parent's health or job, a divorce, or the more recent melt-down of the economy, which had two young families I know living in shelters for awhile. I've lost touch with one family; the other is couch-surfing with relatives and probably will be for some time yet.
One last factor to consider is that if you do find yourself regretting one more child, that's a stressful and lonely emotional burden for a mother to carry. (And some don't carry it graciously – my own mother told me and my 3 siblings often during our childhoods that we ruined her life and she would have chosen not to have at least 2 of us.)
Contentment is actually a choice, a matter of focus. It sounds like your focus is actually very well tuned, since you realize the happiness and blessings in your family of three. You can choose to keep your attention there. All of us probably wonder at times how wonderful some unborn child might have been, but those of us in my age group who have had the most fulfilled lives don't spend much time or energy there. And this is even true of two women friends who never managed to have bablies of their own. They chose work that had them lovingly involved with children anyway, and they have been largely happy.
I stopped with one daughter myself, and honestly never regretted it, though I adore babies and children. Much of my life work has had me involved with children in one way or another. I also fostered a daughter for nearly a year – she was an extremely dysfunctional 13, and it did not go well, unfortunately.
Whatever you choose, I sure do wish you the best.