Someone once said, "Parents of an only child don't really know what it's like to be a parent."
Bill Cosby embellished on that thought by pointing out, "If you've only got one kid and something gets broken you know who to blame. If you've got two kids, you never really know."
Good point ... of course that has nothing to do with your question.
Someone else wisely noted a few wrong reasons for having another child: If you look back through history, Cain and Abel started a trend and siblings do not always get along well together and may, in fact, be less companions than combatants. A lot of this, of course, depends on how they are raised - and you may well be able to help them deal with their differences (and the almost inevitable first child jealousies that are likely to pop up). But you may well find your children's personalities clash to the point that they simply do not get along. Something to consider.
Also, don't have another child just for backup in caring for old Mom and Dad in their greying years. (More oftne than not, such a burden still falls to only one child so there is little support in having more than one.) Make your plans for your retirement years and/or catastrophic occurrences now, before you need that extra help.
As others have already pointed out, this is a very personal decision and, as you already know, not one to be taken lightly. All of your reasons and rationales are valid. The memory of your first pregnancy may well send shivers down your spine and send you into paroxysms of horror and fear at a repeat of that experience. However, outside of some physiological problem which contributed to the difficulties during your first pregnancy, there is no real reason to expect a subsequent pregnancy to progress similarly. You also, quite reasonably, don't want to put your household into financial straits by burdening your collective income with the cost of raising a second child. And it is ever so much easier to find someone to keep one child if you and hubby need to away time for just the two of you than it would be to ask someone to take on the care of two little ones for an extended period.
On the other hand, you don't want to be pressured into having a second child just because people keep asking you when it's going to happen. I find the best response to such a nosey question, if it is a casual acquaintance, hubby's work associates, or church member, but not someone close to you ... just give them a look that says, "Now why would you ask such a personal thing?" and just say, "What a strange question," in such a way that clearly conveys you will not be answering their question. Then quickly move on to another subject. (This saves you from being rude while dodging their rudeness!) Such people are not entitled to input or information on your personal life. If it is family members, but you do not want to share your personal quandary with them, you may want to just brush the question aside with some comment like, "We'll let you know if anything develops." And, should they persist, just look at them, as though at a young child, with a slight smile and a look of indulgence that clearly says, "I am not answering anymore questions on that subject."
If it is someone - family or friend - with whom you feel especially close and whose counsel you trust - you might want to confide in them and get their views on your quandary. But, be aware, if it is a family member, unless you know you can trust them not to discuss your personal life with others, they will likely share the 411 with other family members and you will soon be inundated with unsolicited 'advice'. If that happens, just smile at the 'volunteer' advice and say, "Thanks," and let it go at that. This is a very personal matter and, ideally, should be discussed only with those involved in the decision making and "follow-through".
In closing, I would say there are a lot of ups and downs / plusses and minuses both for a one-child household and for the "Brady Bunch"/"Yours, Mine, and Ours" clan. Only you can make the decision as to which yours is going to be. But even you can't make a fully informed decision because, much like the only-child being asked what it's like to be an only child - he can't really answer that question because he has nothing in his life with which to compare his situation. He has only been an only child so he does not know what it is like to be anything else! And you don't know what your lives would be like with any subsequent child or children because your only experience has been with one child only. If you decide to have subsequent child(ren), you may discover that you thrive on the chaos and enchantment of having your beautiful children in your lives. Or, after a second or third child, you may learn that you simply do not have the patience and mental energy to deal with multiple children. Sort of like driving somewhere new without a map or travel plan. You don't know until you get there. But, if you want to get a taste of what might be in store ... your son being three years-old, consider inviting a few of your child's playmates over for a hours a day two or three days in a row. Volunteer to watch them while their moms take a personal day. Being the sole responsible party to care for them may well give you a little peek at what it would be like to have two or three children of your own.