A.B.
Mama, I agree that it doesn't seem appropriate for him to have been there, but IMHO it doesn't sound like enough proof of cheating to proceed with tearing the family apart at this especially vulnerable time. I think there was much wisdom in the advice of the moms who advised you to wait until after you have given birth and gotten established to decide to proceed in either direction.
That said, it is not healthy for you (or the baby!) to be harboring such anger and living with a man who you resent so. If you want him out, have him leave and live elsewhere for now, but I really think this is a TERRIBLE time to act on proceeding with a divorce. It is a deception to think that it will magically make everything easier and you are very wise to recognize that. It's just not true - it's not that easy or fast. This man will always be part of you and your children's lives. Divorce will not erase him.
Get some space and surround yourself with loving friends and family if you can. Make sure your other children are feeling loved and safe. Take care of your physical needs so you can be strong. Prayers and hugs to you. Good luck.