Welcome to Mamapedia - despite the name, men are welcome and have been members over the years I've been on this site.
What do the kids do during the day when they are with you? Even if you are home with them, or even they have an in-home babysitter/nanny while you are at work, it makes sense to keep them in a consistent situation rather than get pulled out on Fridays to go to a different day care. If they are not with her, they should be with you or in the same situation they are in the other days of the week.
I don't think your rationale should be "they're all sick" because kids get sick in day care and at home and in public school and at the grocery store. You have a lot of kids bringing home germs from school and sharing them with the little ones. I'd suggest you increase hand washing, absolutely do NOT use antibacterial soap (potentially harmful, FDA is investigating, and definitely a factor in the killing of weaker bacteria and letting super bugs flourish), and DO let the kids play in the dirt (and outdoors in general) a lot so they get exposed gently and gradually to the mostly innocuous germs that help build the immune system. There are also things you can do nutritionally to boost their immune systems safely, assuming your ex will participate.
If you don't have a written visitation agreement, then I think you either should get one, or you and she together should work out a schedule. She doesn't tell you when she gets them, you don't tell her when she gets them. I'm not sure she should have them every weekend though - you should be entitled to some time when the older kids are out of school. So I would suggest, if you have an amicable relationship, that you write up a schedule - that means going through the calendar, marking up Father's Day, Mother's Day, parent/kid birthdays, and whatever holidays and 3-day weekends are important to each of you, and then figuring out something for the summer. It might make more sense, however, to work this out with a mediator (which means 1 legal expert with costs shared by the 2 of you, with the final agreement filed with the court at far less expense than each of you having 2 your own lawyer or actually working things out with a judge). Once an agreement is filed, it doesn't mean you can't swap on occasion if it's agreeable to both parents, but at least there's a structure and every single week isn't being negotiated.
I also agree, I'm sorry to say, that you should have paternity tests on the 2 younger kids, at the very least. I think it's very suspicious that she magically ran off with someone and then you find out she's pregnant. Twice. The children will all still be half siblings so it doesn't change their relationship to each other, but it's important for your own peace of mind as well as support agreements and potential medical issues that everyone know who has whose family medical history.
I think you need a professional to help.