Trust your gut. Let your children have their own relationship with their father. But being the type of person he is from reading your previous posts (he was a live in boyfriend and left you after 11 years for another woman and her child) maybe them not being with him a lot isn't the end of the world.
Some fathers work/travel a lot and mom holds down the fort. You can treat it like that. The problem is your kids are dealing with a very real rejection because he left you AND them. So they need help with that. You can remind them that YOU love them (God loves them), their family, friends, lots of people. Or maybe the list is short. I don't know, but it's a list of people who love them. They are loved, period. Help them to focus on the good. Focus on being grateful for things, instead of being bitter about other things. Focus on positive, shared family experiences - just the 3 of you - the majority of the time you are together. Turn off the TV/computer and go out together.
My father died when I was 4. So I had no male in my life growing up.
I'm married (13 yr old and 9 yr old) and I take my kids to dr. appts. and sometimes DH shows up to teacher meetings, but not all. I'm way more involved than he is (he does go to school plays and sports events). It is irritating, but not much I can do.
Don't let your friend's bitterness and anger get to you. Find constructive ways to deal with this. Find ways to love and support your kids, despite their father's shameful behavior.
Here is a counter horror example. A woman has a child (say that child is 3.5) and has 50/50 custody with the father. Father is remarried and now has 2 younger children. Father and new wife don't allow the daughter to call their real mom "mom" and critique every single thing the mom does. In a way, you having them is good because you (they) don't need to deal with a toxic situation. The typical divorce situation is 2 households, 2 sets of rules, and sometimes that confuses kids or is just a recipe for more nonsense and drama. Or worse, the new wife treats your children badly. Ugh. Nothing is perfect.
A toxic thing some parents do when they separate is bad mouth the other parent. Don't do that. His behavior is obvious to them and will be as they grow up. HOPEFULLY he will change his ways before it's too late.
I'm so sorry you and the children are dealing with this. For me, what helps me is my faith. Praying to God, leaning on Him for guidance. If you can find a Bible Study class in your area, start there. Some Christian churches also have Divorce support groups for children.
https://www.bsfinternational.org/