Your ex must have an equal long drive to pick your son up from day care unless the day care is half way between the both of you. If it is half way, have your ex take him to day care on monday mornings and you pick him up monday after school. If it is that far from him to pick up your son, as it is for you to go to pick him up from his dad's then he is doing his part. You could suggest meeting half way, both on Friday and again on Sunday but I rather do the whole drive then two half drives, same distance when it all comes out at the end.
There is thngs you can do to make it more pleasent, take a friend with you, cut off any negative remarks by saying "Excuse me, we need to go" and pack your child up in the car and go. You do not have to listen to him go off on something, you can change the subject and not acknowledge his remarks.
I know it is hard but the thing to remember is this is about your son... parenting has a lot of inconveniences to it and will from now on but that is part of the job and a lot of the rewards of parenting. Your son needs time with his dad and he needs his time with you. My son passed away in June and I meet his wife half way each Sunday to take my granddaughter to her mother. I was going over and getting her on Mondays and taking her home on Tuesdays until the weather turned bad and they are about as far away as your ex is from you. Do I like all the travel, not really, if I had my way they would still live in the same town as we do, but she moved back home with her family and I have to deal with it... it is much better then not seeing my granddaughter each week! You can set ground rules with your ex if you are able to stand up to him and get him to abide by them but all in all, you can't control what he says or does, just your reactions to them.