Was the visitation order issued in Texas or California? That will make a difference--the way I understand it is that the original court maintains jurisdiction unless both parties petition for a change.
Despite some of the responses you have received, it is not always about maintaining control over the child or about fathers having equal rights or about the child support. Child support orders are issued separately from visitation in Texas--i.e. a parent can be ordered to pay support even if they don't have visitation and visitation can't be witheld for non-payment of support.
Some fathers are neither interested nor fit to spend time with the children they father--it is too bad that we don't always figure that out before we have babies. That being said, we have the responsbility as mothers to maintain the relationship the best way we can.
Yes, there is a chance that a judge could issue a "standard" visitation order if he takes you to court. You have a non-standard or modified order. The fact that he agreed to it less than a year ago will likely work in your favor. I hope that you have been documenting which visitations he has missed (if you haven't, start). Regardless of how HE feels, he has the responsibility to be there for his child.
It is not uncommon that children under three years of age DO NOT go for overnight visitation or unsupervised visits--especially if the father has not been an active part of the child's life before the split. For instance, my husband left while I was pregnant and never lived in my house with myself and my daughter. If you are breastfeeding, that relationship is also taken into consideration.
I didn't have so many problems, my husband never even bothered to hire an attorney or show up for court. I informed him of what I was requesting with regard to child support (less than standard) and visitation and he just signed the papers. That being said, I don't restrict his access (within reason). He is not allowed to leave with her because he can't be trusted not to drink and drive. He also is infamous for calling and asking to come and visit right about bedtime--I am the one who has to get up with her in the morning so that doesn't fly either.
When he DOES decide to show up...I let them be. I check on them, but I expect that he changes the diapers, bathes, feeds, cleans-up, etc. when he has his visitation. Just because he doesn't have custody doesn't mean he just gets to do the fun parts.
My advice is to try and keep the legal order you have today and work out some additional visitation that you both can live with. I suspect you are having an especially hard time yielding to his visitation since you had the opportunity to stay home with your child. Try to look at it as a little break for you. It isn't easy being a single mom--I love when he comes and I can just get some things done around the house!
Just a little semi-evil side note: If you don't feel he would be in danger by going with his dad overnight...try it out. Your son won't likely be traumatized by one or two rough nights but his father may decide that it just isn't worth it! People who aren't around children very much don't really realize how rough the nights can be.