I traveled from Southern CA to Virginia/DC once when my son was little, but my husband stayed home. Honestly, it was totally fine for me. I missed my son, who was about 18 months or so, but he did great and I had a really nice time being away. I forget how long I was gone... 3-4 nights probably.
A couple of months before that, my husband and I went away together for a three night cruise. That was harder because we couldn't call home to check on him as easily. Again though, the time away was worth it to rejuvenate, reconnect, and just to have a little break.
My daughter was a little younger the first time we went away without her - 10.5 months - but she was fine too. That trip we were gone for two nights. It was harder on me because I was still nursing and had to pump while we were gone, plus she was terrible about taking a bottle, so I was worried somewhat about her eating enough. But, she was on solids by then so she ate extra food and took one bottle while I was gone. She started nursing again upon my return with no problems.
Don't be nervous. Everything will go fine and you are going to be sooo glad you did it.