It seems to me he just needs your comfort and warmth. Our 3.5 year old is still the same way.
I didn't mind pulling them in bed with me from their crib when they woke in the middle of the night because it was the best way for me to get a good night's sleep, and as a working Mom, driving 100-200 miles/day, I couldn't afford to lose any more sleep.
Our daughter co-slept with us for the first year - sounds crazy. Initially it was because of nursing, but when I was diagnosed with cancer, it made me feel better (and more secure) to have her there most nights.
She's a fantastic sleeper now. So, my personal thought is that what you're doing is OK. Probably won't turn into a long-term habit.
Good luck, and I hope you can start getting some more sleep.