I say give it back. Babies/toddlers let YOU know when they're ready to give up something.
My son, who's 3, has graduated from living on his binky to only taking at night to fall asleep, then he spits it out once he's out like a light and we're good.
Same with his bottle, he took it until he was ready to get off it, then he stopped it on his own.
Binkys are no big deal I don't think. If your son can go most of the day without it and only want it at night, then I don't see why you should fight him...you're only putting yourself through the drama of not getting enough sleep. Why make things hard on you AND your son? Life's hard enough.
Good luck. :)
http://www.daniellewrites.webs.com