Dont worry about the neighbors. I am sure that once you break the habit the crying will stop. In the meantime, I would either let him cry it out or, if you cannot do that, stay in his room but do not take him out of his crib. That way he knows you are there but he has to stay in his crib. I have been reading "Healthy Sleep Habits Happy Child" by marc Weissbluth and it really works for me. One thing they talk about is how early the children learn to manipulate the parents. If he cries and you come, he learns that that behavior works and will continue to do it in order to be with you. He will cry for longer and longer as long as he get the results he wants.
My son got me into a bad routine of continuing to return to his room at bed time. I was worried he was still hungry so I kept bringing him to the kitchen to eat. Finally I decided enough was enough. One night I went up to his room when he was crying, did not take him out of his crib but still hugged him, told him I love him but he has to go to sleep. Once he realize he was not leaving, the games stopped.
Another thing too, try to keep the lights off and your behavior subdude. Dont react much, smile or make the time when you are there fun, give him what he needs and then leave. That may discourage him from wanting to go with you.
I hope this helps and good luck.
A. G