It's okay he doesnt drink a lot of milk, my husband and son (and I) drink very little milk. My son would never take any milk, formula, etc so once he stopped breastfeeding (or I should say *I* stopped, he never was very interested) he didnt drink milk after that.
I worried for a while until I realized that we as a nation right now place far too much emphasis on little kids having milk all the time. Think about how the pioneers did it - they only had milk to drink when their cow had a calf, and then it had to stretch among cooking needs, all the rest of the family, and of course the calf as well. (I do wonder if this is why there are so much milk allergies/intolerance going on, because we are overdoing it on the milk consumption?? Soy milk is not the answer.)
Calcium and vitamin D is a concern but you get that by eating a variety of foods like green leafy lettuce/spinach (the darker lettuce the better- and yes he'll probably eat it if you let him have salad dressing, esp as a dip), broccoli, yogurt, and even orange juice. As long as your child drinks water (which no body can live without, it really should be our main drink, and it is in our house, each one of us LOVES water!) and some other fluids, it doesn't *have* to be milk.
My son also was more willing to drink milk when it was flavored with chocolate or strawberry Quik, once in a while. (I wouldnt put that in the bottle for all day though, make it a once a day thing? Also, juices and milks he got to drink in the kitchen/at the table with my supervision, no wandering around with anything non-water, only water in sippy cups could "wander".)
One thing I have learned with my family is I cannot force them to eat/drink anything they are resistant to. Sometimes there is a reason for it. (I'm not talking about resisting veggies, I'm talking about foods they have been exposed to often enough to not reject it simply because it's new or an unfamiliar taste to them.) My husband only puts just enough milk on his cereal to barely wet it and doesnt drink it otherwise, because it makes him gassy. He also dislikes seafood. Once he ate some at a friend's the night before we were flying out to be polite (and the salmon was delicious!) and ended up visiting the pot all night long. When someone has a natural aversion to a food, there may be a reason for it, dont force or overdo it.
So I'd simply keep milk available as an option for your son to drink, as he gets used to milk (remember he hasn't had *any* for a year and now we want them to drink milk all the time). Yes, make sure he has a variety of healthy food choices and of course water/other fluids, so that his body can get what it needs.