At 4 mos, I'd be doing more, yes. I have a 6mos old, and whenever I set her down, I'd put her on her tummy. Until he can roll, you'll have to move him when he starts to cry (I wouldn't let him just scream face down in the carpet like your doctor is suggesting, that's really mean!) Just give him a break, and next time you move him somewhere, put him back down on his tummy. Use that as your go-to position for him. He'll get used to it quickly.
They all hate it until they can move around. But the more you do it, the stronger he'll get and he'll gain more control.