I have traveled with my daughter when she was 11 months and about 15 months. I wouldnt recommend it.
we were waiting in a very long line for customs in jamaica and there was a young child in line ahead of us... crying hysterically "go to bed" he was sooo tired and overstimulated... It was late at night and the airport was hot and many flights were delayed. the child was not having fun on his vacation.. and neither were the parents.
Young children -even those well behaved children- need to nap and who wants to spend several hours of their vacation stuck in a hotel while the baby sleeps.. Also the child needs to go to bed early - so again you are back in your hotel room so the baby can sleep.
Kids dont enjoy fine dining so going to fancy restarants wouldnot be part of this trip. And kids dont care for sight seeing...
for the few years that the kids are very young.. go on vacation to th places where they want to go.. like disney world... or someother kid friendly place... go to austrailia alone... or take your son when he is older and will enjoy it.