S.T.
my kids have often made ginger bread houses at school, girl scouts, etc, made with small milk cartons as the underpinning and graham crackers as the "gingerbread". I think this acitivity works really well with older kids (8-12) in groups, and individually or with one or two kids at this age of 4. It will get VERY messy and they will want to eat the candy more than "glue" it to the houses.
I would suggest a much less messy craft for kids this age - maybe paper chains (using glue sticks) or baby christmas tree ornaments. A craft that my son made that was adorable was a snowman tealight holder made from a babyfood jar, white tissue paper torn in little peices and glued on, red yarn for a scarf and small bits of black felt for the "coal" mouth, an orange piece of felt for the carrot nose, etc. have fun!