Question about The Wizzard mini game

When I play this game I usually get a score that's worth one gem but on some rare occasions I manage to score such high that it earns me 5 or even 6 gems. But the thing is, I don't really get what I do differently on those occasions. I mean, it doesen't seem to me like I am doing that much better (or any better at all) then when I earn only one gem. What am I missing here? Do any one you guys know how exactly the scoring mechanism works? I would really like to know, so I could start working on reaching such high scores consciously.