Your hunch is correct.
A modern fuel injected engine, fully warmed up and in good tune uses very little fuel to re-start. We're talking about an advantage gained after several
seconds.
Might want to see:
You may want to take starter wear into consideration as well. (I don't worry about it - I haven't had a bad starter since I had a series of old VW Rabbits when I was going to school. Also, starters are usually easy to remove/repair/replace.)