I have measured a slowdown in the warmup time by having the temperature set on hot, even with the blower off. You want the engine to warm up as fast as possible, because a cold engine guzzles much more fuel. see below
Once the car is warm, I don't think it makes a difference, other than the power draw for the fan.
Mini-experiment: cold start fuel consumption and warm-up time - MetroMPG.com