You're taking a
significant hit from leaving it up there when it's not in use.
Mini-experiment: the wrath of roof racks - MetroMPG.com
These roof racks:
... reduced my MPG by 12% at 55 mph -
empty. With the bike in the rack, mpg dropped 27%.
Obviously not directly comparable to your vehicle, but as a rule you should avoid carrying things on top of a vehicle if at all possible.