Passenger mirror is definitely worse. It increases both frontal area
and drag coefficient. Wipers only increase drag coefficient (and likely to a much smaller extent than the mirror).
http://www.metrompg.com/posts/mirrors.htm
I measured a 2.3% increase in fuel economy at 55 mph with the pass. mirror removed, and driver's folded back.