[aerohead]

You'd lose some of the field of view, a safety concern, compromising situational awareness.
The housing would no longer be streamlined.
What was the inner face, parallel with the flow, is now cocked outwards, as a wind-catcher, producing a jet where the flow is now nearly completely choked off.( pressure spike ) on the leading side, directed towards the glass.
The outer housing inwards curvature now exceeds the 23-degree threshold for attached flow, triggering turbulence behind this outer housing area.
Like it or not, the original housing was optimized for this area, and the separated flow-turbulence was helping to scour moisture from the mirror's surface.
As per Subaru's research, simply extending the stalks out, to create an 80mm separation between the mirror housing and side glass would 'solve' any drag issue.