I wouldn't remove my driver's side mirror.
Occasionally folded back is a good compromise and will prove useful I'm sure when I need to use it to see behind (as opposed to beside, which the convex mirror mounted inside the car permits).
And knowing the law, now I don't have to re-deploy the mirror in a panic when I see the police (which I did today).