I siliconed plastic covers on the truck:
On the Prius, which has door-mounted mirrors, I fiberglassed over the hole:
Before you fit smaller mirrors, test them to see if they are actually lower drag than the larger stock mirrors; they may or may not be. Something like this will tell you which mirror has lower drag (the one further from the pole's centerline when there's no torque developed):
But that isn't the whole story; you'll also need to investigate whether the lower-drag mirror by itself is also the lower-drag mirror next to the car, where there is interference between the mirror and body.