I vote yes. Almost everything in modern EVs is recyclable, so it doesn't have to be wasted. Yes, mining and making and refining the materials is harmful to the environment, but so are all the materials used to make conventional vehicles. The rare materials put into hybrids/evs are fairly small in volume.
Toyota has done an analysis (I know, not the best considering they're selling you something) of the 3rd gen Prius and its environmental impact to manufacture vs the environmental benefit of its reduced environmental impact of being used. Their conclusion was that the use is by far where more of the energy is used and therefore the upfront cost is easily worth it.
Here is the full video they put together on the analysis:
I completely realize that there are hybrids out there MUCH less environmentally friendly than the Prius as Jim-Bob has mentioned. This is just the info I have at hand.