It's not the spoiler. Mine does it too.
There's a reason they came stock with a rear wiper.
It's just the shape - the vortices created at the back of the "brick" continually swirl dirt and dust around the rear window.
I'm sure the spoiler provides a bit more vortex control back there. I'd like to see a radical rear-end mod with side wings and/or a central fin to smooth things out back there. I'm not going to do it though, because it's my only nice car. The other two are fair game for crazy mods.