Newhaven: if passers executed their maneuvers smoothly it wouldn't be so bad. Many don't. As noted, even on a 4-lane people will come barrelling up behind you and at the last second either slam on the brakes or swerve around, even if the left lane is clear. Why? I think cell phones have a lot to do with it.
Even, as you describe, if on a 4-lane people "slow", they only do it next to and behind you; then they accel back up to where they were.