I'm still little lost how you are shifting too, course I don't own a burgman to have any point of reference either, but did see some pics on pages 63, 64 (and 32 to get my bearings) in this manual.
Suzuki Burgman AN400 2003
Guessing that you brought the idler shaft end out so you could twist or push on it, but no shot of the internal gears to make sense out of it.