I think we are in the equivalent of... say closed-loop vs. open loop in a gasser. Or maybe between hitting the "power" button on an automatic.
Maybe there is a way to trigger direct (or less laggy) control when needed, I think that's one of the ideas you were having Paul. Or maybe make it user switchable, choose one or the other. Nice smooth everything or jerky movements from jerky pedal. That way they can't complain and just live with whatever choice they make... until they switch it again.