The extra step of unplugging the 5v on the serial duino to program it, and the prospect of a lot of trial and error to get this working right, combined to motivate me to put the 4x20 lcd together with the usb duino.
The 4x20 lcd seems to be working without modification to the code (same pinout as 2x16 too):
Here is the backside with just buttons and lcd hooked up:
