the 7805 should not be even warm. I don't know if that is a clue. Were you able to run blinky without the buttons and lcd attached?
Also, the serial board (and/or my choice in pins) doesn't like to load with everything powered, I have to disconnect the power to the lcd/component board when I upload (pull off orange wire circled in red in bottom picture below). The usb one doesn't exibit this behavior for some yet unexplained reason.
