Replace keyboard, I have seen USB holding up the boot like this.

Try with just keyboard attached too to make sure its not the mouse.