if I plug in a keyboard with a semi failed USB controller the latencies jump to 5k every few seconds

Looks like it at least somewhat works.