the x4 v x8 might only matter for burst speeds from cache. the top of x4 may be lower, however, if you have items in your cache that is what cache is for. faster reading from cached items.
so it DOES affect your card to go to the slower bus speed. everyone always downs cache but during usage there are gonna be periods of time where info comes straight from cache to your system. that is why they have it.
so if it affects your cache transfer speed....it affects your speed!
regardless of whether or not the info comes straight from the disk it is still info. going to your system. it is strangling your cache usage. thus your card is not operating at its highest intended usage.

however, it will not apply to all situations, probably a small percentage of use. depends upon your system usage.