What stick do you have in the 1st slot? If it's one of your good clocking ram sticks, it may have something to do with the SPD of that kit, as all 4 ram slots settings are set by the SPD of the stick in the 1st slot, other than the settings you have set in bios.

It doesn't take more vdimm to run 4 sticks as opposed to 2 (unless the ram you add needs more vdimm for the same clocks as the ram already in the board), but it will take more vNB as it's harder on the MCH.