on BO cooling the nb is important, make sure it stays cool enough so you can touch it for seconds without burning your hand and its good enough
then increase nb volts, cause higher nb volts will let you clock mem higher too
so do that first, and then increase mem volts if you still cant get it stable.

some mem is really sensitive about higher vdimm, idk what mem chips your sticks are using... try to find out and check if they are ok at high volts.
make sure they stay cool as well... and increase trfc and tras, those help with large amounts of memory...