Everyone that have problem with gray screen of death please go read that link:
GSOD explained
Me also at the beginning i had some gray screen of death, took me a long time to know what it was, someday i ran memtest and found out i had one doa memory stick. It was causing me all the problem i had. Got some better memory and i also changed my mobo ( didnt liked my original p55m ud2 so i got a p55 gd65) And since then even with 9.11 driver i never had a single gsod even with some extreme setting to get into top10 on hwbot for vanatage performance hd5770x1. So make sure everything in your overclocking is stable. Make sure you dont have some bad memory stick. Then if you still have those gsod then there is a possibilities it is some faulty memory chip on your gpu. But first off, check in the windows event journal to see what is the error that make your gsod appears. GSOD for hd5xxx is the new BSOD. So really try to rule out everything else, if you had then contact your manufacturer for a rma cause it is possible the memory chip can be bad.
That was just a resume of what was mostly said in the link.
Also, sadly the new hd5xxx have already high memory clock so using even higher memory clock can cause big instability. Even at 1200mhz memory is not a bottle neck on the hd5xxx card. You only need it for 3dmark.