I think you have a problem with contact now that the IHS is gone. Use less armaflex or whatever you use for insulation...
All my GTX580 are without IHS and I benched 3D Mark 2011 at MOA ~1450MHz with 1.48v without any problem. Smoke and slamms did the same also without IHS... so I don't think there's a problem with IHS-less cards

P.S: After IHS removal, it's very important to put some nail polish or vaseline on the little SMDs that are near the core. Otherwise, when you will have some water there the GPU will DIE instantly