ATIs drivers sometimes have issues with two DIMMs per channel being installed. Remove one DIMM from each channel and try again. I've had a BSOD referencing a SYSTEM_SERVICE_EXCEPTION and atikmdag.sys myself, but a hang is also possible. If the memory reduction doesn't work, try moving the video card to a different slot.