Otherwise having a large phase separator gas volume should not be an issue, and if anything, aids in separating the liquid from the gas.
- Insufficient liquid feeding the evaporator
- Excessive system pressure drop (hard to say without downstream pressure readings)
- You need a bigger compressor
If the evaporator HX had poor heat transfer as you suggest, then I would expect to see the feed and return on the primary side to be very close in temperature (very little gradient). the fact that the temperature gradient is large, suggests that very good heat transfer is taking place (assuming that you are not simply running out of refrigerant). Personally I think you still need more low boiling refrigerant flowing in the system. And judging by the compressor discharge temperature of 108 C, you might need a tad bit more butane (you might find that mixing a bit of n-Butane with the iso-Butane would also help in this area). Large systems usually require either a high static balance pressure (225+ psig) or sufficiently sized expansion volume, in order to provide enough low boiling refrigerant when condensed.