Had the same issue, same setup running a debian 64bit container. The thing is, if you check the host memory, its not actually using the full stack of memory when looking at the host node. You can tweak the allocation by modifying your PMS.sh. The variable -Xmx768M controls the heapsize the jvm tries to allocate.
What I found is that you HAVE to also set a video buffer in your PMS.conf file to lower the heap value. I am running with:
exec java -Xms256M -Xmx512M -Djava.encoding=UTF-8 -Djava.net.preferIPv4Stack=true -jar pms.jar net.pms.PMS
Seems to be pretty stable, it streams unconverted 1080p with no issues, and seems to be able to convery 720p mkv without any hiccups. Let me know if you need beancounters or something like that.