OK, I uninstalled 1.54, and installed 1.81 from fresh. Tested that version and sound is screwed!
I copied out the Win32 folder from under PS3 Media Server, then un-installed 1.81, reinstalled 1.54 from scratch - tested that version again, and sound works fine on all channels. I then overwrote the Win32 folder with the version taken from 1.81. Sound is still fine! So its not MEncoder which is the source of the issue, but how PS3MS uses Mencoder in versions of PMS newer than 1.54!
Would be great if the devs could get to the bottom of this one. Its very easy to reproduce - just get hold of a copy of any (EMERSE or EVOLVE) Game of thrones MKV from Season 3 with 720p and 5.1ch sound. Install 1.81 with default options and try playing it on a PS3! I actually found same issues with the S2 episodes I tried to play.