MEncoder doesn't support DTS HD MA, it can extract DTS core only. If you don't have DTS enabled then it is transcoded to AC3(Dolby Digital 5.1, by default in 640Kbits quality)
PS3 doesn't support DTS HD MA passthrough on fat PS3 and what I remember it is supported on slim PS3 only in case it is played from BluRay media and not from DLAN/UPnP.
Anyway DTS-HD MA decoder doesn't exist in MEncoder so only core can be extracted. Because of that PMS transcodes audio to AC3 from DTS core (if DTS is enabled in PMS GUI, you should see PCM 2.0 on PS3 and DTS core on external receiver)