What is transcoding/muxing? http://otmanix.de/english/2009/05/13/java-ps3-media-server-for-dummies-chapter-1-introduction/
Can one force muxing?
You can enforce muxing by choosing tsmuxer engine in XMB. In XMB choose video menu, select PMS, goto folder which contains video, select virtual folder #-- transcode --#, select video file, select tsmuxer
Would this be desirable?
Yes, because you get the original audio and video stream in unchanged quality. If your media file contains not supported video codec PMS has to transcode the video stream to MPG2...
How does one know if he's muxing or not?
First of all check if automatic muxing is enabled:
video files engines: MEncoder > switch to tsmuxer when video when h.264 video is PS3 compatible
Enable this! When MEncoder transcoding engine is used and PMS recognizes a PS3 compatible video it will switch to transcoding engine tsmuxer. The video will then be streamed directly or muxed but not transcoded! This setting will dramatically reduce the need of transcoding. Matroska video (mkv) for example aren’t supported on PS3. Mostly the contained audio and video stream is PS3 compatible. So tsmuxer creates a PS3 compatible m2ts container which contains the original audio and video stream and the PS3 is able to playback. This feature reduces both network traffic (because of better compression as MPG2) and CPU power needed for playback.
When you playback a video file, e.g. a h.264 HD-video, press select on your PS3 controller. If MPG2 is displayed for video codec PMS is transcoding, otherwise AVC is displayed which means muxing.
There are some parameters for muxing/transcoding audio stream which depend on your hardware. If you have an PS3 and A/V receiver capable of decoding DD/DTS you may want to change certain settings. Please read chapter "common transcoding settings": http://ps3mediaserver.org/forum/viewtopic.php?f=6&t=3507
and a last comment: subtitles enforce transcoding...