It's not really the m2ts container that causes a file to transcode, it's the stream that's within the container.
For an example, if the audio stream uses a DTS-MA codec or some other high quality codec, then it will transcode so that it will play on your ps3 properly. If the the video is encoded at a higher level than L4.2, then it will not play. If the max bitrate is higher than 35,000mbps, then it may not play.
Use MediaInfo and see what the actual stream content is encoded in and determine what needs to be re-encoded in order for it to play on the ps3. It's most likely just the audio that is causing it to use mencoder, since most video is encoded @ 4.2 or less.
If that turns out to be the case, then you need to re-encode the audio. There's several programs you can use to do that, but here's what I normally do when I encounter high quality audio:
- Determine the current audio codec using MediaInfo
- Demux (extract) an exact copy of the audio using HdBrStreamExtractor and selecting the same codec I found with MediaInfo. I do this generally just to save a copy of the original audio stream in case I ever want to restore it.
- Demux (extract) the audio again, except this time I extract it as AC3. This will re-encode the audio to a format that the ps3 can read.
- Use TSMuxerGUI to mux (combine) the AC3 audio and the original video stream back together.
~5 minute process.
If it's the video, then there's no way around re-encoding the video, which takes a lot longer.