Thanks everyone for your replies,
Yes, after a bit of internet searching, yes it was a VC-1 encoded M2TS file. Also, yes it was being streamed from my desktop via PS3MS to my PS3 downstairs, so its not on a USB drive or copied to the PS3 HDD. I believe I have the latest firmware, and it still didn't like the VC-1 encoded M2TS - however if a firmware update changes, then please let me know, it would be great to not have to re-encode a M2TS file just from VC-1 to AVCHD format.
Additionally I found out that the M2TS streaming doesn't like DTS-MA audio, I had to re-mux that to DD AC3 5.1 for the M2TS file to play without transcoding.
So after some tinkering (and too long to wait for the encoding process!) here are my personal settings for good quality BDRips for streaming via PS3MS;
Using RipBot264, I transcode the original M2TS file using;
[Level 4.0] HD.BluRay.Consoles
CQ=18 (18 is nearly lossless and gives equivalent size as original, 20 is a good and I can't see a difference on 100" screen, 22 is acceptable if trying to save space)
Audio Profile Dolby Digital AC3 5.1 (I have my PS3 audio go through HDMI which doesn't stream DTS-MA or DDTHD - even though my amp supports decoding)
AVCHD (then delete the resulting BD file structure and copy and rename the Stream\MT2TS file to the title of the movie)
If PS3 supports streaming of DTS-MA or TrueHD please let me know.