As shagrath said, there is no way to know for sure that the PS3 will play a file 100%. Therefore i would go the other way.
Check the file for QPEL -> Transcode per default as it will never play
Check the file for GMC -> Transcode, see above.
Check for some custom matrix -> Transcode...
Maybe some muxer types or settings will also conflict with the PS3, maybe these could then be remuxed instead of transcoded!?
Just add rules from experience and feedback that will 100% prevent the PS3 from playing that file and then transcode...if a rule is missing or impossible to implement (like the example from shagrath) then you would have to manually go to the #transcode# folder.