Sorry I can't help you, as I've never downloaded/used a XBOX 360 encoded movie on the PS3. I think my PS3 would need a bath after an experience like that.
But rather than just crack wise all day, let me try to help (even though I've got no experience in this). When you add extensions to "force transcode" make sure a couple things;
1) use the latest beta: http://ps3mediaserver.org/files/pms-mac ... a-r350.dmg
2) in the force transcode field, the syntax is 3 digit file extension with comma separation, no periods. i.e.; mp4,mv4,avi,wmv,mkv
3) After you alter any settings in PMS, always hit "save" and (especially for the latest betas) quit the app and relaunch for it to take effect. The latest betas don't seem to "restart http server" correctly.
Also make sure the #transcode# folders are not disabled, and you can try navigating into the #transcode# folder that relates to your movie, and drill down into it that transcode folder until you see the movie name in there. It usually is set up to use the mencoder engine, try launching it from there.