my PS3 Media Server runs on a G4 Mac mini. This is much too slow for real-time transcoding with mencoder, and the G4 machine won't run current tsMuxeR builds. So I've hacked up a small script that will transcode everything that needs transcoding and link the rest. The script acts like a master-slave synchronizer, i.e. you manage your video files in one directory hierarchy (SRCDIR) and the script manages a second directory hierarchy (DSTDIR) for you. Everything you put underneath SRCDIR will be transcoded or linked underneath DSTDIR, everything you delete underneath SRCDIR will be deleted from underneath DSTDIR. You serve up DSTDIR to the PS3. Currently I run the script manually, but it would be trivial to have it run every 5 minutes to check if anything has changed below SRCDIR.
Transcoding / linking happens through shell commands, and the script is extendable for arbitrary file types (currently it knows VOB, AVI and MKV).
I've put it up here: http://giesler.biz/~bjoern/blog/?p=196. I'm not really supporting this thing but I'd like to hear from anyone who finds it useful.