Here's a howto for Avisynth, ffdshow and coravc from me: http://otmanix.de/2009/04/19/java-ps3-media-server-fuer-dummies-teil-5-videolan-und-avisynth/
Sorry, only available in German language...
here's a short description from PMS readme:
B) Using Avisynth/FFDshow. Paired with CoreAVC, it's quite efficient with multi cores.
1- If not already done, get and install a codec pack with ffdshow (VSfilter must be included for subtitles).
My favorite is CCCP Codec Pack [http://www.cccp-project.net/download.php?type=cccp
2- Get and install AVISynth 2.5.7 (I've met some framerate problems with 2.5.8 betas)
3- Set up the FFDShow Audio Mixer to 5.1 (3/0/2 LFE)
4- Set "Autoload VS Filter" to true
5- Configure Haali Splitter (audio/subtitles language priorities)
Optional Codecs: CoreAVC is a good choice for H264 decoding, Quicktime Alternative and Real Alternative are also useful (Apple 1080p trailers for example).
You can enforce transcoding in two ways:
a) set avisynth/mencoder as default video engine, save, restart PMS
b) go to virtual folder #transcoded# in your PS3 menu, select file and transcoding engine avisynth/mencoder