A guide to make decode with coreavc for mencoder, works for me very nicely:
1- Grab the latest copy mencoder_mt package from http://oss.netfarm.it/mplayer-win32.php
2- Extract archive, rename mencoder.exe to mencoder_mt.exe, copy this file and dsnative.dll to C:\program files\ps3 media server\win32
4- Install coreavc. ($10 to buy)
3- Copy C:\Program Files\CoreCodec\CoreAVC Professional Edition\CoreAVCDecoder.ax to C:\program files\ps3 media server\win32\codecs
4- Open Transcoding Settings tab on the PS3 Media Server
5- Video Files Engines->Mencoder->Expert Settings: ....
6- Write to custom parameters textedit area "vcodec == h264 :: -noass -mc 0.005 -vc coreavc -mt" without quotes.
Should work with non-mt builds.
Edit: Oh, BTW, it really helps to make faster transcodes.
Anyone knows quality performance of coreavc? I've heard it's a little bit lower than the ffmpeg's decoder. Is that true?