Transcoding is already set up by default in the PMS settings, although you may need to tweak it so it performs within the capabilities of your PC/network (read Max Bandwidth and Video Quality settings).
When you're browsing through the list of files on the PS3, there should be a folder at the top of each directory labeled #Transcode#. Go into that folder and you'll see a relisting of your files as folders. Go into a folder and you'll see the transcode options available for that file. It'll make more sense when you actually see it.
I provide NO application support via PM or email, so please post your question to the forum.