I'm not entirely sure what it is that you want to achieve.
Transcoding options can be edited in the corresponding tab in PMS, e.g. open the tab, then click on "MEncoder" to see its specific options. Under the "Help" tab you will find an explanation of each option.
As for the "# Transcoding #" folder, that allows you to handpick the way you want a specific movie to be transcoded. E.g. choose "[FFmpeg]" if you want to see a specific movie transcoded with FFmpeg instead of the default MEncoder.
Does this help?