So next steps where:
I installed P3 Mediaserver on my quadcore. Transcoding via MEncoder worked fine. Global CPU Usage about 3%
OK that ist not much and should work with my Pentium M.
My quadcore PC (to much powerconsumption as server) could only transcode via MEncoder because there was no codec-pakage installed on it.
So, I thougt it may be a codecproblem with my homeserver an not a performance problem.
therefore I installed K-lite codec, Mplayer and more.
Than tried again. MEncoder transcodings still does not work
But now AviSynth/FFmpeg transcoding workes now !
(at 100% CPU usage)
But still i would be very happy if someone may got a solution für my MEncoder problem. Every transcoding with Mencoder included seems ot to work.