I have the same issue with flac files. The "skip transcode" option doesn't seem quite right. On the PS3, files are still listed and identified as PCM (I think, not in front of it right now...), but are not recognized as unsupported until you try to play one. They also do not play correctly on my Denon receiver, which supports flac.
Using Mediatomb, if I disable transcoding, the files are correctly recognized right off by the PS3 as being unsupported. They also play correctly on the receiver.
This is on Ubuntu 8.04, using the medibuntu packages for mplayer, mencoder, and ffmpeg.
Outstanding piece of software, btw. Tried several others, really dig this one.