I have done some extensive command line testing with MEncoder and movies with (embedded and external) subtitles on Mac OSX.
The culprit turns out to be fontconfig, which does not manage to find any fonts. As a result MEncoder could not find any font to render the subs with. On top of that: if MEncoder is compiled with fontconfig support, the "-font" flag should provide a font name, not a file path as PMS did.
I
committed a fix that will always supply a font file path to the MEncoder command line for OSX, plus I
updated MEncoder to a version compiled (for OSX 10.7+) without fontconfig so it will accept that font file path.
All subs in my test files show up now, even Hebrew and Arab subs! The one exception are Japanese subs, which show as blocks. Before the fix these didn't show at all, not much of an improvement there.
There still is room for improvement: ASS/SSA subs will now only use the default font. Some subs may look different, but at least they show.
I hope this solves the long running problems with broken subs on Mac OSX.