hummm, actually, mplayer suports idx/subs in OS X (-vobsub) so piping mplayer's output to mencoder should make this possible...
- EDIT -
Yup, should be possible. Take a look at this example:http://lists.mplayerhq.hu/pipermail/mpl ... 76699.html
Haven't tested it.
Anyway, using mkfifo instead of temp files could be even better.