Check the discussion and code line 1612: https://github.com/ps3mediaserver/ps3me ... 9c220bdfe1
New Subjunk maybe uses expand instead of scale but anyway this code can be in conflict with overscan or your custom mencoder options.
I don't remember the code how overscan is handeled but I suppose when you will use overscan and also custom "expand" command in MEncoder parameters, maybe conflict arise.
Check the debug log with overscan enabled and disabled and provide the log. Also specify what custom MEncoder params are you using in renderer.conf or in PMS Gui(pms.conf)