Just an update, I have VLC working with 95% of the random assortment of video's I threw at it (all of which were in different formats). This is already significantly more than I ever got working with ffmpeg/mencoder.
My repo for those who are interested: https://github.com/LordQuackstar/ps3mediaserver
I think the issue that previous attempts have had is that VLC transcoding hasn't really been all that good in the 0.x series and 1.x series. However with VLC 2.0.2 it is very robust, handling even the bug someone detailed in net.pms.encoders.VideoLanVideoStreaming (which BTW would of been nice to know existed before I wasted 2 days trying to figure out the API. My fault).
I have essentially duplicated the VideoLanVideoStreaming class in net.pms.encoders.VLCVideo and did various modifications to it so that it would work. As I couldn't figure out the encoder priority in code or configuration I had to disable all other encoders and push VLC trans to the top.
- One of my video's caused VLC to core dump. I've reported it on their forums here: http://forum.videolan.org/viewtopic.php?f=13&t=102593
- I've only tested the modifications I finished 10 minuites ago on a PS3. I'll be testing on an Xbox in a little bit. I have also only been testing on Fedora Linux. Windows testing will be later
- I do not know PMS's api all that well, so I have no idea what happens when the buffer fills. Will PMS kill VLC and restart it? Will the bufer just fill continuously?
- Yes the GUI looks bad. Yes this is temporary. This is my 3rd attempt to learn the JGoodies Form syntax, and it wasn't that successful again. I will be trying later, but I really just wanted to focus on the more important transcoding and less important gui.
- The policy with VLC versions that I will be using is this: Only the most recent version of VLC will be supported. This is currently VLC 2.0.2. I've even put in the PMS init method code that warns about using versions less than that. Reasoning: Old versions are broken, have bugs, caused headaches, etc. They have also ruined people's opionions of VLC. If we stick to the latest and greatest it should limit the amount of bug reports and complaining "vlc sux, use a real encoder".
If anyone has some time it would be really helpful to do testing on your own machine or just do a basic code review. I do not know PMS's internals that well so I tried my best. But I've probably done something wrong, so any critisism is welcome.