Network throughput

[Raptor399]

i think the buffered output is one of the most important features.

I respectfully disagree. IMHO the buffered output is one of the most overrated and overengineered features.

Since this is a rather bold statement coming from someone relatively new to the code like me, let me explain why.

Looking at it from a high level, why would PMS even bother to buffer output?

The client knows its own limitations best. You can bet that whatever player you are using, it will have some kind of buffering solution that makes sure you get the best playback experience. Having another buffer (like PMS) between the source and the destination will not improve playback.

Compare streaming; when PMS is streaming, it does so in 8192 byte chunks with no extra buffer code whatsoever. This works like a charm on every client, and we don't get complaints like streamed content missing the last few seconds or sound being garbled, nor is skipping or stuttering a problem or is streaming too slow (give or take the odd network problem). Your best experiences with PMS are from streamed content. If you would transcode files manually to another file, then stream the result using PMS, all problems would be gone.

PMS is very good at matching clients with what they can play, and transcoding source material based on that. Just examine the debug output and you'll be very happy that PMS takes all that hard work out of your hands.

PMS is also good at connecting to multiple clients and allowing them to browse the available media.

What PMS is not good at IMHO, is buffering transcoded data and sending it to a client.

I've been looking at the code for weeks now, trying to figure out what it is that makes it break. Or simply trying to understand what it wants to do and how.

This morning I woke up and it dawned upon me: "Why bother with buffering at all?"
If streaming a transcoded file works, then why not simply stream the piped content from the transcoder without any buffering or rewriting attempt whatsoever? Sure, the bytes will come in slower than if it were a file, but the client will probably just wait till its own buffer is filled and then play.

I'm probably oversimplifying here. Examine BufferedOutputFile and you'll see there is *a lot* of extra code to shift audio and video, compensate for WDTV etc.

Still, I am convinced that avoiding the use of BufferedOutputFile in OutputBufferConsumer or ProcessWrapperImpl (line 131) would solve a lot of the transcoding problems. It would probably solve more problems than the extra code currently fixes. And maybe the extra code could be piggybacked onto the new (almost) bufferless solution once that is proven to work.

Unfortunately my reasoning exceeds my coding skills.
I cannot figure out which OutputStream to couple the InputStream of the process to. Doh!

Is there some Java guru out there who is daring enough to attack this problem?

[ExSport]

If streaming a transcoded file works, then why not simply stream the piped content from the transcoder without any buffering or rewriting attempt whatsoever? Sure, the bytes will come in slower than if it were a file, but the client will probably just wait till its own buffer is filled and then play.

In most cases people are concerned that in high spikes,complicated scenes,that playback will start to shutter because there will not be enough data already transcoded/buffered in memory.
Also that FWD will be worse because no data will be prebufferred....

[Raptor399]

In most cases people are concerned that in high spikes,complicated scenes,that playback will start to shutter because there will not be enough data already transcoded/buffered in memory.
Also that FWD will be worse because no data will be prebufferred....

Even then it may be better if PMS doesn't meddle with the process. After all, PMS does not know anything about the performance of the transcoding process, nor does it know anything about the playback performance. PMS has no way of determining whether problems will occur on either end. So what you end up with is a mandatory pause at the start of a transcoded movie for an arbitrary amount of time, in the hope that it will fix possible stutters later on. Which sucks if you have plenty of performance on your hands, or if you lack performance and problems occur anyway but half an hour later.

[DeFlanko]

Is there some Java guru out there who is daring enough to attack this problem?

i might know someone. <.<

[mazey]

i wouldnt mind trying it out with no buffer if it improves it but if it doesnt then i think we should fall back to a buffer since its obviously worked good for a long time.

[Raptor399]

Is there some Java guru out there who is daring enough to attack this problem?

i might know someone. <.<

No need, thanks. Looks like I managed to get it working anyway.
The code needed a bit of refactoring and in r804 I added an implementation with a radically different approach: UnbufferedOutputFile.

It works, but like many solutions it opens up a can of worms. In this case removing the seek option means it'll need to be compensated for somewhere else in the code.

[jochem]

Hi Raptor399,

After studying and changing the buffer code for low memory usage (see viewtopic.php?f=6&t=12071) I sort of agree with you on the buffer code. Especially the magic timings (and thread lockups if you fiddle with the also hardcoded sizes) worry me a bit. But maybe I am wrong, even after some studying I still do not totally understand the buffer code...

I looked at your UnbufferedOutputFile, but it did not manage to get it working. The write function on the pipe did not work, because the pipe was already closed by the time it should be written. Any clues? Are you on Linux also?

[Raptor399]

Hi Raptor399,

After studying and changing the buffer code for low memory usage (see viewtopic.php?f=6&t=12071) I sort of agree with you on the buffer code. Especially the magic timings (and thread lockups if you fiddle with the also hardcoded sizes) worry me a bit. But maybe I am wrong, even after some studying I still do not totally understand the buffer code...

I looked at your UnbufferedOutputFile, but it did not manage to get it working. The write function on the pipe did not work, because the pipe was already closed by the time it should be written. Any clues? Are you on Linux also?

Ah, I saw your low memory usage thread and I wanted to link this one, but I see you've found it already.

Hehehe... I've broken my head many a night on that buffering code and I still don't get it.

What didn't you get to work?
As I recall, replacing is fairly straightforward and documented in UnbufferedOutputFile.java.

Oh, and I'm on OSX.

[jochem]

As I recall, replacing is fairly straightforward and documented in UnbufferedOutputFile.java.
Oh, and I'm on OSX.

Yes, replacing the standard implementation with your Unbuffered variant is easy enough if you bother to read the source code . It compiled and started well.

But as soon as I started using it, it gives an IOException in the 'write' method(s). The exception message is 'Pipe closed'. So somehow Linux decides to close the pipe. I guess this is an OS dependent problem, so therefore I asked about your OS. Maybe it depends on certain timeout values. But I did not dive deeper in it than this. Any clues on how to easily debug this further?

[Raptor399]

An update that has been long in the works:

In my no-buffer2 branch, you will find a version of PMS that uses a 4 KB (as opposed to 600 MB! ) buffer, provided you disable http v2.

It still has quirks that need solving (sometimes a stream won't work, but when you retry it will), but it is functional enough to prove that buffer size is irrelevant.
It will also make painfully obvious how good your CPU is at transcoding since there will be barely any lag between the transcoding process and the renderer.