Comments on: Notes about low latency video http://www.schleef.org/blog/2009/03/27/notes-about-low-latency-video/ Just another WordPress weblog Mon, 12 Dec 2011 22:37:25 +0000 hourly 1 http://wordpress.org/?v=3.1 By: Gregory Maxwell http://www.schleef.org/blog/2009/03/27/notes-about-low-latency-video/comment-page-1/#comment-2395 Gregory Maxwell Tue, 07 Apr 2009 20:22:21 +0000 http://www.schleef.org/blog/2009/03/27/notes-about-low-latency-video/#comment-2395 If someone is looking for a fun project— <a href="http://www3.elphel.com/" rel="nofollow">Elphel</a> makes a line of fully open-hardware video cameras. They use a CMOS sensor with an electronic rolling shutter. The image data is streamed into a FPGA, where on-chip demosasicing and jpeg compression is performed. The verilog for the FPGA is available. If someone was interested they could have these fairly inexpensive cameras producing output with a few scanlines latency with some development work. If someone is looking for a fun project— Elphel makes a line of fully open-hardware video cameras. They use a CMOS sensor with an electronic rolling shutter. The image data is streamed into a FPGA, where on-chip demosasicing and jpeg compression is performed. The verilog for the FPGA is available.

If someone was interested they could have these fairly inexpensive cameras producing output with a few scanlines latency with some development work.

]]> By: Jean-Marc Valin http://www.schleef.org/blog/2009/03/27/notes-about-low-latency-video/comment-page-1/#comment-2392 Jean-Marc Valin Thu, 02 Apr 2009 00:04:40 +0000 http://www.schleef.org/blog/2009/03/27/notes-about-low-latency-video/#comment-2392 Being involved in low-delay audio transmission (through CELT and other projects), I also got interested in low-delay video transmission. After all, it's no use having a conference with a low-delay audio codec if your video path has 200 ms delay (either audio and video are out of sync or you have to delay the audio). It turns out that systems that work on whole frames are basically hopeless for low-delay video. In your example above of 60 ms encoder-decoder delay, the problem is that there's a lot of other delays that need to be added. Even a "perfect" one-frame-at-a-time camera will have 30 ms delay, then the display side will have another 30 ms at least. Then you'll have another ~30 ms for the network buffer. At last, the general idea of compressing is to have a size that's close to your network bandwidth. When that's the case, then the time it takes to transfer an entire frame is also 30 ms. Add to that the speed of light, maybe 10 ms if you're close, and you get a total of *at least* 160 ms. In practice, it'll be very hard to go below 200 ms with that kind of approach. So not only do we need input and output hardware that works on scan lines, but we also need codecs that can handle that. As far as I understand (I could be wrong), the Theora bit-stream requires the encoder to have seen the entire frame before it can output anything. That's a fundamental limitation that Dirac does not have. The standard version of Dirac will still have a delay of several lines because of the vertical wavelet transform, but not that bad. IIRC Dirac Pro can use a simpler Haar transform that adds less delay. Being involved in low-delay audio transmission (through CELT and other projects), I also got interested in low-delay video transmission. After all, it’s no use having a conference with a low-delay audio codec if your video path has 200 ms delay (either audio and video are out of sync or you have to delay the audio). It turns out that systems that work on whole frames are basically hopeless for low-delay video. In your example above of 60 ms encoder-decoder delay, the problem is that there’s a lot of other delays that need to be added. Even a “perfect” one-frame-at-a-time camera will have 30 ms delay, then the display side will have another 30 ms at least. Then you’ll have another ~30 ms for the network buffer. At last, the general idea of compressing is to have a size that’s close to your network bandwidth. When that’s the case, then the time it takes to transfer an entire frame is also 30 ms. Add to that the speed of light, maybe 10 ms if you’re close, and you get a total of *at least* 160 ms. In practice, it’ll be very hard to go below 200 ms with that kind of approach.

So not only do we need input and output hardware that works on scan lines, but we also need codecs that can handle that. As far as I understand (I could be wrong), the Theora bit-stream requires the encoder to have seen the entire frame before it can output anything. That’s a fundamental limitation that Dirac does not have. The standard version of Dirac will still have a delay of several lines because of the vertical wavelet transform, but not that bad. IIRC Dirac Pro can use a simpler Haar transform that adds less delay.

]]> By: romulo http://www.schleef.org/blog/2009/03/27/notes-about-low-latency-video/comment-page-1/#comment-2387 romulo Sun, 29 Mar 2009 15:29:02 +0000 http://www.schleef.org/blog/2009/03/27/notes-about-low-latency-video/#comment-2387 The guys at OnLive conference didn't mean 1ms latency on the video compression. What they said is that since players "clients" run on the same computer, they will see no lag between themselves. Its not about video streaming, its about how players interact with each other inside the server. The guys at OnLive conference didn’t mean 1ms latency on the video compression. What they said is that since players “clients” run on the same computer, they will see no lag between themselves. Its not about video streaming, its about how players interact with each other inside the server.

]]>