The Linear Systems cards are probably better known by their reseller, DVEO. Entropy Wave uses both of these cards in the E1000 Live Encoder appliance, we’ve found that aside from some motherboard incompatibilities in the DeckLink cards, they both work great in Linux. While we’re primarily interested in live capture at the moment, output has also been implemented.

We slightly prefer the Linear Systems cards – mainly because the drivers are open source, but also because the API allows lower level access to the hardware, including SDI clocking and raw VANC and HANC data. It also allows subframe latency, although not implemented in the GStreamer plugin, it will be nice to use in the future.

In comparison, the DeckLink driver and SDK are not open source (which means I can’t fix any bugs), although they conveniently provide open source headers and shim code for interfacing with the SDK. This allows the GStreamer plugin to be completely open source and legally distributable separately from the SDK, but will only work if the SDK libraries and driver are present. Optical fiber connections are only available in the DeckLink, and the DeckLink cards tend to be less expensive.

It will take a few weeks for these to be available as part of a GStreamer release, however, they are available in the Media SDK now.

(Reposted from my Entropy Wave blog.)

The Open Video Conference is happening again, and coming up in a few short weeks (October 1-2).  It’s a great mixture of people from both sides of open video: open content and open technology.  I’ll be giving a tutorial that ties the two sides together, explaining to content producers how to use the technology to its fullest.

In association with OVC is the Foundations of Open Media workshop (October 3-4), which is mostly a bunch of video and media hackers getting together to talk about our next steps in taking over the world.

Partly the reason for the long release cycle is that it took more time than expected merging several of the encoding tools from dirac-research.  But now there are fewer loose threads and development and releases can proceed at a more even pace.  I’m hoping to do new releases at the pace of about one a month.  (But I’ve said that before…)

Schrödinger now requires Orc to build.  Switching from liboil to Orc has made decoding a lot faster, sometimes as much as 4 times faster.

Look forward to separate blog posts about some of the new features.  Encoding quality has improved quite a bit for typical cases, and hugely in cases where there were bugs that were being ignored.

An Ogg/Theora video of Big Buck Bunny being played back on a Beagle Board via the C64x+ DSP coprocessor

The goal behind porting to the C64x+ is to run on OMAP3 SoC from TI, which has an ARM Cortex A8 core and also has a C64x+ DSP coprocessor. This SoC (System on Chip) is best known as being the base behind Nokia’s N series of mobiles (including the N900), the Motorola Droid, Palm Pre, and the Beagle Board. The DSP coprocessor is commonly used for audo and video processing, including video encoding and decoding, and TI makes codecs available for MPEG-4 video decoding, AAC decoding, etc.  Having Theora decoded on the DSP fits into Mozilla’s Fennec project, making Firefox with video useful on a mobile platform.

One of the engineering reasons behind having a separate processor for media handling is that it separates real-time tasks (media decoding) from non-real-time tasks, such as running web browser software. From the standpoint of software running on the ARM, the video decoder looks and acts just like a hardware video codec. The DSP on the OMAP3 is even more compelling for video decoding because attached to the DSP are several units that accelerate motion vector copying, VLC decoding, and loop deblocking. Unfortunately, these pieces are not publicly documented by TI, so the current Theora port (which is open source) is unable to use them. A future Entropy Wave project will likely add support for these acceleration units which would allow the performance of the Theora decoder to be similar to TI’s MPEG-4 codec, which can do 800×480 playback (possibly more?). As it looks now, the resulting code would necessarily be closed source until such a time when TI wishes to make the specifications public.

As it currently stands, the Theora decoder plays 640×360 24fps at slightly more than 100% speed on average. This isn’t quite good enough to call it “real time”, since some frames take longer than the allotted time to decode, but it’s pretty close and the results are good. Additional speed improvements in libtheora would require internal changes, which would be a project in itself. One clear area for improvement is that the DSP spends a substantial part of its time idle, because the host code is serialized with the DSP processing. Fixing this is likely to put the above case firmly into the “real time” category. Given that 640×360 is larger than the iPhone display resolution and almost as large as the N900 resolution, it’s clearly good enough, even if it is less than TI’s hardware accelerated MPEG-4.

On the Entropy Wave site is a page describing the demo, including where to download images and how to compile source code.

A big thanks to the people that laid the foundations for this work, especially Felipe Contreras.

Firefox does the conversion correctly, so it’s unlikely you’ll see the pattern. However, some video display drivers still get this wrong, so you might see the pattern when playing the video in a standalone program that uses XV. For those of you with working kit, I created a demonstration video that simulates a bad conversion:

Sometimes it’s possible to see the pattern very faintly due to rounding in even a correct conversion. This is unavoidable because the RGB->YCbCr->RGB round trip is lossy.

One such application that is covered is chroma subsampling and color matrixing for video, semi-incorrectly referred to as “colorspace conversion” in GStreamer.  There has been a colorspace element in Cog (cogcolorspace) for some time, but I never really bothered to do any speed comparisons between it and the default GStreamer colorspace element (ffmpegcolorspace), which is based on code copied from FFMpeg.  However, recently I did, and was somewhat surprised (although I shouldn’t have been) that cogcolorspace is the same speed as, or much faster than, ffmpegcolorspace for almost all operations.  (Please note that the FFMpeg code was forked a long time ago and heavily modified, so it does not reflect FFMpeg itself, only GStreamer’s ffmpegcolorspace.)

This is a scatter plot of the run time (in ms) for converting 1000 frames of 320×240 video between a variety of uncompressed video formats:

The axes are execution time (in ms), with cogcolorspace on the horizontal axis and ffmpegcolorspace on the vertical axis.  The green line represents same execution time, thus for points below the line, ffmpegcolorspace was faster, for those above, cogcolorspace was faster.  Most of the points clustered around the green line are statistically the same as the green line, since my timing method is quite crude.  Things to observe from this graph are that 1) many cases are very similar in speed, indicating that both ffmpegcolorspace and cogcolorspace are using similar code paths, 2) some cases, cogcolorspace is a lot faster, probably indicating that there isn’t an assembly fast path in ffmpegcolorspace for that conversion, and 3) a few cases (which, not coincidentally, are the most heavily used cases) ffmpegcolorspace is slightly faster than cogcolorspace.

The conclusions to draw from this are that 1) by writing very generic code with Orc, you can get very similar results to hand-crafted assembly code, and 2) a developer can cover a lot more cases with a small amount of work, and 3) there are a few cases where special-case Orc code would be beneficial.

This is only the low quality mode that cogcolorspace supports, which is similar or identical in quality to ffmpegcolorspace.  Higher-quality conversion is also implemented in most cases, and is only slightly slower in speed.  This is the real advantage of Orc — Orc takes care of huge number of combinations of options, and produces good SIMD code for all of them.

Orc is now to the point where it can not only reproduce about 90% of the code that is currently in liboil, but also generate 90% of the code that should be in liboil, but nobody ever wrote.  At runtime.  And the Orc language allows you to describe your own liboil-style functions.  At runtime.  Or, you can also use it like a normal compiler, converting Orc language source into N different assembly source files for every possible vector instruction set combination.

A large part of the decoding path in Schroedinger has been converted to optionally use Orc, where speed is either slightly faster or 20-30% faster than the previous liboil code.  The real benefit is that takes only a few minutes to convert code that took weeks to develop originally.  A side project of mine, Cog, has turned into a showcase for Orc, with demonstrations of video processing GStreamer elements, such as format and colorspace conversion and scaling.  I’ve found that since it is so easy and fast to create vectorized code, it now becomes possible to offer additional features to users, such as quality vs. speed tradeoffs.

Orc can generate code for MMX and SSE on x86 and x86_64, and Altivec on PowerPC, as well as NEON for ARM and c64x+DSP code.  The NEON and c64x+ backends are not currently open source.

Download 0.4.0.  Online documentation.

Existing and upcoming products include:

• A GStreamer-based Media SDK that allows developers to rapidly create and deploy applications on major platforms (Windows, Linux, OS/X)
• QuickTime plugins for DiracPro (SMPTE VC-2)
• A video encoder application geared toward content producers putting video on the web
• A capture application compatible with Numedia‘s line of DiracPro hardware encoders

In addition, Entropy Wave can provide support and custom development services in a variety of areas including open media.