Today I want to do a quick survery of some popular media processing libraries that are "graph-flavored". By that I mean that their design is based around creating nodes that describe sources, destinations (sinks) and intermediate processings steps, connecting them and then managing the data by "executing" the graph in some form or another.
Libraries
The defining characteristic of these libraries is having composable nodes acting on buffers.
- Media Foundation (MF) from Microsoft. I used this for a long time for various projects.
- GStreamer. Very similar to MF, but supports C and ends up depending on GObject to describe object-oriented type system constructs, and GLib as the base utility library.
- ffmpeg. Mostly geared around specific tasks and so the generic graph portion of it doesn't come as strong unless you look at the libavfilter library as well.
Design Notes
- The composable nature of the library lends itself well to building discrete functionality and utilities, like network or file sources/sinks, delaying/repeating transformations, tee'ing components, etc.
- The model around this makes it easier to debug and visualize the network of algorithms and data running, and you might see or build utilities for serializing/deserializing graphs.
- There is flexibility in allowing underspecified connections to be resolved at runtime, so for example two components might negotiate more efficient formats, or a governor component might inject adaptors where there is no built-in compatibility.
- There is some hardware that can map well to this model, like specialized audio processing nodes that can in fact route buffers.
Libraries with other styles
There are media libraries which don't have this, at least at the library surface. Here are some notable ones.
- Apple AVFoundation. There are assets and various objects to operate on them and for managing playback, but they tend to be very purpose-specific.
- Android Media.
For examples of how these libraries end up using more specific components with occasional interfaces to cobble them together, see the Grafika examples from Android, or read the SurfaceTexture docs for how they can be chained in more flexible ways (but not at all quite like what Media Foundation or GStreamer support).
Happy media processing!
Tags:
graphics
Home