Home

Video processing on the fly

Home

Challenge and Problem

The kernel is deemed to be the soul of UNIX/Linux. It does simple jobs like a multiplexer and has gracefulness to deal I/O like accessing files. However, with rapid boom of internet technology, streaming media (with transmission deadline and real-time considerations) become another type of "files" that cannot be simply dealt with like what the kernel does. With libraries for network, decoder and encoder that have different programming interface, prototyping and development of streaming media software become difficult. As streaming media becomes increasingly prevalent, there is a need to manipulate them in a useful way. For instance, in a web cast production, we may want to mix audio streams from multiple sources or transcode video streams for transmission to different clients. Our goal is allow developers to write complex manipulations on streaming media data in few lines of code, thus supporting rapid prototyping and experimentation.

What is Plasma?

Plasma is scripting framework that processes heterogeneous streaming media (scriPting Language for real-time manipulation and Adaption of Streaming MediA). Here we mean the scripting framework a generic abstraction of complex media processing libraries and codecs with a programming interface as a scripting language. Scripts are used widely in Unix/Linux environment to write administrative routines such as file batch processing. Media processing are complex jobs which involves decoding/encoding, image processing and network transmission. But we are resorted to either use full-fledged software or command-line programs with long list of options. The former requires a volume of resources but has limited customizability while the latter is not user-friendly. Plasma provides a solution "in-between" for users who needs rapid solution with maximized customizability and reusability so that no complex GUI is need to perform simple tasks while a script can be easily integrated in another program.

Plasma is designed to perform real-time mixing and manipulation without interruption of decoder and encoder, thus ensuring real-time delivery of data. To provide user with maximum ease of programming. It is deployed as a Tcl/Tk extension that can be plugged in any Tcl/Tk application. It also provides unified interface for various codecs (It uses libavcodec so it supports most popular codecs). Most remarkable, it adopts the concept of UNIX's IO redirection. Which means programmer can direct or multiplex any media stream from network, file or device to any of above mentioned possibilities as destinations.

Plasma has extremely simple object-oriented syntax. It only has around 10 operators which allow users to pack video streams spatially like Tk widgets (i.e. an empty video is a container while a sub video can be component inside the container or a new layer) It also has built-in events to make media stream produced by Plasma adaptive. For example, Plasma has motion detect events that can detect movement above specific threshold and key events raised by the GUI. In addition, the ability define new events, insert text into the video, and other effects allow video streams to be animated with great programming efficiency.

In terms of application, Plasma can be used to program interactive media services or as processing engine for any kind of media authoring or animation software. VJOnline, an digital video/audio mixing software that allows user veejaying various av streams and broadcast using RTSP was written in around 300 Plasma and Tk lines. It is used by Nemesys research group of National University of Singapore as a media gateway script for an adaptive video surveillance project while a Tk video editor is being developed using Plasma as processing engine. Besides, we also have developed some examples to demonstrate the possible application that can be written in Plasma.