HYDRA - High-performance Data Recording Architecture

Sponsor and Supporters

This research is funded in part by an NSF award from the program for Advancing Collaborative and Intelligent Systems and their Societal Implications, IIS-0534761. We have received equipment gifts from Intel, Hewlett-Packard, Raptor Networks Technology, and Sun Microsystems.

Research activities at the USC Data Management Research Lab are focusing on the design and implementation of a High-performance Data Recording Architecture (HYDRA). The goal of HYDRA is to improve current and enable new applications by acting as an efficient media stream coordinator that manages the transmission, recording, and playback of many different data streams simultaneously. The objective of HYDRA is to use a unified paradigm that integrates multi-stream recording, retrieval and control in a synergetic manner. HYDRA aims to provide the same services for all media, independent of their bandwidth requirements, resolution or modality. One of the applications that we are exploring for this technology is a Distributed Immersive Performance where musicians and audiences are geographically disbursed in different locations.

HYDRA is capable of high definition live video streaming. We unveiled this feature on October 28, 2003. Video at 1280x720 pixel resolution (in HDV format) is transmitted at approximately 20 Mb/s over traditional IP networks such as the Internet. Each MPEG-2 transport stream video is acquired from a JVC JY-HD10U camera via FireWire, packetized, transmitted and displayed with a software or hardware decoder on Linux. Additionally, two audio channels are transmitted as well. In the past, achieving this level of media quality required very costly equipment. However, improvements in information technology now present us the opportunity to apply more cost-effective hardware and hence enable many more interesting applications.

The above image shows the pre-event tests of a 4-channel HYDRA live streaming setup that was used to stream The Miró Quartet: Live & Virtual for the Gala Event at the Fall 2004 Internet2 Member Meeting (Austin, Texas). Four HD cameras (on the right) are used to acquire live video which is then transmitted to and rendered on four receivers over a commodity IP network. (Note, no color calibration had been done at the time the picture was taken.)

The above images show our first long-distance test that was performed in January 2004 via a trans-pacific Internet2 link from the University of Hawaii at Honolulu, to the USC campus in Los Angeles, CA.

The above picture shows a trans-pacific, live two-way linkup between Inha University, Incheon, South Korea, and USC on October 8, 2004.

The above pictures show HD live streaming between two sites on the USC campus. Low latency rendering is achieved with an optimized software decoder that utilizes the MPEG decoding hardware-assistance available on recent graphics processing units (GPUs).

The HYDRA architecture is based on a scalable cluster design. Each cluster node is a off-the-shelf personal computer with attached storage devices and, for example, a Fast Ethernet connection. The HYDRA server software manages the storage and network resources to provide real-time service to the various clients that are requesting media streams.

The HYDRA project has motivated a number of different sub-projects that are aimed at improving various aspects of the system.

• Admission Control: The task of the admission control algorithm is crucial to ensure that no more streams are admitted than the system can handle with a predefined quality. Unlike traditional data management systems, whose performance may degrade gracefully under high load conditions, streaming media servers experience undesirable stream disruptions (also called ``hiccups'') when resources are exhausted. The challenge of designing a good admission control algorithm is to allow the all resources to be utilized to their fullest potential while preventing overload conditions from occurring. The resources under consideration are (at least) main memory, processor utilization, effective disk and network bandwidth.
• Memory Management: Managing the available memory efficiently is an important aspect of any multimedia streaming system. Main memory is required to temporarily hold data items while they are transferred between the incoming and outgoing network as well as the permanent disk storage. For efficiency reasons, network packets are generally much smaller than disk blocks. The assembly of incoming packets into data blocks and conversely the partitioning of blocks into outgoing packets requires main memory buffers. With a stream recorder, incoming data can be held in memory variable amount of time and the deadline by which a data block must be written to disk can be extended. This can reduce disk contention and hence the probability of stream disruptions.

Furthermore, HYDRA is the basis of the Distributed Immersive Performance (DIP) project. DIP is a testbed that integrates many of the technologies that are the result of multiple research efforts. The goal of the DIP is to reproduce the complete aural and visual ambience of an environment that includes people and other real and virtual elements.

We have performed experiments across both LAN and WAN environments. Our most recent tests were conducted via a trans-pacific Internet2 link from the University of Hawaii at Honolulu, to the USC campus in Los Angeles, CA.

• September 28, 2004: A 4-channel HYDRA live streaming setup that was used to stream The Miró Quartet: Live & Virtual for the Gala Event at the Fall 2004 Internet2 Member Meeting (Austin, Texas).

  

• March 17, 2004: The HYDRA HD live streaming system was demonstrated at the CENIC 2004 conference, in Marina del Rey, California, at the Ritz-Carlton Marina del Rey Hotel.

  Abstract PDF.

• Febuary 5, 2004: Aloha Broadband! USC-Honolulu Videoconference Demonstrates High Quality, Low-Cost Internet Streaming, School of Engineering News.

Article PDF (100 KB).

• January 28 & 29, 2004: I attended the Techs in Paradise (TIP 2004) conference at the University of Hawaii, organized by the Asia-Pacific Advanced Network Consortium (APAN). I was part of a Birds-of-a-Feather session on issues related to HDTV and we demonstrated a two-way HYDRA HD live streaming system between USC and the University of Hawaii (approx. 5,000 km).

Abstract PDF.

• Kun Fu, Ph.D., (Admission Control, Memory Management)
• Moses Pawar, M.S., (System Architecture)
• Amit Pande, M.S., (HD Live Streaming)
• Beomjoo Seo, Ph.D.
• Min Qin, Ph.D.
• Hong Zhu, Ph.D.
• Dwipal A. Desai, M.S., (Alumni)

The HYDRA technology is available for licensing. More information can be found at USC's Office of Technology Licensing.

• High-quality Video Service for Access Grid at the Networked Media Lab., Gwangju Institute of Science and Technology (GIST).
• ACE Advanced Collaboration Environment on Access Grid at the Networked Media Lab., GIST.
• libmpeg2, a fast, open source Linux MPEG-1 and MPEG-2 software video decoder.
• Vela Research, home of the CineCast and CineCast HD hardware MPEG-2 decoders.
• RME, home of the multi-channel RME 9652 "Hammerfall" sound card.
• RTP: the Real-Time Protocol for streaming media.

Get the PDF reader from Adobe.

For additional publications see the following project pages: Continuous Media Storage, Advanced Media Transmission, and Remote Media Immersion.

• High Definition Live Streaming.
  Min Qin and Roger Zimmermann. Book chapter in Encyclopedia of Multimedia, editor Borko Furht, published by Springer, 2006.

---

• High Resolution Live Streaming with the HYDRA Architecture.
  Roger Zimmermann, Moses Pawar, Dwipal A. Desai, Min Qin, and Hong Zhu.
  Published in the ACM Computers in Entertainment (CiE) journal, volume 2, issue 4, October/December 2004.
  Abstract Postscript (2,446 KB) PDF (643 KB)

---

• Memory Management for Large Scale Data Stream Recorders.
  Kun Fu and Roger Zimmermann.
  Accepted for presentation at the 6th International Conference on Enterprise Information Systems (ICEIS 2004), Porto - Portugal, April 14-17, 2004.
  Abstract Postscript (373 KB) PDF (1,300 KB)

---

• HYDRA: High-performance Data Recording Architecture for Streaming Media.
  Roger Zimmermann, Kun Fu and Dwipal A. Desai. Book chapter in Video Data Management and Information Retrieval, editor Sagarmay Deb, University of Southern Queensland, Toowoomba, QLD 4350, Australia. Published by Idea Group Inc., publisher of the Idea Group Publishing, Information Science Publishing and IRM Press imprints, 2004.

---

• Comprehensive Statistical Admission Control for Streaming Media Servers.
  Roger Zimmermann and Kun Fu.
  Accepted for publication at the 11th ACM International Multimedia Conference (ACM Multimedia 2003), Berkeley, California, November 2-8, 2003.
  Abstract Postscript (1,055 KB) PDF (348 KB)

Maintained by Roger Zimmermann
Last updated: Friday November 3, 2006.
All Rights Reserved © NUS Data Management Research Laboratory 1999 - 2007.