Partner Organizations: Integrated Media Systems Center: Financial Support; Facilities; Personnel Exchanges
The Integrated Media Systems Center (IMSC) is an NSF Engineering Research Center (ERC) in the area of multimedia. We received financial support for graduate research assistants as well as part of the PI's salary. In addition, IMSC provided equipment and laboratory space to the project.
Raptor Networks Technology contributed two high bandwidth, multi-layer Ether-Raptor-1010 (ER-1010) network switches to our laboratory for use in the HYDRA project.
We received an equipment grant consisting of 13 NetStructure 7140/7145 traffic directors.
We have received a Sony Vaio VGN-UX180P ultra-light palmtop on loan for mobile and wireless experiments in the PWICE project (Pratt & Whitney Institute for Collaborative Engineering), established jointly between Korean Air, Pratt & Whitney, Inha University and USC's Integrated Media Systems Center.
QUAD Corporation, located in Irvine, California, designs and manufactures carrier-grade streaming media server hardware. The company is in the process of signing a research license for a test deployment of the first generation HYDRA streaming software on their platform. They have provided us with storage and computing resources on one of their Converged Delivery Platform (CDP) servers. We will be conducting real-world experiments during the second year of this project.
We have started a collaboration with the NSF sponsored ITR RESCUE project (Responding to Crises and Unexpected Events, awards #0331707 and #0331690) at the University of California at Irvine. We have received access to their video streams and we will be testing the HYDRA recording capabilities with these data.
Other collaborators:
Prof. Elaine Chew (Daniel J. Epstein Industrial and Systems Engineering, music computation and cognition) helped us with the design of the Distributed Immersive Environment experiments, which were recorded with HYDRA. Prof. Shri Narayanan (Ming Hsieh Department of Electrical Engineering, multimodal interactions) has contributed to the design of the PWICE Remote Maintenance & Training System (RMTS). Prof. Christos Papadopoulos (Computer Science, Colorado State University, networking) collaborated with us on the network protocol aspects for live streaming support. Prof. Seon Ho Kim (Computer Science, University of Denver, storage) worked with us on a new method called Z-RAID to improve the bandwidth efficiency of multimedia storage. Drs. Sumit Roy and Michelle Covell (Hewlett-Packard Labs, analytical modeling) helped us to define a reliable calibration model based on extensive data measurements for streaming media services.
Activities and findings:
Research and Education Activities: During the first year of the project our focus has been three-fold. First, we have implemented the multi-stream recording capabilities in our streaming architecture. Our activities focused on a unified architecture that integrates multi-stream recording and retrieval in a coherent design. We have worked on efficiently supporting multizone disk drives, variable bit rate media, and disk drives that have different write versus read bandwidth. Our emphasis has been on high-performance algorithms that scale to very large systems in terms of the number of streams supported. To test the capabilities of HYDRA, an initial version of the software was used to capture streams of different modalities in our Distributed Immersive Performance (DIP) experiments. The goal of the DIP activities is to gain insights into the challenges of networked performances, i.e., to provide a seamless environment for remote and synchronous musical collaboration. Specifically, the experiments explore the effects of auditory latency on musical ensemble and interpretation in order to determine the thresholds for usability. We focused on two classes of data streams that were generated within the DIP environment. The first category consists of high-resolution isochronous media streams, namely audio and video. The second class contains MIDI data produced by electronic instruments. We investigated consistent recording and retrieval of multiple streams, which were correlated with temporal indices. We successfully collected over 1.5 terabytes of data (audio, HD video and MIDI streams) and in an ongoing process the data is being analyzed. Initial results have been published in a number of conferences. The second aspect of HYDRA has focused on live, real-time digital video streaming for collaborative applications. In addition to emphasizing scalability in terms of number of streams, we are also exploring a second scalability dimension: the diverse capabilities of the end-systems. Future collaborative systems must effectively support a variety of heterogeneous clients ranging from low-power, wireless handhelds to high-definition, fixed room installations. The live streaming capabilities complement the recording functions in that they allow users to interact and observe events with low delay, while all session data is being archived for later analysis and playback. For example, a telepresence training session may be captured in its entirety and then be time-shifted for teaching purposes. This year we have refined our HD live streaming modules and tested them on cross-pacific WAN connections. In addition we also have performed successful experiments on the corporate network of Korean Air as part of the PWICE project (Pratt & Whitney Institute for Collaborative Engineering). This presented real-world challenges as the Korean Air internal network is quite heterogeneous and bandwidth is a scarce resource at certain facilities. Our most recent step has been the integration of the HYDRA high definition live video streaming capabilities and the HYDRA recording functions. We have verified through extensive testing that our prototype can record and playback multiple HD video and audio streams simultaneously. We demonstrated this integration as a part of the PWICE Remote Maintenance & Training System (RMTS) presentation during IMSC NSF site visit and the Scientific Advisory Board meeting on June 12-13, 2006.
Findings: During the first year of the project we had some major findings on both the technical and psychophysical performance aspects. We plan to further pursue the results of our findings and use them to further study and refine our HYDRA design. We deliberately designed HYDRA to handle very heterogeneous workloads (e.g., different clients and low to high-bandwidth stream types). In such a case allocating main memory buffers uniformly for each stream may not result in the highest overall system performance. To maximize system throughput while satisfying each client's QoS requirements we investigated an effective resource management framework composed of a dynamic memory allocation strategy and a novel deadline setting policy (F-DSP). Furthermore, we constructed a probability model based on the classic M/G/1 queueing model and the recently developed Real Time Queueing Theory (RTQT) to evaluate the performance trade-offs among different buffer allocation strategies and deadline setting policies. We were able to demonstrate that our new F-DSP policy results in the fairest scheduling when various, heterogeneous streams are serviced. Another finding resulted from the qualitative and quantitative analysis of the audio, video and MIDI (Musical Instrument Digital Interface) streams we collected in the DIP experiments. Our aim was to explore the effects of auditory latency on musical ensemble and interpretation in order to determine the thresholds for usability. We proposed two measures - the segmental tempo difference and the segmental tempo ratio from a baseline performance - as objective quantifiers of performance strategies. We demonstrated that the proposed analyses of the performance data, captured in MIDI format, accurately identifies the usability threshold to be between 50 and 75 ms. The tempo difference and tempo scaling across logical segments of the piece showed marked increase in variability when the auditory delay was in the range of 50 to 100 ms (for two examples) and 50 to 75 ms (for the most rapid example). These findings indicate that segmental tempo difference and ratio analyses are useful indicators of performance decisions, and that quantitative analysis of performance data may be a viable way of evaluating the psychophysical effects of collaborative performance under various immersive conditions.
Training and Development: This project integrates components from various technical disciplines such as networks, storage and operating systems. Currently, the students working on this project are exposed to all these different areas and get a hands-on learning experience. In addition, our collaborators include musicians and researchers with backgrounds in human performance evaluations. Therefore, the students are involved in a truly interdisciplinary team. The prototype HYDRA system is being continuously offered to interested collaborators. Our experiments with Inha University and Korean Air have resulted in students, researchers and operators being trained on the use of these new technologies.
Outreach Activities: We are in the process of deploying the HYDRA server software on QUAD Corporation's Converged Delivery Platform at their headquarters in Irvine, California. We hope that the expansion of HYDRA beyond our academic laboratory setting will allow us to demonstrate our findings to more interested parties, both industrial and academic. We have started a collaboration with the NSF sponsored ITR RESCUE project (Responding to Crises and Unexpected Events) at the University of California at Irvine. The RESCUE environment presents an ideal testbed that allows us to explore the scalability of HYDRA in a heterogeneous environment (i.e., different types of cameras, wired and wireless networks). In RESCUE potentially thousands of cameras may be connected via a large network. Our goal is to gain additional insights into the capabilities of our platform. HYDRA is a key component of the telepresence component of the PWICE (Pratt & Whitney Institute for Collaborative Engineering) project. We will be deploying HYDRA's live streaming capabilities at three Korean Air sites in early 2007. In this real-world, heterogeneous networking environment, HYDRA will be used to facilitate maintenance and training activities between Korean Air and Pratt & Whitney technicians. This year in preparation of the testbed deployment we have performed a number of cross-pacific experiments of high-definition live streaming between Inha University in Incheon, South Korea, and our laboratory in Los Angeles. The prototype HYDRA system is being continuously offered to interested collaborators. For example, the CCRMA (Center for Computer Research in Music and Acoustics) group at Stanford University has obtained the HYDRA HD live streaming software. In addition, we have sent our software to a collaborator at the Louisiana Biomedical Research Network (LBRN), situated at Louisiana State University. While these activities are still in their early stages, we hope to gain valuable insight into the performance of our algorithms under different, real-world conditions. HYDRA has been utilized within the GeoDec project (Geospatial Decision Making). GeoDec is a prototype testbed that is being built at USC's Integrated Media Systems Center. The goal of GeoDec is to enrich the immersive reality experience with real world data and information such that its users can naturally and efficiently acquire the required knowledge to make critical decisions. In addition to images (maps), alphanumeric and geospatial vector data, GeoDec is also integrating numerous video streams acquired from video sensors are accessible from within the GeoDec environment. An important feature is a temporal history browser that allows the user to go back in time by moving a slider on the user interface. HYDRA will enable this functionality by continuously recording a sliding temporal window of all the incoming video streams. GeoDec project provides an excellent application platform to evaluate our HYDRA design.
Journal Publications:
Other Specific Products:
A four minute video clip illustrating the recording and event indexing capabilities of HYDRA has been produced.
This video clip is available from the project web site. It has been shown during a number of talks. For example, it will be presented at the annual meeting of NSF ERC's on Friday, December 1, 2006.
The scalable high-definition live streaming component of HYDRA.
It will be installed in a test deployment by March 2007 at three airports of Korean Air: Incheon, Gimpo and Pusan.
http://dmrl.usc.edu/hydra/nsf-hydra-report-06.html http://dmrl.usc.edu/hydra.html
Contributions:
Contributions within Discipline:
The main contribution of this project by far has been the design and implementation of a versatile, general-purpose continuous media recording and dissemination architecture that provides the functionalities required by a broad class of applications that manage streaming media. Our efforts focus on three major system components: (1) stream acquisition (error protection, admission control), (2) storage and memory management (packet and block handling), and (3) query and retrieval (including synchronization). The preliminary work that led to this grant addressed item (1), specifically the admission control mechanism. An effective admission control algorithm must manage the dynamically changing system resources and we realized that existing techniques did not account for the difference in read and write bandwidth of modern disk drives when a mix of retrieval and playback streams are serviced. To address this challenge we designed a novel statistical admission control algorithm called TRAC (Three Random variable Admission Control). Our extensive measurement and analysis showed that the proposed algorithm can greatly increase the number of supportable streams and that it achieves a much more realistic resource utilization (up to 38% higher) as compared with the best, previously proposed algorithm based on a single random variable (1RV) model. One recent contribution addressed item (2). We investigated the media block placement in large server clusters. Random placement of data blocks has been proven to be an effective approach to balance heterogeneous workloads in multi-disk steaming architectures. However, we realized that existing techniques have a significant deficiency in that statistical variations can still result in short term load imbalances in disk utilization. We found that a much more fine-grained packet level randomization (PLR), as opposed to block level randomization (BLR), can greatly improve short term load fluctuations. We quantified the exact performance trade-off between the PLR approach and the traditional BLR techniques through both theoretical analysis and extensive simulation. Our results show that the PLR technique can achieve much better load balancing in multi-disk streaming architectures at a cost of using only slightly more main memory. For example, in one of our experiments servicing 205 streams the PLR technique reported 0% missed block retrieval deadlines compared with 37.9% with the BLR technique. The query and retrieval aspects, item (3), are being investigated as part of the ongoing efforts.
HYDRA was used to collect over 1.5 terabytes of data as part of the Distributed Immersive Performance experiments. The data is hosted online at http://datalab.usc.edu/dip and can be made available to researchers and other interested parties. The data is currently being analyzed and has led to some interesting new insights. The results of this ongoing analysis have been disseminated at several conferences through presentations, for example, 'Distributed Immersive Performance: Ameliorating the Psychophysical Effects of Network Latency' at the CENIC 2005 meeting, and 'Segmental Tempo Analysis of Performances in Performer-Centered Experiments in the Distributed Immersive Performance Project' at the Sound and Music Computing '05 conference in Italy. Therefore, our work on capturing massive streaming events is having an impact in the areas of collaborative and networked performances.
An undergraduate student, Mr. Jon Christen, was actively involved with our peer-to-peer streaming project. He has shown significant interest in pursuing a research career.
A test installation of the HYDRA server is being deployed at QUAD Corporation, located in Irvine, California, and a research license is being negotiated with QUAD. The recording and playback capabilities of HYDRA are being evaluated with data made available by the ITR RESCUE project (UC Irvine). This grant and the preliminary work leading to the grant have resulted in the following two pending patents: 'HYDRA: High-Performance Data Recording Architecture for Streaming Media;' U.S. utility patent no. 11/157,557 filing based on provisional application no. 60/581,888, June 21, 2005. Patent application published electronically on December 22, 2005; Roger Zimmermann and Kun Fu. 'AudioPeer - A Multi-Party Audio Conferencing System Based on a Scalable Peer-to-Peer Architecture;' U.S. utility patent no. 11/504,536 filing based on provisional application serial no. 60/707,816, August 11, 2006; Roger Zimmermann, Leslie S. Liu, Kemal Oral Cansizlar and Beomjoo Seo.
Special Requirements for Annual Project Report: