The AOF Method for Production, Use, and Management of Instructional Media Wolfgang Hürst Institut für Informatik, Albert-Ludwigs-Universität Freiburg, Georges-Köhler-Allee 51, D-79110 Freiburg, Germany huerst@informatik.uni-freiburg.de Rainer Mueller imc AG, Office Freiburg, D-79110 Freiburg, Germany rainer.mueller@im-c.de Thomas Ottmann Institut für Informatik, Albert-Ludwigs-Universität Freiburg, Georges-Köhler-Allee 51, D-79110 Freiburg, Germany ottmann@informatik.uni-freiburg.de Conf. Proceedings Index: HÜRST, Wolfgang, MUELLER, Rainer, OTTMANN, Thomas Abstract: Automatic recording and capturing of live lectures and presentations has been explored and evaluated by various research projects since the mid 90s and has meanwhile been adapted by a number of commercial systems. Here, we discuss and summarize “Authoring on the Fly” (AOF) which is a special incarnation of this idea. The purpose of this paper is to give an overview of the AOF approach, the experience we have obtained and the requirements we have identified during its development over the last few years as well as to report on current and future research issues involved. Keywords: Presentation and lecture recording, multimedia authoring, production of e-lectures, system overview Introduction Lightweight content production using presentation recording instead of traditional authoring systems has become an increasingly popular method of creating instructional media for offline use, since it makes use of already existing educational potential at universities and companies. Teachers are normally well experienced in face-to-face education and training. Nowadays, live events, such as training sessions in companies or ex-cathedra lectures at universities, are in many cases already based on some kind of electronic material, such as PowerPoint-slides, digital images, video clips, etc. The slides are used in the live-event, graphically annotated and verbally explained by the presenter. Hence, it is natural to exploit this effort already invested in live-events for the automatic production of instructional content for later off-line use as well. “Authoring on the fly” (AOF) represents a specific incarnation of this idea of content production by presentation recording. Its main objective is to automatically capture live lectures and presentations in classrooms and lecture halls, make them available as e-lectures in a variety of output formats for access over the internet, and use them as kernels of web-based learning modules. The AOF concept, which started in the mid 90s, has been realized and implemented into several prototype systems over the last couple of years which have been used with increasing success at our university as well as by many other institutions. Many other research projects have explored this method, as well, for example the Classroom 2000/eClass project at the Georgia Tech (Abowd 1999) or the Berkeley Lecture Browser (BMRC 2004). Today, a number of commercial products exist for this task, such as Camtasia (Camtasia 2004), Microsoft Producer (Microsoft Producer 2004), or LECTURNITY (Lecturnity 2004), which is the commercial implementation of the AOF project. These products make automatic presentation recording available to a wider, non-technical user community. In more than five years of development, evaluation and continuous improvement and elaboration of the AOF approach, we identified several requirements for presentation recording under real-world conditions in actual classroom teaching. These are not restricted to the recording itself but include subtasks in all four phases of the whole production process when e- lectures are automatically created through presentation capturing (compare Figure 1). For example, the goal to automatically capture all material which will be presented in the classroom draws implications on the creation and composition of these data by the presenter in the preparation phase. Besides the actual recording, the design of the user interface is probably the most critical and most important challenge during the live event. After the actual lecture, the recordings have to go through a post-processing phase, where the system creates the final file or e-lecture based on the data streams that were captured during the live presentation. Finally, different issues arise and have to be addressed in the final post-usage, where the produced files are accessed and replayed by the end users. In the following section, we highlight the key features of AOF in order to delineate it from the many other approaches to presentation recording. Then, we go through each of the four phases of the recording and replay scenario in order to illustrate the significance of these features in detail and to discuss related past, current, and future research issues. Preparation Phase Presenter / author Live Event & Recording Post-Usage Post-Processing Phase Authoring on the Fly system for automatic presentation recording Student or other user Figure 1: Overview of all four phases which are involved in the process of automatically producing content via live-lecture and presentation recording including the final usage by students or other users. Key Features of the AOF Method Content production is lightweight and therefore efficient to realize, if the used method and its implementation is easy, quick, intuitive and flexible (Müller and Ottmann 2003). In the ideal case, the presenter or author is not even aware of the fact that everything will be recorded and content will be produced. This is usually subsumed under the principle of Ubiquitous Computing, in these days often named Seamless Computing as well. Authors and presenters should be able to keep their personal style of talking, presenting and teaching. All presentation material and media usually applied in such a live event should be supported without additional preparation effort. For quality, retrieval and archiving purposes, the information loss arising from the recording process should be kept to a minimum. Finally, the method and its implementation should provide the flexibility to produce target documents for arbitrary distribution scenarios and replay infrastructures. The following key features realized in AOF build the conceptual scaffold to approximate the described vision as well as possible: a multi-stream capturing method, an open and flexible intermediate format and a generic replay architecture. The multi-stream capturing method allows for separate capturing of the different media streams independent of type and scalable in their number. Lossless compression and vector-based recording is used wherever possible and meaningful, i.e. the lossless vector-based description is available and does, in its proprietary format, not contradict the generic idea of arbitrary input media support. Thus, new media types can be integrated easily. Single streams, such as the audio stream with the presenter’s voice, the video stream of presenter and audience, slides and graphical annotations, or applications, can be removed, suppressed or edited separately. The lossless storing supports indexing, archiving and retrieval. In the open intermediate format of AOF (SRM-Media, Müller and Ottmann 2000), the different streams are kept separately, only loosely coupled by the time. The streams are structured in a flat hierarchy oriented at one assigned audio stream usually covering the presenter’s voice. This allows a tight temporal synchronization and quality assurance for the semantically most important stream. The kind of stream bound together with the lossless compression forms the basis of supporting arbitrary distribution types and output formats, whether streaming vs. CD/DVD, video vs. Web vector graphic standards, or image sequences. Finally, the generic replay architecture (Hürst and Müller 1999) of AOF completely adapts the flexible structure of the individual documents. Separate processes, threads, applications or instances (Slaves) replay the streams. They are loosely coupled through simple time communication with the process replaying the assigned audio stream (Master). This architecture does not restrict the flexibility of SRM-Media and can be realized in nearly all target technologies and formats (e.g., Flash, QuickTime, SMIL, HTML+Time). Preparation Phase Nowadays, almost every live presentation is based on electronic material. The most widely- spread format for slides is PowerPoint, but LaTeX/PDF is popular at universities as well. Though one may graphically annotate these slides during a presentation using the features of the respective presentation software, these systems do not have a recording facility that supports the key features mentioned above. However, due to their widespread distribution and the familiarity with their usage, the users often prefer these tools for slide preparation to the editors provided by special recording or authoring systems. Therefore, AOF applies import filters for standard formats such as PowerPoint and LaTeX which support the most commonly used features for standard presentations (cf. Müller et al. 2002). They produce documents for the recording system which maintain the symbolic representation of all source objects, since AOF tries to maintain object-based sources wherever feasible; an important feature in order to preserve the best possible quality for later replay and to offer the highest flexibility for post- processing. If authors want to present applications, animations or video clips in a lecture as well, they will normally also use dedicated standard tools for their preparation. However, we cannot expect that all data streams generated by running arbitrary applications can be captured in a symbolic, object-oriented form. Therefore, screen grabbing is the only reasonable, universally applicable option for a recording of those streams. Today’s common DCT-based compression methods are tailored to photo-realistic images and videos. They are usually unsuitable for coding vector graphic images and videos, such as recorded applications, thus resulting in suboptimal quality and high data volumes. These methods do not take into account the characteristics of vector graphic, e.g.: sharp edges, long runs or large regions of identical color, very small deltas of consecutive frames. The Techsmith Screen Capture Codec coming with Camtasia realizes a compression optimized for vector graphics. The LECTURNITY Screen Grabbing Codec used in AOF is a more recent and a more efficient example for such a compression method. For animations, however, we did quite a bit more: If the animation has been programmed using the Java-library JEDAS (Java Educational Animation System, Jedas 2004), the running animation can not only be annotated during the live presentation but also recorded in object oriented form, cf. Lauer et al. (2001). The Live Event and Automatic Recording Live lectures are usually held in seminar rooms, classrooms or digital lecture halls and use an electronic presentation system. The traditional device for presenting slides is a PC running some presentation software and being connected to a large screen projector. Keyboard use and/or mouse paging slides forwards and backwards as well as starting application programs is possible. However, instead of using the desktop paradigm for interaction, in many lecture scenarios, the pen-based input is a more natural way of human-computer interaction. In recent years, a wide variety of input devices has become available which can be used in a presentation scenario. They range from palm-sized screens (hand-helds), over standard and portable screen sizes (Tablet PCs, Convertibles and touch screens), up to wall-sized interactive displays (composed of several serially mounted electronic whiteboards). We obtained best results with large, wall-mounted interactive displays (see Figure 2) and a specially designed lectern (see Figure 3) which allows for presenting and graphically annotating slides in a way that most teachers are used to from overhead projectors. The lectern integrates a Wacom- Cintiq, the audio equipment, and a PC running AOF or LECTURNITY for presentation, recording and content production. Figure 2: Wall-mounted interactive displays with touch sensitive surface for pen input. Our experience shows that, in addition to the usage of appropriate input devices, the design of the user interface of the integrated presentation software is crucial for a wide acceptance among teachers and should therefore be adapted to the respective scenario and devices. The AOF / LECTURNITY system follows the classical blackboard-and-chalk interaction paradigm (see Figure 4). The lecturer finds tools to change or annotate slides, such as pens or markers in different colors, in a foot-bar below the slides. In the same bar a thumbnail image of the video and an automatic control for the audio input-level is shown. Our experience with the tool’s usage in real-world settings highlighted the importance of such controls in order to prevent any problems that might occur during a recording. During the live-presentation, all actions including the pen-based annotations on the slides or on specifically opened empty pages are captured. A special feature is the possibility of switching from the object-based recording mode to the raster-based mode by using a screen grabbing facility which allows for recording arbitrary applications shown during the presentation, as described before. Our current research includes the proposal of a new interface design for presentation software (Hürst and Meyer 2004) which makes massive usage of a special version of pie menus which have proven to be supperior under certain circumstances in relation to pen-based computing. For large, wall sized interactive whiteboards, different interaction modes may be more appropriate when giving a presentation than the ones realized by a traditional user interface design. Instead of a menu-based form of human-computer interaction, a gesture-based interaction can be much more convenient in many situations: Forward or backwards paging commands for changing slides, erasing or insertion commands for text, as well a several other actions might be initiated by drawing respective gestures using the pen which are automatically recognized by an underlying gesture recognition system running in the background and interacting with the presentation and recording system. We have experimented with this approach and proposed an agent-based system for gesture recognition which can distinguish between “gestures” and domain specific “sketches” and which can be run in the background of arbitrary applications, in particular behind the scenes of a presentation and recording system (Mohamed 2003). Figure 3: Lectern with integrated PC, audio equipment, and a Wacom Cintiq tablet for freehand input. Figure 4: Snapshot of the LECTURNITY presentation software used by the lecturer during the live event. Even with available pen-based input devices, we realized that most presenters did not use freehand writing during their presentations, because the representation of the lecturer’s annotations in “digital ink” on the screen is generally not very satisfactory. In order to study the presentation and recording of handwriting – an issue where surprisingly few effort has been invested in the past – we developed an experimental system, called CrePCaR (Belenkaia et al. 2004), which supports the Creation, Presentation, Capture, and Replay of freehand writings. Our aim is to provide digital inks (pen-traces) in such a form that they are acceptable substitutes for their traditional physical counterparts. This requires maintaining the personal flavor, allowing pixel-based as well as object-based manipulations, storing digital ink in such a compact form that it can still be arbitrarily scaled in size without any loss in display quality in order to allow a synchronous replay with other data streams and to provide a strong random access facility. With the increasing availability and popularity of laptop computers among students and wireless network connections available on more and more campuses, students want to use their own computers during live lecture and to make and capture their own personal annotations on the material presented by the lecturer in digital form. Since the AOF system was originally developed as a tele-teaching tool, it has a communication interface and the AOF whiteboard can therefore be used as a shared whiteboard. This allows for distributing the slides shown by the lecturer to all students’ machines. However, this scenario introduces many new difficulties which are similar to those occurring in synchronous collaborative scenarios. For example, personal annotations of the students must not interfere with each other and with the lecturer’s annotations. Therefore, we extended traditional recording approaches by proposing a multi- layer recording for this scenario, cf. Lienhard and Lauer (2002). Post-Processing by the System As a result of the AOF recording process, documents in an open intermediate format are obtained where all recorded data streams are still kept separately. This allows for a seamless modification, transformation and integration of the resulting document. The data streams are loosely coupled along the time axis and can be replayed in such a way that both the inter- stream and the intra-stream synchronism are maintained. That way, the different streams can easily be integrated into target documents. However, no manual interference of the author is necessary in order to obtain an e-lecture for offline use. Not only is the mark-up of the recording with metadata according to common standards facilitated and automated as much as possible, but it is also easy to automatically transform them into a large variety of common output formats since the different data streams have been recorded by lossless codecs in the best possible quality. Currently, the following output formats are supported: the video of the lecturer, of the audience and other (e.g. of experiments in physical sciences) can be captured and replayed by using arbitrary standard codecs (like DivX, Indeo, MPEG-4). The same holds true for the integration of pre-processed videos to be included into a presentation. Slides shown on the AOF-whiteboard including the handwritten annotations of the presenter are recorded in an object-based form as XML/SGML-files. They can be transformed into Flash and SVG as well as into a specific format allowing for random access and random visible scrolling. Screen grabbings are obtained by a special lossless codec which is optimized for vector-based raster graphics. The audio stream of the lecturer can be recorded with different sample frequencies and is captured in uncompressed form. It can therefore be post-processed and transformed into various standard formats. For example, e-content obtained by LECTURNITY can be published in its own specific format, as well as in Flash, or in the standard streaming formats RealMedia and Windows Media. The importance of supporting standard output formats for recorded presentations is further discussed in Lauer, Trahasch, and Müller (2004). Issues Related to the Usage of the Documents The design and development of the AOF system has always gone hand in hand with the simultaneous usage and evaluation of the corresponding prototype systems in real-world situations, i.e. in actual lectures at our and many cooperating universities. With this approach, we were not only able to benefit from the development process of the presentation recording method itself, but also identified and highlighted requirements in usage scenarios for the produced e-lectures. Based on our experience with both, students using AOF in an educational scenario and customers of LECTURNITY using it in an industrial setting, we can conclude that both request for a seamless integration of the produced files into the overall learning process instead of using them as single, stand-alone products. AOF is well suited for such an integration due to its open and flexible recording and preservation approach as described in the previous sections. Our current and future research includes the integration of a hyperlink functionality into lecture recordings in order to easily combine it with online tests and exercises, the development of more flexible, user-friendly interfaces for browsing and reviewing lecture recordings (Hürst et al. 2004, Hürst and Götz 2004), automatic indexing and search that is optimized for the specific scenario and data type (Hürst 2003), as well as post-annotation support using pen-based interfaces (Lienhard and Lauer 2002). In addition, we started the development of approaches and tools for the usage of e-lectures as kernels for web-based learning modules, including a web-based annotation service (Fiehn et al 2003) in order to support group-work and anchored discussions or their usage in context with peer-assessment (Trahasch 2004). Evaluation and Field Reports The production of e-lectures by presentation recording using the AOF System, its commercial alternative LECTURNITY, or by using other systems, such as Camtasia, has become a standard service for CS students in Freiburg and at several other cooperating universities. The production process for AOF recordings is almost fully automated such that it is assured that the produced e-lectures can be made available online in different formats on the same day when the live lecture is held. E-lectures were not only used locally by the students on campus, but were also utilized occasionally for teaching students in pure distance mode. The latter teaching mode requires it to embed the AOF recordings into a learning management system, to enhance the e-lectures by quizzes and self-assessments, to provide tutorial support, and to organise assignment submission and correction over the internet. We carried out an empirical study for this group of students studying in pure distance mode, and found that the acceptance of distance courses based on e-lectures was generally very high. The satisfaction, however, depends also very much on other parameters beyond the quality of the e-lectures. These are the features of the learning management system, the mode of assignment submission and correction, and other technical prerequisites, like network access, and personal computers. However, empirical studies were also carried out for students on campus who had the option both to attend the live lectures and to access the AOF recordings. These studies allow a direct comparison of live lectures and e-lectures without the disturbing influence of the learning environment. We identified usage patterns and preferences for the preferred delivery formats of e-lectures by log file analysis and the results of questionnaires. Details can be found in Zupancic and Horz (2002) and in Lauer, Müller, and Trahasch (2004). The major findings may be summarised as follows: The average length of a user session working with an AOF recording was roughly two thirds of the average lecture length. Frequency of usage increased slightly at the beginning of the term, but significantly during the last weeks before the final examination. AOF-users can be classified into three different groups. First, there is a group of AOF-Non-Users, who exclusively attend the live lectures. Then there is a group of Occasional-AOF-Users. They showed a generally lazier working pattern. The third group, the Intensive-AOF-Users, were the hardest working students for the total event (lecture, exercises, and self-learning). It is interesting to note that the students who watched the AOF recordings longer and more often agreed that an AOF recording is a real alternative for the traditional live lecture. This observation is also supported by informal reactions obtained from students studying a course based on AOF recordings in pure distance mode. A recent survey among the 112 participants of an introductory course on Algorithms and Data Structures for second-term students reveals that about 60% of the students attended the live lectures regularly (at least once per week of the two lectures per week), 8% never showed up, 14.3% attended less than 5 lectures during the whole term, and 15.2 % attended roughly every second live lecture. More than 93 % used their own personal computers at home for learning, 2/3 of them had fast internet access (DSL, LAN, or WLAN). 59% agreed that e-lectures are acceptable replacements of live lectures. Because the AOF recording process records all data streams separately in best possible quality, it is possible to produce different delivery formats for the recorded e-lectures. First, there is the proprietary AOF format which maintains the symbolic representation of the slides and which provides very comfortable navigation facilities (random access and visible scrolling). This format however requires it to fully download the recording and to install the player software. On our server, we offer two versions of e-lectures in this format, one containing the lecturer’s video (leading to a data volume of roughly 500 MB for a one-hour lecture), the other one without the video and a size of approximately 1/10 th . Beyond that we can provide streaming versions (e.g., in RM or WM format regarding LECTURNITY) as well as Flash-versions of the e-lectures. It turns out that the slides of the lecture, the audio stream with the teacher’s voice, and the annotations made by the lecturer on the whiteboard were considered the most crucial constituents determining the quality of e-lectures. The teacher video is considered as not very important. Local availability with comfortable navigation facilities like sliding forward/backward and visible scrolling, thumbnail overviews, and scalability of slides without deterioration of display quality are considered as the most important features. Moreover, users expect at least simple retrieval functions like keyword and full-text search. Note, however, that the latter requires that the symbolic representation of the slides is not lost during the recording process. Therefore, it is not surprising that the Lecturnity format was the most frequently used version of e-lectures. Its overall rating was much higher than for Flash or Real Media. Summary In this paper we addressed the question of how multimedia material for teaching purposes, in particular e-lectures, can be produced automatically by capturing the live event of an actual presentation or classroom lecture. While many approaches and systems have been developed for this task in the last couple of years, the AOF method introduced by our group provides some outstanding features which we highlighted in the second section. The recording functionality offered by AOF as well as the existance of many commercial systems for automatic lecture and presentation capturing prove that the basic problem related to this task, namely how the data can be captured automatically while still preserving a high quality, is basically solved today. However, a lot of open research questions remain around this core functionality of AOF as well as any other system for presentation recording. One of the main contributions of this paper was to identify these current and future research issues by going through all the phases of the production process, including questions arising at the end when students actually use the produced documents for learning. References Abowd, G.D. (1999). Classroom 2000: An Experiment with the Instrumentation of a Living Educational Environment. IBM Systems Journal, Special issue on Pervasive Computing, Vol. 38, No. 4, October 1999. Belenkaia, L., Mohamed, K., Ottmann, T. (2004). Creation, Presentation, Capture, and Replay of Freehand Writing in e-Lecture Scenarios. Proceedings of ED-MEDIA 2004, World Conference on Educational Multimedia, Hypermedia & Telecommunications, 2004. BMRC Lecture Browser (2004). http://bmrc.berkeley.edu/frame/projects/lb/ Camtasia Studio, Techsmith Corp. (2004). http://www.techsmith.com/products/studio/default.asp Fiehn, T., Lauer, T., Lienhard, J., Ottmann, T., Trahasch, S., Zupancic, B. (2003). From Lecture Recording towards Personalized Collaborative Learning. Proceedings of CSCL 2003. Hürst, W., Müller, R. (1999). A Synchronization Model for Recorded Presentations and its Relevance for Information Retieval. Proceedings of ACM Multimedia 1999. Hürst, W. (2003). Indexing, Searching, and Skimming of Multimedia Documents Containing Recorded Lectures and Live Presentations. Proceedings of ACM Multimedia 2003. Hürst, W., Meyer, J. (2004) A New User Interface Desing for Giving Lectures and Presentations. Proceedings of E-Learn 2004, World Conference on E-Learning in Corporate, Government, Healthcare & Higher Education, 2004. Hürst, W., Götz, G., Lauer, T. (2004). New Methods for Visual Information Seeking Through Video Browsing. Proceedings of the 8 th International Conference on Information Visualisation, IV 2004. Hürst, W., Götz, G. (2004). Interface Issues for Interactive Navigation and Browsing of Recorded Lectures and Presentations. Proceedings of ED-MEDIA 2004, World Conference on Educational Multimedia, Hypermedia & Telecommunications, 2004. JEDAS, Java EDucational Animation System. Computer Science Institute. University of Freiburg, http://ad.informatik.uni-freiburg.de/jedas/ Lauer, T., Müller, R., Ottmann, T. (2004). Animations for Teaching Purpose: Now and Tomorrow. In Special Issue on “Future of Computer Science Symposium (FOCSS)” of Journal of Universal Computer Science (J.UCS), 7(5), 2001 Lauer, T., Trahasch, S., Müller, R. (2004). Web Technologies and Standards for the Delivery of Recorded Presentations. In Proceedings of ITHET 04, 2004. Lauer, T., Müller, R., Trahasch, S. (2004). Learning with Lecture Recordings: Key Issues for End-Users. In Proceedings of ICALT 2004, 2004. LECTURNITY ® , imc AG (2004). http://www.lecturnity.de Lienhard, J., Lauer, T. (2002). Multi-Layer Recording as a New Concept of Combining Lecture Recording and Students' Handwritten Notes. Proceedings of ACM Multimedia 2002. Microsoft Producer for PowerPoint, Microsoft (2004). Mohamed, K. A. (2003). Fast interpretation of pen gestures with competent agents. In P. Vadakkepat, W.W. Tan, K.C. Tan, and A.P. Loh (editors), Proceedings of the Second International Conference on Computational Intelligence, Robotics and Autonomous Systems, Singapore, December 2003. IEEE press. Müller, R., Ottmann, T. (2000). The “Authoring on the Fly” System for Automated Recording and Replay of (Tele)presentations. ACM/Springer Multimedia Systems Journal, Vol. 8, No. 3, May 2000. Müller, R., Ottmann, T., Zhang, H. (2002). Presentation Recording as a means to go virtual for Campus- based Universities. Proceedings of IRMA 2002. Müller, R., Ottmann, T. (2003). Content Production and the Essence of Simplicity. Proceedings of IEEE ITHET ‘03. Trahasch, S. (2004). Towards a Flexible Peer Assessment System. In Proceedings of 5th Int. Conf. on Information Technology Based Higher Education and Training: ITHET ?04 , IEEE, 2004. Zupancic, B., Horz, H. (2002). Lecture recording and its use in a traditional university course. In Proceedings of ITICSE, 7 th Annual Conference on Innovation and Technology in Computer Science Education, 2002. Acknowledgments This work was supported by several projects of the German Research Foundation (DFG) including the two research initiatives “Distributed Processing and Exchange of Digital Documents (V3D2)” and “Net-based knowledge communication in groups” as well as the project “Algorithmen und Datenstrukturen für ausgewählte diskrete Probleme”. Part of this work was carried out while the third author was visiting the UWA, Perth, under a Gledden Visiting Senior Fellowship.