Workshop on Deep Learning
--- Systems, Algorithms and Applications
Time & Venue
16 September 2015 @ Seminar Room 1 (COM1-02-06), School of Computing, NUS
|Beng Chin Ooi||NUS||Welcome and Introduction||10:00-10:15|
|Chonho Lee||NUS||Apache SINGA: Levelling the playing field||10:15-10:50|
|Qirong Ho||I2R||Petuum: A New Platform for Distributed Machine Learning on Big Data||10:50-11:25|
|Steven Hoi||SMU||Large Scale Deep Learning for Image Classification and Retrieval||11:25-12:00|
|Gang Wang||NTU||Increasing the capacity of deep neural networks by learning spatial dependency and fine-grained features||13:15-13:50|
|Sim Khe Chai||NUS||Dissecting the Deep Neural Networks for a Better Insight||13:50-14:25|
|Yue Zhang||SUTD||Deep Learning for Stock Market Prediction||14:25-15:00|
|Sheng Wang & Wei Wang||NUS||Apache SINGA: From a User’s Perspective||15:00-16:00|
|Tea & Interaction||16:00-17:00|
Welcome and Introduction
Biography: Beng Chin is a Distinguished Professor of Computer Science and Director of IDMI at the National University of Singapore (NUS), and an adjunct Chang Jiang Professor at Zhejiang University. He obtained his BSc (1st Class Honors) and PhD from Monash University, Australia, in 1985 and 1989 respectively. His research interests include database system architectures, performance issues, indexing techniques and query processing, in the context of multimedia, spatio-temporal, distributed/parallel/P2P/in-memory/cloud database systems and applications. Beng Chin has served as a PC member for international conferences such as ACM SIGMOD, VLDB, IEEE ICDE, WWW, and SIGKDD, and as Vice PC Chair for ICDE’00,04,06, co-PC Chair for SSD’93 and DASFAA’05, PC Chair for ACM SIGMOD’07, Core DB PC chair for VLDB’08, and PC co-Chair for IEEE ICDE’12. He was an editor of VLDB Journal and IEEE Transactions on Knowledge and Data Engineering, Editor-in-Chief of IEEE Transactions on Knowledge and Data Engineering (TKDE)(2009-2012), and a co-chair of the ACM SIGMOD Jim Gray Best Thesis Award committee. He is serving as a co-Editor-in-Chief of Elsevier’s Journal of Big Data Research, an editor of IEEE Transactions on Cloud Computing and Springer’s Distributed and Parallel Databases, and a PC co-Chair of IEEE Big Data’15 (2015). He is also serving as a Trustee Board Member and President of VLDB Endowment, and an Advisory Board Member of ACM SIGMOD. He co-founded yzBigData in 2012 for Big Data Management and analytics.
Beng Chin is the recipient of ACM SIGMOD 2009 Contributions award, a co-winner of the 2011 Singapore President’s Science Award, the recipient of 2012 IEEE Computer Society Kanai award, 2013 NUS Outstanding Researcher Award, and 2014 IEEE TCDE CSEE Impact Award. He is a recipient of VLDB’14 Best Paper award. He is a fellow of the ACM and IEEE.
Apache SINGA: Levelling the playing field
Abstract: Apache SINGA is a general distributed deep learning platform for training big deep learning models over large datasets. It is designed with an intuitive programming model based on the layer abstraction. A variety of popular deep learning models are supported, namely feed-forward models including convolutional neural networks (CNN), energy models like restricted Boltzmann machine (RBM), and recurrent neural networks (RNN). Many built-in layers are provided for users. SINGA architecture is sufficiently flexible to run synchronous, asynchronous and hybrid training frameworks. SINGA also supports different neural net partitioning schemes to parallelize the training of large models, namely partitioning on batch dimension, feature dimension or hybrid partitioning. We will introduce the design and implementation of the system in this talk. Sample applications will also be demonstrated to help audiences get started with Apache SINGA.
Biography: Dr. Chonho Lee works as a research fellow at School of Computing, NUS. He received his Ph.D in Computer Science from University of Massachusetts, Boston. He has investigated self-adaptation mechanisms in distributed systems such as data centers and sensor networks based on his expertise in multi-objectives optimization, evolutionary computing, game theory and auction theory. His current research interests include big data analytics in health-care area. He has been working on analysis of visually-guided motor performance of Parkinson’s disease patients to help doctors diagnose the amount and time of L-dopa dosage. He recently joins SINGA team and tries to design predictive analytics model for particular disease.
Petuum: A New Platform for Distributed Machine Learning on Big Data
Abstract: What is a systematic way to efficiently apply a wide spectrum of advanced ML programs to industrial scale problems, using Big Models (up to 100s of billions of parameters) on Big Data (up to terabytes or petabytes)? Modern parallelization strategies employ fine-grained operations and scheduling beyond the classic bulk-synchronous processing paradigm popularized by MapReduce, or even specialized graph-based execution that relies on graph representations of ML programs. The variety of approaches tends to pull systems and algorithms design in different directions, and it remains difficult to find a universal platform applicable to a wide range of ML programs at scale. We propose a general-purpose framework, Petuum, that systematically addresses data- and model-parallel challenges in large-scale ML, by observing that many ML programs are fundamentally optimization-centric and admit error-tolerant, iterative-convergent algorithmic solutions. This presents unique opportunities for an integrative system design, such as bounded-error network synchronization and dynamic scheduling based on ML program structure. We demonstrate the efficacy of these system designs versus well-known implementations of modern ML algorithms, showing that Petuum allows ML programs to run in much less time and at considerably larger model sizes, even on modestly-sized compute clusters.
Biography: Dr. Qirong Ho is a scientist at the Institute for Infocomm Research, A*STAR, Singapore, and an adjunct assistant professor at the Singapore Management University School of Information Systems. His primary research focus is distributed cluster software systems for Machine Learning at Big Data scales, with a view towards correctness and performance guarantees. In addition, Dr. Ho has performed research on statistical models for large-scale network analysis — particularly latent space models for visualization, community detection, user personalization and interest prediction — as well as social media analysis on hyperlinked documents with text and network data. Dr. Ho received his PhD in 2014, under Eric P. Xing at Carnegie Mellon University’s Machine Learning Department. He is a recipient of the Singapore A*STAR National Science Search Undergraduate and PhD fellowships, and is a recipient of the SIGKDD 2015 Doctoral Dissertation Award.
Large Scale Deep Learning for Image Classification and Retrieval
Abstract: Recent years have witnessed the booming advances of deep learning techniques and their influences to a wide range of multimedia applications, including image classification and image retrieval. The boom of deep learning partly thanks to the advances of new GPU technologies with increasing parallel computation powers. In this talk I will address some open issues with state-of-the-art deep learning techniques for image classification and image retrieval: (i) large-scale training of Deep Convolutional Neural Networks using multiple GPUs; and (ii) effective adaptation of deep learning techniques to resolve content- based image retrieval tasks with pre-trained deep learning models on image classification datasets.
Biography: Dr. Steven Hoi is an Associate Professor in the School of Information Systems (SIS), Singapore Management University (SMU), Singapore. Prior to joining SMU, he was a tenured Associate Professor at the School of Computer Engineering of the Nanyang Technological University (NTU), Singapore. He received his Bachelor degree from Tsinghua University, and his Master and Ph.D degrees from the Chinese University of Hong Kong. His research interests include large-scale machine learning with application to tackling big data analytics challenges across a wide range of real-world applications, including multimedia retrieval, social media, web search and information retrieval, computer vision and pattern recognition, computational finance, cyber security, mobile and software data mining, etc. He has published over 100 papers in premier conferences and journals, and served as an organizer, area chair, senior PC, TPC member, editors, and referee for many top conferences and premier journals.
Increasing the capacity of deep neural networks by learning spatial dependency and fine-grained features
Abstract: Deep neural networks have achieved great success on many vision tasks. Compared to traditional shallow learning methods, deep neural networks have higher capacity to represent more complicated visual patterns in images. Currently, researchers increase the representation capacity to fit a large amount of training data by adding more convolution layers and pooling layers. In this talk, I am going to introduce two alternative methods: one is to learn spatial dependency between image regions using a DAG-RNN network; the other one learns fined-grained CNN networks to represent discriminative features between subsets of categories. Both methods can produce richer and deeper neural networks to extract more informative features from images. Experiments on benchmark datasets have proven the effectiveness of our proposed methods.
Biography: Wang Gang is an Assistant Professor with the School of Electrical & Electronic Engineering at Nanyang Technological University (NTU) and was a research scientist in ADSC, A’Star. He received his B.S. degree from Harbin Institute of Technology in Electrical Engineering in 2005 and the PhD degree in Electrical and Computer Engineering, University of Illinois at Urbana-Champaign in 2010. During his PhD study, he is a recipient of the prestigious Harriett & Robert Perry Fellowship (2009-2010) and CS/AI award (2009) at UIUC. His research interests include computer vision and machine learning. Particularly, he is focusing on object recognition, scene analysis, and deep learning.
Dissecting the Deep Neural Networks for a Better Insight
Abstract: Deep Neural Network (DNN) has been found to yield superior performance compared to the conventional Gaussian Mixture Model (GMM) based systems for automatic speech recognition. However, DNN has been used pretty much as a black box without much insights as to what the DNN has learned. This talk will present a novel approach for interpreting the DNN model, which is based on analysing the hidden activity pattern. This technique constructs a 2-dimensional hidden activity space where interpretable regions can be defined. This technique can be used to facilitate the understanding and comparison of the hidden activity patterns across different hidden layers, networks and time frames.
Biography: Dr. Sim Khe Chai is an Assistant Professor at the School of Computing (SoC), National University of Singapore (NUS). Previously, he was a research engineer at the Institute for Infocomm Research (I2R), one of the research institutes of Agency for Science, Technology and Research (A*STAR). He received the B.A. and M.Eng degrees in Electrical and Information Sciences from the University of Cambridge, England in 2001. He worked on the Application Programming Interface (API) for Hidden Markov Model Toolkit (HTK) (known as the ATK) for his Undergraduate final year project under the supervision of Prof. Steve Young. He was then awarded the Gates Cambridge Scholarship to persue the course of Computer Speech, Text and Internet Technology (CSTIT) at the same university. He completed his M.Phil dissertation “Covariance Matrix Modelling using Rank-One Matrices” in 2002 under the supervision of Prof. Mark Gales. He joined the Machine Intelligence Laboratory (MIL) (formerly the Speech, Vision and Robotics (SVR) group), Cambridge University Engineering Departmentin the same year as a research student, supervised by Prof. Mark Gales. He received his Ph.D degree in July 2006. He is also an alumnus of Churchill College. His main research interest is in statistical pattern classification and acoustic modelling for automatic speech recognition. He also worked on the DARPA funded Effective, Affordable and Reusable Speech-to-text (EARS) project from 2002-2005 and the Global Autonomous Language Exploitation (GALE) project between 2005-2006. He was also in the IIR team which participated in the NIST 2007 Language recognition Evaluation (LRE) and the NIST 2008 Speaker Recognition Evaluation (SRE). He is a recepient of the Google Faculty Research Award 2014.
Deep Learning for Stock Market Prediction
Abstract: It has been shown that news events influ- ence the trends of stock price movements. However, previous work on news-driven stock market prediction rely on shallow features (such as bags-of-words, named entities and noun phrases), which do not capture structured entity-relation informa- tion, and hence cannot represent complete and exact events. Recent advances in Open Information Extraction (Open IE) techniques enable the extraction of struc- tured events from web-scale data. We propose to adapt Open IE technology for event-based stock price movement pre- diction, extracting structured events from large-scale public news without manual efforts. Both linear and nonlinear mod- els are employed to empirically investigate the hidden and complex relationships be- tween events and the stock market. Large- scale experiments show that our event-based system out- performs bags-of-words-based baselines, and previously reported systems trained on S&P 500 stock historical data. We further use deep learning to improve the representation of event for enhanced generalization, and to capture the influence of long-term and short-term historical events simultaneously. These techniques boost the accuracy of index prediction from 60% to 66%, and significantly enhance the accuracy of individual stock prediction.
Biography: Yue Zhang is currently an assistant professor at Singapore University of Technology and Design. Before joining SUTD in July 2012, he worked as a postdoctoral research associate in University of Cambridge, UK. Yue Zhang received his DPhil and MSc degrees from University of Oxford, UK, and his BEng degree from Tsinghua University, China. His research interests include natural language processing, machine learning and artificial Intelligence. He has been working on statistical parsing, parsing, text synthesis, machine translation, sentiment analysis and stock market analysis intensively. Yue Zhang serves as the reviewer for top journals such as Computational Linguistics, Transaction of Association of Computational Linguistics and Journal of Artificial Intelligence Research. He is also PC member for conferences such as ACL, COLING, EMNLP, NAACL, EACL, AAAI and IJCAI. Recently, he was the area chairs of CLING 2014, NAACL 2015 and EMNLP 2015.
Apache SINGA: From a User's Perspective
Abstract: Attendees will be guided to train example deep learning models on their own machines using Apache SINGA. Particularly, we will train a convolutional neural network for image classification, a restricted Boltzmann machine for pre-training an auto-encoder model, and a recurrent neural network for language modelling. Each model is representative for one model category. Distributed training will also be conducted to compare with stand-alone training.
Note: Attendees are expected to bring your own laptops. You can download the VirtualBox Images (username: singa, passwd: singa) before the workshop, which have installed the dependent libraries of SINGA.
Biography: Sheng Wang is a fifth year PhD student in the Department of Computer Science, National University of Singapore. His research interests include log-structured systems, especially in indexing, query processing and data management. He is now a committer of Apache SINGA.
Biography: Wei Wang is a fifth year PhD student in the Department of Computer Science, National University of Singapore. His research interests include multi-modal data retrieval and distributed deep learning training. He is now a committer of Apache SINGA.