16 June 2017 Department of Computer Science Faculty Student Research

 

16 June 2017 – Dr. Wang Wei is the recipient of NUS Computing’s Doctoral Dissertation Award 2017, for his thesis entitled “A Training Framework and Architectural Design for Distributed Deep Learning”.

Supervisors nominate their students’ works to the evaluation panel, which considers the significance, novelty and expected impact of the research presented, and the award is presented to the doctoral student with the best PhD thesis. Dr. Wang was supervised by Professor Ooi Beng Chin and joined the Department of Computer Science as an Assistant Professor, in March.

 

About Dr. Wang’s research:
Deep learning made a break-through in image classification in 2012. Since then, various deep learning models have been proposed for real-life applications (like machine translation and speech recognition) by both academia and industry (e.g, Facebook, Google). It is also one of the major factors that fuelled the proliferation of AI in recent years.

However, it is tricky and slow to train deep learning models due to the high demand on computational resources. In view of this challenge, Wang Wei's thesis focuses on the architectural design of a distributed deep learning system, which is essential for accelerating the training speed of large deep learning models since the speed of a single device (e.g, CPU or GPU) is almost saturated.

This thesis proposed a flexible architecture for distributed training with different consistency models [1,2,3]. The abstractions are extensible for different hardware and optimization techniques. The implemented system, called SINGA, has shown good scalability on both CPU and GPU devices.

SINGA [8] is the first deep learning project accepted by Apache incubator (the world‘s largest open-source organization). It is also Apache's only project in Singapore. SINGA provides a platform for further research on distributed training of deep learning and machine learning models. Various applications have been developed on top of SINGA, including a food recognition App [6] by IDMI of NUS, and a medical image analysis system for hospitals in Zhejiang, China. The startup company Shentiliumn [7] uses SINGA as the core deep learning engine for financial data analysis, e.g., financial news understanding.

In addition, this thesis proposed deep learning models [4, 5] for processing data from multiple modalities, whereas traditional deep learning models were focused on single modal data, e.g, images or text separately. The proposed approaches outperformed the state-of-the-art algorithms significantly and generated many follow up research works.
The papers [3, 5] related to this thesis were nominated for the Best Paper Award of ACM Multimedia 2015 and VLDB 2014 respectively.

References:
1. W. Wang, et al.. Deep learning at scale and at ease. TOMM, 2016
2. W. Wang, et al. Database meets deep learning: Challenges and opportunities. ACM SIGMOD Record, 2016
3. W. Wang, et al. SINGA: putting deep learning in the hands of multimedia users. In ACM Multimedia, pages 25–34, 2015
4. W. Wang, et al. Effective deep learning-based multi-modal retrieval. VLDB J, 25(1):79–101, 2016
5. W. Wang, et al. Effective multi-modal retrieval based on stacked auto-encoders. PVLDB, 7 (8):649–660, 2014
6. Food recognition App. https://foodlg.nusidmi.com/
7. Fin-tech startup, Shentilium https://shentilium.com/
8. Apache SINGA, http://singa.apache.org