Current Work
Hacking into current cloud systems:
- We conducted an in-depth performance study of MapReduce in its open source implementation, Hadoop. We identified four factors that have significant performance effect on MapReduce, and investigated alternative strategies for each factor. The experimental results show that it is possible to build a MapReduce-based system that is not only flexible and scalable, but also efficient.
- To achieve the support for analytic workload in MapReduce-based system, a query optimization scheme is also proposed. Specifically, we embed a query optimizer into Hive, which is designed to generate an efficient query plan based on our proposed cost model.
- We also propose a new cluster-based data warehouse system, called Llama. Llama is a hybrid data management system which combines the features of row-oriented and column-oriented database system to achieve efficient load and query processing performance.
- We have modified Hadoop (indexing support, footprint-based sorting and efficient parser), and the optimized performance of Hadoop is comparable with HadoopDB when processing analytic workload.
- We constructed an in-house commodity cluster awan for our system. The network topology of awan is illustrated in Figure 1, which basically belongs to the category of flat neighborhood networks. awan contains a single master node and 72 slave nodes, which are connected via three switches. The master node is responsible for all kinds of network administration services, such as gateway, nfs server, name server and dhcp server. The slave nodes are evenly divided into three racks and are used to accommodate our data storage system. The running status of awan cluster can be viewed from here.
- We have built a data storage system of epiC, which is essentially designed to operate on a large cluster of shared-nothing commodity machines. The epiC storage system employs both vertical and horizontal data partitioning schemes. In this hybrid scheme, columns in a table schema that are frequently accessed together in the query workload are grouped into a column group and stored in a separate physical table. This vertical partitioning strategy facilitates the processing of OLAP queries which often access only a subset of columns within a logical table schema. In addition, for each physical table corresponding to a column group, a horizontal partitioning scheme is carefully designed based on the database workload so that transactions which span multiple partitions are only necessary in the worst case. The architecture of the system is shown in the graph below:
- We also address the problem with OLTP queries and OLAP queries of high record selectivities: It is not efficient to perform sequential or parallel scan on the whole table to retrieve a few records. However, scanning the whole table is inevitable if query predicates do not contain attributes that determine the horizontal data partitioning scheme in the system. To handle this problem, we maintain various types of distributed secondary indexes over the data in epiC’s cloud storage system. This design facilitates different kinds of queries; for example, distributed hash indexes support single-dimensional exact queries, distributed B+-tree-like indexes support single-dimensional range queries, and distributed multi-dimensional indexes support multi-dimensional range and KNN queries. In practice, we cannot afford the cost of maintaining multiple overlays in the cluster for different types of distributed indexes.
- We develop an extensible, but simple and efficient indexing framework that enables users to define their own indexes without knowing the structure of the underlying network. It is also designed to ensure the efficiency of hopping between cluster nodes during index traversal, and reduce the maintenance cost of indexes.
Figure 1: Architecture of Awan Cluster
Figure 2: Overview of epiC Storage System