Code Design for Distributed Storage Systems with Awareness of Network Topology

Data in the world is expected to grow to 163 ZB by 2025, which is ten times the data generated in 2016. This rapid growth of new data needs to be stored, which creates a surging demand for data center services. In Hong Kong, the data center sector has been growing fast because of the robust telecommunication infrastructure and reliable power supply. Besides, the HKSAR government has been committed to fostering the city as the prime location for data centers in Asia Pacific. This project aims to develop novel techniques for data storage, which can benefit both local and global data center industries.The unprecedented growth of data poses a formidable challenge to the design of reliable storage systems. The traditional way of providing data reliability is by means of 3- replication, which means that a data file is replicated three times and stored in three different storage nodes, so that it can tolerate simultaneous failures of two storage nodes. Its storage efficiency, however, is only one third, which is too low to cope with the ever-increasing volume of data. To reduce storage cost, novel coding techniques are needed.In practical systems, storage nodes fail frequently. How to replace them at minimum cost is an open problem. Based on existing coding methods, it is often very costly to regenerate the lost portion originally stored in the failed nodes, as the repairing process requires decoding of the entire file by accessing the data in many storage nodes. To speed up the repair process, a mechanism called local repair, which requires the access of only a few storage nodes, has been proposed. The design of locally repairable codes is now of key importance and has attracted much attention from the research community recently.This project distinguishes itself from others in that it takes into account the underlying network topology when designing locally repairable codes. In the literature, most code designs are based on classical algebraic coding methods, which implicitly assume that all storage nodes are identical and fully connected with each other. In a data center network, however, the data centers are geographically distributed and mesh-connected, and the storage nodes within a data center are interconnected often in a tree topology via switches. This project will exploit the network topology. Storage codes with high reliability, low storage cost, and low repair cost will be delivered as the project outcome.