HRaft : Adaptive Erasure Coded Data Maintenance for Consensus in Distributed Networks

Research output: Chapters, Conference Papers, Creative and Literary WorksRGC 32 - Refereed conference paper (with host publication)peer-review

4 Scopus Citations
View graph of relations

Author(s)

Related Research Unit(s)

Detail(s)

Original languageEnglish
Title of host publicationProceedings - 2022 IEEE 36th International Parallel and Distributed Processing Symposium, IPDPS 2022
PublisherInstitute of Electrical and Electronics Engineers, Inc.
Pages1316-1326
ISBN (electronic)978-1-6654-8106-9
ISBN (print)978-1-6654-8107-6
Publication statusPublished - 2022

Publication series

NameProceedings - IEEE International Parallel and Distributed Processing Symposium, IPDPS
ISSN (Print)1530-2075
ISSN (electronic)530-2075

Conference

Title36th IEEE International Parallel & Distributed Processing Symposium, IPDPS 2022
LocationVirtual
PlaceFrance
CityLyon
Period30 May - 3 June 2022

Abstract

Distributed data services usually rely on consensus protocols like Paxos and Raft to provide fault-tolerance and data consistency across global and local-distributed data centers. Erasure coding replication has appealing storage and network cost saving compared with full copy replication, which helps consensus protocols achieve low latency, high fault tolerance, and high throughput for data access. Applying erasure coding in consensus protocols directly will degrade the liveness level when the number of failure servers reaches a certain level. To address the challenge, CRaft just stores full copy replication instead of erasure coding replication when the number of failed servers reaches a certain threshold. In such situation, CRaft will be downgraded sharply to the same storage and network costs as Raft. To overcome the shortcoming of CRaft, we propose a protocol, called HRaft, which can adapt the placement of data blocks in order to always have enough blocks to recover the stored value when servers fail. By replenishing some coded blocks in healthy servers instead of full copy replication, it can avoid switching to the full replication when a certain threshold on the number of failures is reached. We designed and implemented a key-value (KV) storage prototype to validate the proposed protocol and evaluate its performance. The experimental results show HRaft can significantly reduce storage and network costs and improve write performance while keeping the liveness level compared to CRaft.

Research Area(s)

  • Consensus protocol, Erasure coding, Fault tolerance, Network storage, Paxos, Raft

Citation Format(s)

HRaft: Adaptive Erasure Coded Data Maintenance for Consensus in Distributed Networks. / Jia, Yulei; Xu, Guangping; Sung, Chi Wan et al.
Proceedings - 2022 IEEE 36th International Parallel and Distributed Processing Symposium, IPDPS 2022. Institute of Electrical and Electronics Engineers, Inc., 2022. p. 1316-1326 (Proceedings - IEEE International Parallel and Distributed Processing Symposium, IPDPS).

Research output: Chapters, Conference Papers, Creative and Literary WorksRGC 32 - Refereed conference paper (with host publication)peer-review