Accelerating Large-Scale Recommender Systems

Abstract

Deep learning-based recommender systems are widely used in modern online services to offer personalized recommendations. These recommendation models have grown to the terabyte- and even petabyte-scale, with large embedding tables. However, the size of these embeddings often exceeds the memory capacity for typical servers. Traditional recommendation inference that loads entire models into memory cannot support this scale. One potential solution is storing embedding tables on much larger solid-state drives (SSDs). Nonetheless, substituting SSDs for DRAM naively could significantly reduce system throughput and introduce long tail latency.

This thesis aims to address the performance degradation of SSD-based recommendation inference from a storage system perspective. The issue is mainly attributed to two factors: the long I/O stack and the significant read amplifications of block-interface SSDs. Additionally, introducing slow SSD storage makes it challenging for a recommender system to maintain low tail-latency, such as 99th percentile latency because inference requests are not completed until all queries are finished. Even if only one inference in a batch misses the embedding cache and must access SSD embeddings, it can negatively impact the rest of the batch. The existing scheduling scheme is unaware of this problem and becomes suboptimal when handling mixed recommendation inference requests.

Based on these observations, this thesis proposes FlashEmbedding as a novel recommender system architecture. It leverages SSD as the embedding storage and DRAM as the embedding cache to improve overall performance. FlashEmbedding leverages an embedding semantic-aware SSD, an embedding-oriented software cache, and pipeline techniques to improve the overall performance. The proposed SSD embedding storage supports both embedding vector lookups and normal I/O requests by taking advantage of two-stage fined-grained read and MMIO interface. The proposed DRAM embedding cache further reduce the accesses to slow SSD by applying the bitmap filter and robin hood hashing-based LRU cache.

This thesis also introduces Rabbitail, an inference scheduler that addresses the issue of long tail latencies in large-scale recommender systems using a hierarchical DRAM embedding cache and SSD embedding storage. The approach classifies recommendation inferences into two categories: hit inferences and miss inferences, which are processed differently based on their embedding cache lookup results. For miss inferences, Rabbitail uses an on-demand embedding table lookup strategy and a tail latency-aware reordering mechanism to optimize the retrieval process of embeddings stored on SSD. For hit inferences, Rabbitail utilizes priority and parallelism at multiple levels to efficiently execute MLP tasks while controlling the longest execution time by splitting large batch sizes of MLP operations.

We evaluate the performance of FlashEmbedding with our FPGA-based prototype SSD on a real-world public dataset. FlashEmbedding delivers up to 2.89x lower end-to-end latency than baseline solution in a memory capacity-limited system. Our evaluation of Rabbitail with state-of-the-art recommendation models shows that Rabbitail reduces the 99th percentile latency by up to 53.7% while maintaining the same throughput compared to the baseline.

In conclusion, this thesis aims to explore the potential of constructing large-scale recommender systems using SSDs in scenarios where memory capacity is restricted. Our experimental results indicate that the methods proposed in this thesis can effectively tackle the issue of limited memory capacity in recommender systems.

Date of Award	26 Apr 2023
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Chun Jason XUE (Supervisor)