Near Data Filtering for Distributed Database Systems

Over the past decade, data movement costs dominate the execution time of data-intensive applications for distributed systems and they are expected to be even more important in the future. Near data processing is a straightforward solution to reduce data movement which brings compute resources closer to the data source. This paper explores near data processing in a generic distributed system to improve the performance by reducing data movement. An efficient near data filtering solution is designed and implemented by introducing a filter layer which performs tuple-level near data filtering. In order to reduce idle time of processing nodes and improve data transmission throughput the proposed solution is extended to support block-level near data filtering by creating index for each data block. Furthermore, to answer the question when and how to perform near data filtering this paper proposes an adaptive near data filtering solution to balance the computation and data transmission throughput. Experimental results show that the proposed solutions are superior to the best existing method for most cases. The adaptive near data filtering solution achieves an average speedup factor of 4:59 for queries with low selectivity.