Abstract
Traditional distributed graph processing systems are generally designed for static graph. However, common approaches for these systems are no longer suitable for time-evolving graph, it will lead to problems like load unbalance and higher memory usage. Moreover, heterogeneous resources among worker nodes increase the degree of imbalance workload, further have a decrease in system performance. In this work, we present a distributed graph processing system DASH that adopts dynamic graph data loading to reduce memory usage and avoid graph pre-partition time. We also use asynchronized and incremental computing to have faster algorithm convergence, and we have a workload-aware task scheduling method to have more balanced load in the system. Our evaluations using real world datasets show that the architecture of DASH and the approaches we adopt has more than 3x speedup in homogeneous resources and 8x speedup in heterogeneous resources compared to another well-designed distributed graph processing system GPS. Besides execution time, DASH also saves 60% of memory usage since it doesn’t need to load the whole graph partition into worker nodes.