Anser: Adaptive Information Sharing Framework of AnalyticDB

The surge in data analytics has fostered burgeoning demand for AnalyticDB on Alibaba Cloud, which has well served thousands of customers from various business sectors. The most notable feature is the diversity of the workloads it handles, including batch processing, real-time data analytics, and unstructured data analytics. To improve the overall performance for such diverse workloads, one of the major challenges is to optimize long-running complex queries without sacri.cing the processing e.ciency of short-running interactive queries. While existing methods attempt to utilize runtime dynamic statistics for adaptive query processing, they often focus on speci.c scenarios instead of providing a holistic solution. To address this challenge, we propose a new framework called Anser, which enhances the design of traditional distributed data warehouses by embedding a new information sharing mechanism. This allows for the e.cient management of the production and consumption of various dynamic information across the system. Building on top of Anser, we introduce a novel scheduling policy that optimizes both data and information exchanges within the physical plan, enabling the acceleration of complex analytical queries without sacri.cing the performance of short-running interactive queries. We conduct comprehensive experiments over public and in-house workloads to demonstrate the e.ectiveness and e.ciency of our proposed information sharing framework.

Automated summary

The surge in data analytics has driven the demand for AnalyticDB on Alibaba Cloud, which excels in handling diverse workloads including batch processing, real-time data analytics, and unstructured data analytics. To improve overall performance, the challenge lies in optimizing long-running complex queries without compromising the efficiency of short-running interactive queries. To address this challenge, a new framework called Anser is proposed to enhance the design of traditional distributed data warehouses by incorporating an information sharing mechanism. This framework enables efficient management of dynamic information across the system, leading to novel scheduling policies that optimize both data and information exchanges within the physical plan.