In May of this year, we at SphereEx proposed the Database Mesh 2.0 concept. Database Mesh is a dynamic concept that is constantly evolving, focusing on database traffic governance and providing sharding, load balancing, observability, and audit capabilities based on database protocol awareness. These capabilities address some of the traffic governance issues. Furthermore, Database Mesh emphasizes the development of database reliability engineering (DBRE), providing easier-to-use and superior database governance capability.

iQiyi shares our view for the concept and vision of Database Mesh, which is to achieve high-performance database expansion while tackling data governance issues in the cloud.

iQiyi expanded ShardingSphere-JDBC based on its business requirements and conducted a series of tests combined with Pisanix, for the implementation of the Database Mesh concept.

How did iQiyi prepare to go cloud-native with Database Mesh & ShardingSphere-JDBC?

With the expansion in the number of features, products & service offerings and the surge in the number of users, enterprises have diversified the number of promotional activities to engage and retain users (flash sales, events, etc.). In successful cases, this led to a huge amount of traffic putting great pressure on their databases.

As a result, enterprises encounter database issues such as secondary delays and slow queries, with some operations failing to meet business requirements. Microservices and cloud-native bring new possibilities for the business roll-out process and governance. But with more diversified business scenarios and stovepipe data application schemes, data control tends to be isolated. Tech teams face problems such as difficult technology selection, high costs, and complicated management and control.

Particularly, cloud-native architecture is growing mature, and the relationship between business applications and database infrastructure is changing gradually. iQiyi hopes to grasp this new trend and adopt unified management to expand and update databases, thus supporting more businesses and applications migrating to the cloud.

To meet the requirements for database performance and availability in cloud environments, iQiyi needed to migrate ShardingSphere's local distributed capability to the cloud. To achieve this, iQiyi was looking for a tool that can unify cloud database traffic access in a cloud-native environment and achieve the unified and efficient management of cloud traffic and data.

While investigating and testing Pisanix, a Database Mesh solution provided by SphereEx, iQiyi also redeveloped ShardingSphere-JDBC, to meet the requirements for sharding, load balancing, configuration and storage, and security when accessing businesses to database governance platforms.

1. Preparation: iQiyi transforms ShardingSphere-JDBC

Currently, iQiyi uses a unified config center to store database connection configuration. KMS technology is used to encrypt database access configuration and ShardingSphere-JDBC is used to implement sharding and load balancing. The complete architecture is shown below.

When businesses are connected to the data governance platform, they apply for relevant connection configuration. After they are transformed and the access information is encrypted via KMS, they are stored in the unified configuration storage center. When the application starts, the transformed ShardingSphere-JDBC fetches the configuration and monitors configuration changes to support hot configuration updates.

Before the transformation, when there was a need to change the configuration, scale-out sharding clusters, upgrade cluster version, or migrate a database to the cloud, it usually required the release of a new version. Also, the DevOps teams had to design complicated procedures such as switchover, rollback, timing selection, grayscale traffic, and data verification, to account for various scenarios.

After the transformation, the customized ShardingSphere-JDBC can support sharding cluster scaling or binding changes when adding or modifying table sharding configuration. In the configuration center, you can perform visual operations to modify configurations or bind clusters, and select the configuration of reload timing. When the SDK receives the latest configurations, it starts asynchronous tasks to close the old connection pool and replace the existing one. This facilitates the smooth migration of read/write traffic and greatly simplifies the migration of data governance capabilities to the cloud environment.

iQiyi plans to introduce Pisanix-Proxy by accessing Database Mesh, further sinking data governance capability from SDK to Sidecar.

2. Data governance capability with Sidecar and building a unified data governance based on Pisanix

In the traffic access layer, as cloud-native applications move closer to microservices and Serverless, users need to configure complex routing rules, support multiple application-layer protocols, and ensure service access security and the observability of traffic. In response to these requirements, iQiyi used middleware to manage Redis and MySQL at the very beginning.

Additionally, SmartJedis provided a unified configuration center to support iQiyi's hybrid cloud deployment. In the unified configuration center, configurations in different environments could be dynamically supported. In a non-mesh environment, a direct connection is adopted; while in a mesh environment, RedisProxy in Envoy is used to manage Redis protocol traffic and support hot updates of connection configuration, avoiding downtime after Redis is moved to the cloud.

In terms of MySQL, iQiyi's R&D team tested Pisanix, the specific implementation of Database Mesh. Written in Go and Rust for the Kubernetes environment, Pisanix currently supports MySQL. It includes three components: Pisa-Controller, Pisa-Proxy, and Pisa-Daemon, which provide a local database for users and applications. It supports multi-protocol pluggable architecture, shields the status of real data sources, and provides unified database traffic control capabilities for data DevOps teams.

Currently, iQiyi still uses ShardingSphere-JDBC to support Java applications. Once Pisanix will be further implemented by iQiyi, the company will implement standardized automatic database maintenance via Pisanix, and achieve the cloud-native orchestration of multiple database governance behaviors by supporting multi-language applications. Based on Database Mesh's standard CustomResourceDefinition, such as unified database access declaration configuration and programmable database access resource limitations, iQiyi can rapidly achieve the governance and orchestration of cloud-native databases.

3. iQiyi's plan for Pisanix

1）Data sharding: achieve high performance on par with ShardingSphere-JDBC in the cloud

Data sharding is an effective way to deal with massive data storage and computing, which is why iQiyi chose Pisanix for cloud-native and non-Java scenarios. Data sharding mainly includes four modules: SQL parser, SQL rewriting, SQL router, and result merger.

To facilitate the migration of ShardingSphere's powerful local sharding capability to the cloud, Pisanix provides data sharding governance capabilities in the cloud based on the underlying database, allowing users to achieve horizontal scaling computing through Pisanix. At the same time, more custom metrics are available to achieve intelligent, stable, and advanced auto-scaling for Pisa-Proxy.

Based on the Pisa-Controller plane, iQiyi can achieve the management and control of data plane components. Pisa-Proxy can also be combined and deployed in the same Pod with business applications in Sidecar mode to monitor MySQL protocol and obtain the traffic of applications accessing the database. Pisanix also provides iQiyi with a variety of governance capabilities:

• SQL traffic governance: achieve multiple load balancing strategies and current limiting by paring SQL.
• Access control: achieve fine-grained permission control based on the relationship between users and data permission.
• SQL firewall: prevent high-risk SQL from executing.
• Observability: expose various database access metrics such as throughput and latency.

From iQiyi's point of view, Pisanix enables the high-performance sharding of both Java and non-Java services in the cloud environment. This achievement lays the foundation for the smooth transition of more businesses.

2）Read/write splitting: increase database throughput.

To improve throughput and availability, many systems adopt a primary-secondary database architecture configuration mode, which is a bit complicated. Therefore, when read requests outnumber write requests, read/write splitting should be used to overcome the performance bottleneck of the database in real-world application scenarios.

Read/write splitting is a widely used technical solution to improve throughput in primary-secondary scenarios, and is capable of improving query performance and reducing server load. It also brings the same problem with data sharding, which makes it more complicated for DevOps teams to operate databases.

Currently, iQiyi evenly distributes query requests to multiple data copies through the configuration mode of one primary and multi-secondary, which improves the processing capability of the system. This method improves throughput and the availability of the system - even when a database breaks down or a disk is physically damaged, the system can still maintain normal functioning.

iQiyi plans to adopt Pisanix's dynamic read/write splitting feature to manage multi-primary and multi-secondary database clusters. After connecting to Pisanix, iQiyi will be able to transparently manage the primary/secondary database with read/write splitting so that users can use the database with the primary/secondary architecture just like a monolithic database.

Future plan

Currently, iQiyi has completed its internal transformation for ShardingSphere-JDBC. In the future, it plans to combine Pisanix and ShardingSphere to achieve the unified governance of MySQL.

Driven by the ShardingSphere and Database Mesh communities, Pisanix will continue to develop cloud solutions to meet various usage scenarios, with SphereEx providing reliable technical support for iQiyi and accelerating the transition speed to the cloud.

Pisanix is a very young project, which means there are some shortcomings. iQiyi's test shows that Pisanix is limited in its expression support for database and table sharding, and its special configuration for SQL needs to be further improved.

Next, the community will focus on improving Pisanix's online capabilities, including operating status visualization, metrics, circuit breaker degradation strategy, and tracing. Additionally, SQL audit, Pisa-Controller's merge with Istio, and other issues related to compatibility and performance have also been put on the agenda.

In the coming future, iQiyi will build a MySQL-based unified data access standard and solution based on ShardingSphere-JDBC and Pisanix that is still evolving under the Database Mesh concept.

Through a unified configuration center and customized Sidecar, iQiyi will gradually make the database access details fully transparent to developers. This way, it can simplify the operating process while enhancing the security of database access, simplifying moving applications moving to the cloud.

For more information about Database Mesh and Pisanix, follow the links below: Database Mesh 2.0: Database Governance in a Cloud Native Environment Pisanix is Available! An Open Source Database Mesh Solution Launched by SphereEx Pisanix GitHub