A New Redis Sync Tool

Open Source

We have open-sourced a tool for Redis data synchronization. Please refer to GitHub for details.

Why Do We Need Real-Time Data Synchronization with Redis?

For important data, we often deploy database in multiple IDC to avoid data loss or inaccessibility due to data center failures. We also face the challenge of cross-data center data synchronization. Now, we have developed a real-time sync tool for Redis data.

Our vision is to develop the tool into a distributed system for Redis data governance, aiming to address the data governance challenges of Redis. We named this system GunYu, derived from the mythological story of Gun and Yu's governance of water disasters.

What Features Does GunYu Provide?

From product perspective, let's compare Redis-GunYu with several mainstream tools:

What other advantages does GunYu have?

• Minimal impact on stability
  • Ingest source: Specify whether to sync data from a slave, master or prefer slave
  • Local cache + resuming from breakpoints: Minimizes the impact on the source Redis
  • Splits big keys of RDB to synchronize them
  • Lower replication latency
• Data security and high availability
  • Local cache supports data verification
  • High availability of the tool: Supports master-slave mode, self-election based on the latest records, automatic and manual failover; the tool is P2P architecture, minimizing downtime impact
• Fewer restrictions on Redis
  • Supports different deployment modes of Redis on the source and target, such as cluster or standalone instances
  • Compatible with different versions of Redis on the source and target, supports from Redis 4.0 to Redis 7.2
• More flexible data consistency strategies, automatic switching
  • When the shards distribution of the source and target is the same, batch writes in pseudo-transaction mode, and offsets are updated in real-time, maximizing inconsistent
  • When the shard distribution of the source and target is different, offsets are updated periodically
• User-friendly for dev-ops
  • API: supports HTTP API, such as full sync, checking synchronization status, pausing synchronization, etc.
  • Monitoring: Rich monitoring metrics, such as replication latency metrics in time and space dimensions
  • Data filtering: Filter by certain regular keys, databases, commands, etc.
  • Redis topology: Real-time monitoring of topology changes in the source and target Redis (e.g., adding/removing nodes, master-slave switch, etc.), to change consistency strategies and adjust other functional strategies

Under the Hood

Synchronization Principle

For each source Redis node, redisGunYu maintains a dedicated pipeline with the following structure:

• Input: Impersonates a Redis slave to synchronize data from the source Redis node.
• Channel: Local cache, currently only supports storage in the local file system.
• Output: Sends RDB and AOF data of the source Redis to the target Redis.

High Availability Architecture

For each source Redis node, redisGunYu has a dedicated pipeline. Each pipeline independently elects a leader. The redisGunYu nodes form a P2P architecture, where they act as leader and follower for each other. The leader is elected based on the latest cached data and it impersonates a Redis slave to synchronize data from the source Redis node to the target node. Simultaneously, the data is sent to the followers. This P2P structure minimizes the impact of tool failures.