Architecture
The Go Scheduler service consists of three major components - http service layer, poller cluster and datastore.
-
Golang: The service layer and poller layer of the Go Scheduler service are implemented using the Go programming language (Golang). Golang offers high throughput, low latency, and concurrency capabilities through its lightweight goroutines. It is well-suited for services that require efficient memory utilization and high concurrency.
-
Cassandra: Cassandra is chosen as the datastore for the Go Scheduler service. It provides horizontal scalability, fault tolerance, and distributed data storage. Cassandra is widely used within Myntra and is known for its ability to handle high write throughput scenarios.
The service layer in the Scheduler service handles all REST traffic. It provides a web interface and exposes various endpoints for interacting with the service. Some of the important endpoints include:
- Register Client: This endpoint allows administrators to register a new client. A client represents a tenant, which is another service running its use case on the Go Scheduler service. Each client is registered with a unique ID.
- Schedule Endpoints: The service layer includes endpoints for creating schedules, cancelling schedules, checking status of the schedules, reconciling schedules etc. These endpoints are accessible only to registered clients.
The Scheduler service utilizes Cassandra as the datastore. It stores the following types of data:
- Schedule State Data: This includes the payload, callback details, and success/failure status after the trigger.
- Client Configuration Metadata: The datastore holds metadata related to client configurations.
- Poller Instance Statuses and Poller Node Membership: The status and membership information of poller instances are stored in the datastore.
The data layout in Cassandra is designed in a way that during every minute, the poller instance request goes to a single Cassandra shard, ensuring fast reads. The schedule table has the following primary key structure: (clientid, poller count id, schedule minute).
For example, the data for the use case mentioned earlier will look like:
C1, 1, 5:00PM, uuid1, payload
C1, 2, 5:00PM, uuid2, payload
...
C1, 50, 5:00PM, uuid50000, payload
Here, uuid1 to uuid50000 are unique schedule IDs.
In this data model, we perform 50,000 Cassandra writes and 50 Cassandra reads for the given use case.
It's important to note that not every schedule fire requires a read operation. The number of Cassandra reads in a minute is equal to the number of poller instances running for all clients. As Cassandra is well-suited for high write throughput and lower read throughput, this data modeling and poller design work effectively with the Cassandra datastore layer.
The Poller Cluster in the Scheduler service utilizes the Uber ringpop-go library for its implementation. Ringpop provides application-level sharding, creating a consistent hash ring of available Poller Cluster nodes. The ring ensures that keys are distributed across the ring, with specific parts of the ring owned by individual Poller Cluster nodes.
Every client within the Scheduler service owns a fixed number of Poller instances. Let's consider the total number of Poller instances assigned to all clients across all nodes as X. If there are Y clients where each client owns C1x, C2x, ..., CYx Poller instances respectively (where C1x + C2x + ... + CYx = X), and there are N Poller Cluster nodes, then each node would run approximately X/N Poller instances (i.e., X/N = C1x/N + C2x/N + ... + CYx/N).
The Poller Cluster exhibits scalability and fault tolerance characteristics. When a node goes down, X/N Poller instances automatically shift to the available N-1 nodes, maintaining the distribution across the remaining nodes. Similarly, when a new node is added to the cluster, X/(N+1) Poller instances are shifted to the new node, while each existing node gives away X/N - X/(N+1) Poller instances.
This approach ensures load balancing and fault tolerance within the Poller Cluster, enabling efficient task execution and distribution across the available nodes.
The GoScheduler follows a specific workflow to handle client registrations and schedule executions:
Clients register with a unique client ID and specify their desired poller instance quota. The poller instance quota is determined based on the client's callback throughput requirements.
Each poller instance fires every minute and is responsible for fetching schedules from the datastore. Each poller instance can fetch a maximum of 1000 schedules, with each schedule having a maximum payload size of 1KB. Assigning Poller Instances: When a client registers, they are assigned a specific number of poller instances. For example, if a client requires a callback requirement of 50000 RPM, they might be assigned 50 (50+x, where x is a buffer for safety) poller instances. These poller instances are identified by the client ID followed by a numeric identifier (e.g., C1.1, C1.2, ..., C1.50).
When a client creates a schedule, it is tied to one of the assigned poller instances using a random function that ensures a uniform distribution across all poller instance IDs. For example, if 50000 schedules are created with a fire time of 5:00 PM, each poller instance for this client will be assigned approximately 1000 schedules to be triggered at 5:00 PM. The schedules tied to each poller instance are fetched and triggered. The response to this callback is recorded and the status of the schedule is updated.
The GoScheduler can be horizontally scaled based on the increasing throughput requirements. For higher create/delete peak RPM, additional service nodes or datastore nodes (or both) can be added. Similarly, for higher peak callback RPM, the number of poller instances for a client can be increased, which may require adding new nodes in the poller cluster or datastore (or both). This scalability ensures that the service can handle increasing throughput by augmenting nodes in the service layer, poller cluster, and datastore.
Go-Scheduler has traditionally been used to handle one-time scheduling needs, where a client specifies a Unix timestamp for when a desired callback is triggered. However, the utility of Go-Scheduler has now been extended to support recurring schedules - tasks that need to run at regular intervals.
The introduction of recurring schedules required distinguishing between one-time schedules and recurring ones. To accomplish this, new tables were added to our Cassandra database to handle recurring schedules: recurring_schedules_by_id, recurring_schedules_by_partition, and recurring_schedule_runs.
-
recurring_schedules_by_id: As the name suggests, this table stores details of the recurring schedule, indexed by the schedule id. This allows for the retrieval and deletion of schedule details given a schedule id.
-
recurring_schedules_by_partition: This table indexes data based on the partition id. Each poller is assigned an id on startup, and during each run, it selects crons whose partition id matches with its own.
-
recurring_schedule_runs: This table maintains the various system-generated schedules and their corresponding mapping to the parent recurring schedule. The data is indexed on parent_schedule_id so that given a schedule id, the system can fetch all the runs corresponding to the recurring schedule.
The schema for these Cassandra tables can be found here.
The system saves each recurring task or cron job in two tables in our Cassandra database: recurring_schedules_by_id and recurring_schedules_by_partition. These tables maintain the details of each cron job. The partition-based table is specially designed to allow our pollers to distribute the load efficiently.
Go-Scheduler employs special app pollers to manage the recurring schedules. These pollers are tasked with querying the recurring_schedules_by_partition table to fetch the set of crons that are due to be triggered within a configurable time window. The app poller operates on a defined time window (which is configurable via a config file). For instance, if the poller operates on a 5-minute window, it filters the crons that need to be fired within the next 5 minutes. For each relevant cron, the poller creates a one-time schedule in the schedules table. The way these pollers work is the same as the approach used to create any other app in Go-Scheduler.
After identifying the relevant cron jobs, the poller creates a corresponding one-time schedule for each job in the schedules table. The system also documents the relationship between the newly created one-time schedule and its parent recurring schedule in the recurring_schedule_runs table.
Once the time to trigger a one-time schedule arrives, the Go-Scheduler sends the respective callback to the specified endpoint. The response to this callback is recorded and the status of the one-time schedule is updated. If the callback fails, the system retries the callback based on the configured retry policy.
By utilizing this approach, the Go-Scheduler is able to provide a highly scalable and efficient solution for handling recurring schedules while maintaining simplicity and clarity in its data model.