Serving Millions Of Personalised Offers To Our Players

RummyCircle has recently introduced a new offer construct to its players, coined as a money-multiplier. Core objective of MM offers is to optimize our spending and offer personalized experience. MM offer helps players to multiply their game winnings with a certain value and hence give an opportunity, to the players, to use their skill for better rewards. The offer is personalized for each user, backed and recommended by a DS model.

Normal Game

Game with MM

Player1 to Games24x7 — Sounds easy. Tell me, how can I get a multiplier ?
Game24x7 to the Player1 — First we will give you a challenge and you need to complete that challenge in the given time. If you are able to do so, we will give a money-multiplier card.

What is a Money Multiplier?

In money multiplier , we provide Multiplier-cards to users.Multiplier-cards basically has 3 sub constructs –

• Card value — for eg. 3x
• Earning Criteria — Win 3 cash games
• Earning Expiry — for eg. 30 minute

Design Considerations

We started with listing down our crucial flows. Main focus was to ensure the below for each of the flows:

• Can a given flow be made async and event driven ?
• Do we need strong consistency/eventual consistency for given flow ?
• What if the system goes down abruptly, how can we replay things for a given flow ?
• Ensuring things were getting reconciled eventually within the system for given flow.
• [Optional] We wanted our read and write operation to possibly remain segregated.

Journeys and Underlying trade offs

Write vs Read

Our use case required high write throughput. To scale write operation we evaluated below things:

• Write operation should deal with a min number of indexes.
• Checking possibility of doing batch based write operations whenever possible.
• Prefer append based write operations.

Command and Projection

Since we were designing event driven flows, we treated our constructs as commands. For example:

• Targeting is a command
• Tracking is a command
• Expiry is a command
• Earning/Redemption is a command.

This led us to design and build a set of micro-services which could handle commands that are flowing inside any modern distributed queue. Thus, we explored the Command and Query Responsibility (CQRS) Design Pattern like below

Micro Services Details

We were now clear on our design considerations and underlying feature construct. Next step was to identify microservices details with clear responsibilities. While coming up with services, we wanted to leverage and reuse platform services.

Trigger Service

• This was core service for our MM tech stack.
• This service was responsible for allowing business to create rule based segmentation for MM offer targeting and decorate segmentation with a complex schedule.
• Throughput expectation from the trigger service was to broadcast around 10K events/second.

Targeting Service

• Targeting service was supposed to be reactive in nature. Whenever a registered command is received on given transport, it reacts to commands.
• Core responsibility of targeting service was to run targeting steps atomically and scale based on triggered segmentation from trigger service.
• Targeting Service was also responsible to support scalable experimentation, ensure cappings on campaign budget and offer powerful MM offer recommendations to end users.
• Min throughput of the targeting service was to handle 5K targeting commands / second.

Tracking Service

• This service was again command based. On receiving user commands (User did add cash, won / lost the game, logins) , It checks registered earning criteria and processes them as per the underlying rule (rule has actions).
• This service maintains and tracks the criteria process and announces MM offers to the player when the criteria is met.
• Throughput of this service was around 10K tracking events / second.

Expiry Service

• This service was responsible for maintaining the lifecycle of targeted MM offers for each user across all the campaigns.
• This service runs a pool of coroutine based schedulers to check and expiry on top of the timeline stored in Mongodb.
• Once it finds out that a given offer has expired, it updates the player.
• Tracking Service and others react to expiry events.
• Throughput of this service was supposed to raise and maintain 10K/second expiries.

Offer Recommendation Service

• This service is built on top of python and backed by an intelligent data science model.
• This service registers business objectives and sets itself for offering MM offer recommendations accordingly.
• This service enjoys a scale equivalent to Targeting service.
• More on this part will come later as a separate blog.

Projection/Query Service

• This service receives heavy reads (5K/second)
• It registers and maintains various projections on top of mongo and exposes read-only API(s).
• This service subscribes to Targeting / Tracking / Expiry service to maintain projections.

Redemption Service

• This service maintains and runs the life cycle of MM inventory for a given user.
• It facilitates and orchestrates redemption of a given multiplier on a given game followed by offering rewards to the end user.
• It is backed by a scalable ledger service for wallet/money management.

At high level , Our interaction diagram of main services look like below:

High Level Architecture

At this moment , we had discovered our underlying services and their interaction. After several brainstorming and iterations, we settled on the below architecture.

Tech Stack

Scaling Notes

We wanted to do thorough load testing of each flow and service before we go into production. We divided our load test suit into below 2 theme

• Load test suite for realistic scenario.
• Load test suite for bullish scenario

While running load test , We encountered numerous challenges for below flows:

• Targeting Flow / Service
• Tracking Flow / Service
• Expiry Flow / Service

Scaling Notes of Targeting Journey

Targeting flow steps

• Targeting flow was write heavy and high throughput based flow.

Scaling and Optimisation Steps

Scaling Notes of Tracking Journey

Tracking Flows Steps

Challenge Tracking flow was the second major flow where we keep track of the current challenge count for a particular user.

While scaling up the tracking flow we had following observations –

As the time goes, the number of campaigns keeps increasing (300 campaigns). And a user may be part of multiple campaigns.

Scaling and Optimisation Steps

About the Author

This doc is the cumulative effort of Shwetank Tripathi and Sandeep Pandey. Shwetank is SDE(2) engineer in the platform team of Games24x7. He loves photography apart from work. On the other hand , Sandeep is working as EM with the RummyCircle backend team. Sandeep loves spending time with his daughter in free time.

‍