Discover FamPay's Engagement Service: Missions

What is engagement and why does it matter?

Engagement plays a pivotal role in ensuring product retention, as it incentivises users to keep coming back for more. At FamPay, our target audience consists of GenZs who are typically the CTOs of their households and are known to have a competitive streak. They’re always looking for the best way possible to make payments fun and engaging. Our feature, Missions, achieves this by introducing challenges that require our users to use FamPay's services. By completing these missions, users are rewarded in numerous ways - with FamCoins, monetary rewards, gift cards, and other incentives.

What do Missions look like?

Spend X amount using FamCard in Y duration and win Rs Z

Completing a mission on FamPay involves spending a specified amount on any of our services or a third-party platform using FamPay as the payment method within a certain timeframe to earn a reward. Sounds easy? It would be if we were handling just one case. However, making this feature modular for the product team to customise, and scalable to handle millions of transactions while maintaining external dependencies and a seamless user experience presents significant engineering challenges. These complexities are further compounded by FamPlay, the underlying engagement service, which requires modification and integration to avoid any negative impact on user flow, making it even more challenging for our engineers.

In this blog post, we will break down how this engagement service at FamPay is built.

The core workings

We’ll begin by providing an overview of the entire service. Afterwards, we'll analyse its individual components in detail.

Architecture for the engagement service — Missions

By looking at the architecture, we can divide the service into these main components:

• User-Facing APIs
• Internal panel APIs
• Event stream handler
• Task scheduler and worker

User-facing APIs

User-facing APIs allow direct user interaction with the system, including data retrieval and challenge progress tracking. These APIs are secured by our Auth Service, which ensures that all the requests that reach the server are legitimate and no malicious users can access the system.

Building these APIs was relatively straightforward, given that the data and processing required for fetching user data is comparatively simple.

They include APIs like fetching user progress, reward status, mission details, etc.

Internal Panel APIsThe APIs enable the product team to manage Missions with basic CRUD operations. However, ensuring that the backend data schema can handle a diverse range of filters with minimal overhead while creating and filtering Missions was the challenging part. Here's a sample of some Mission filters that we need to support.

[{
  id:"amount",
  operator:"gte",
  values: 100
},{
  id:"merchant",
  operator:"eq",
  values: 'swiggy'
}]

The filters can be multiple or single, and are stored in an array format and follow the AND rule, meaning all filters must be valid for the transaction to be counted as valid. For example, a filter may check if the transaction value is greater than or equal to 100, if not, it fails, and if so, it proceeds to check if the merchant is Swiggy. Only if both are true, it updates the Mission progress. There are additional processes involved in marking Missions as successful for a user, but that's beyond the scope of this blog.

Other internal APIs include creation of rewards, handling ads, managing brands, and more.

Event stream handlerAt FamPay, our backend is composed of microservices that utilise Kafka as our primary inter-service communication tool. While event stream handling and task scheduling are closely related, we'll cover each topic separately to provide a more comprehensive understanding.

In FamPay's backend architecture, the event relay service acts as the connection hub between services, ensuring data integrity at all times. Meanwhile, the event stream handler, which is implemented in the engagement service, maintains a ledger of all processed events and performs pre-processing to guarantee that data is consistently in the correct format. To illustrate, our primary transaction service emits events for various monetary transactions, and it's the handler's responsibility to inform the scheduler of the event type (e.g., payment, reversal) in order to create and schedule relevant tasks. Additionally, the event stream handler safeguards the service from malicious events.

The reason for handling reversals is to ensure that malicious users don’t take advantage of the feature by reversing their transactions with our partners while still obtaining the rewards.

Task scheduling

Task scheduler and worker

In order to manage the daily event load and scaling requirements of our applications, we've created several internal libraries tailored for event handling. These libraries include:

1. Nimble
2. Mammoth

Let’s dive deeper.

Nimble

Nimble is FamPay's custom-built task scheduler and runner, utilising Redis and bullMQ as its foundation. Our Nimble implementation is optimised for handling micro tasks that require low latency and fast task switching. While bullMQ offers a wide range of task scheduler features, we chose to implement Nimble to provide custom error handling and unique functions. Nimble's task queue works efficiently in-memory and handles sequential tasks that need to be executed, such as real-time score updates, transaction checks, and notification sending for a single event and record.

Mammoth

Mammoth is a powerful library designed to handle heavy operations that require significant processing power. It excels in processing tasks that involve bulk data processing, where multiple records that require similar processing can be batched together in a single task. These tasks are then scheduled at specific intervals, optimising resource utilisation and ensuring efficient processing of events. This library is built on top of Agenda, which uses a database internally. Since these tasks are not latency-sensitive, they can be queried from the database without causing any significant delays or compromising the SLA. These tasks include processing rewards, updating the status of missions and challenges, and other similar operations that require considerable processing power.

While Agenda and BullMQ provide most of the out-of-the-box features, we needed certain features to handle our custom logic like:

• Data persistence : This is crucial in some tasks, while in others, it may not be necessary. To achieve fine-grained control over which data should be persisted and which should not, it is important to implement custom logic.
• Retries : Although both Bull and Agenda offer built-in retry mechanisms, it's crucial to have control over the number of retries each task should receive.
• Exponential backoff :  When a task fails, it's often due to the unavailability of the variables it needs for processing, which may become available after a few seconds. This delay in data availability can be addressed through a technique known as exponential backoff. Unfortunately, this feature is not currently available in the open-source versions of Bull or Agenda.

Exponential backoff

By combining exponential backoff with a custom number of retries, developers can achieve precise control over the tasks they create and schedule using these libraries.

The task scheduler and runner operate on a separate server, known as the worker server, from the original API server. We have optimised the worker servers to handle more than a million events per hour, ensuring that the system can efficiently handle a high volume of tasks.

The engagement service is now live for users to use and explore. It can be found in the rewards section within our app or on the home screen. We're currently rolling it out to users in batches, so if you don't see it yet, please be patient as we're working to make it available to everyone as soon as possible. With this new service, you can expect a variety of rewards and benefits, so stay tuned for more exciting updates.

To learn more about Engineering at FamPay, check out our Tech blog and our fampay.in site. If you’d like to help us bring ambitious ideas to life, explore our Careers page.

Written by Jugal Bhatt