This is a guest post co-authored by Rama Badrinath, Divay Jindal and Utkarsh Agrawal at Meesho.
Meesho is India’s fastest growing e-commerce company, with a mission to democratize internet commerce for everyone and make it accessible to India’s next billion users. Meesho was founded in 2015 and today focuses on buyers and sellers across India. Meesho Marketplace provides micro, small and medium enterprises and sole traders with access to millions of customers, selection from 30+ categories and 900+ subcategories, pan-India logistics, payment services and customer support capabilities to effectively manage their businesses on the Meesho ecosystem.
As an e-commerce platform, Meesho aims to improve user experience by providing personalized and relevant product recommendations. We wanted to create a generalized feed ranking that takes into account individual preferences and historical behavior to effectively display products in each user’s feed. Through this, we wanted to drive user engagement, conversion rates and overall business growth by tailoring the shopping experience to each customer’s unique needs and providing the best value for money.
We used AWS machine learning (ML) services as Amazon SageMaker develop a powerful generalized food classifier (GFR). In this article, we discuss the key components of GFR and how this ML-based solution has streamlined the ML lifecycle, ensuring effective infrastructure management, scalability and reliability within the ‘ecosystem.
Solution Overview
To personalize user feeds, we analyzed extensive historical data, extracting feature insights that include browsing patterns and interests. These valuable features are used to build ranking models. GFR personalizes each user’s feed in real-time, taking into account various factors such as geography, previous purchasing habits, acquisition channels, etc. Several interaction-based features are also used to capture user affinity toward an item, item category, or item properties such as price, rating, or discount.
Several user-independent item-level features and scores are also used. These include an item popularity score and a propensity score to purchase an item. All these features serve as input to the Learning to Rank (LTR) model which attempts to output the probability of click (PCTR) and probability of purchase (PCVR).
For diverse and relevant recommendations, GFR searches for product candidates from multiple channels, including exploitation (known user preferences), exploration (new and potentially interesting products), popularity (trending items), and recent ones (latest additions).
The following diagram illustrates the GFR architecture.
The architecture can be divided into two different components: model training and model deployment. In the following sections, we discuss each component and the AWS services used in more detail.
Model training
Meesho used Amazon DME with Apache Spark to process hundreds of millions of data points, depending on the complexity of the model. One of the main challenges was organizing distributed training on a large scale. We used Dask, a distributed computing framework for data science that integrates natively with Python libraries, on Amazon EMR to scale training tasks across the entire cluster. Distributed training of the model reduced training time from days to hours and allowed us to schedule Spark jobs efficiently and cost-effectively. We used an offline feature store to keep a historical record of all feature values that will be used for training the model. Model artifacts from training are stored in Amazon Simple Storage Service (Amazon S3), providing convenient access and versioning.
We used a temporal sampling strategy to create training, validation, and testing datasets for model training. We tracked various metrics to evaluate model performance, the most important being the area under the ROC curve and the area under the precision-recall curve. We also followed model calibration to avoid overconfidence and underconfidence issues while predicting probability scores.
Model deployment
Meesho used SageMaker inference endpoints with autoscaling enabled to deploy the trained model. SageMaker offered ease of deployment with support for various ML frameworks, allowing models to be served with low latency. Although AWS offers standard inference images suitable for most use cases, we created a custom inference image that specifically meets our needs and pushed it to Amazon Elastic Container Registry (AmazonECR).
We built an in-house A/B testing platform that made it easy to monitor live A/B metrics, allowing us to quickly make data-driven decisions. We also used the A/B testing SageMaker functionality to deploy multiple production variants to an endpoint. Through A/B experiments, we observed an improvement of approximately 3.5% in the platform’s conversion rate and an increase in the frequency with which users opened apps, highlighting the effectiveness of this approach.
We tracked various drifts such as feature drift and prior drift several times a day after model deployment to prevent model performance from deteriorating.
We used AWS Lambda to configure various automations and triggers required during model retraining, endpoint updates, and monitoring processes.
The recommendation workflow after model deployment works as follows (as shown in the solution architecture diagram):
- Input requests with user context and interaction features are received at the application layer from Meesho’s mobile and web application.
- The application layer retrieves additional features such as user historical data from the online feature store and adds them to input requests.
- Added features are sent to endpoints in real-time to generate recommendations.
- The model’s predictions are fed back to the application layer.
- The application layer uses these predictions to personalize user flows on the mobile or web application.
Conclusion
Meesho successfully implemented generalized feed ranking using SageMaker, resulting in highly personalized product recommendations for each customer based on their preferences and historical behavior. This approach significantly improved user engagement and led to higher conversion rates, contributing to the company’s overall business growth. By using AWS services, our ML lifecycle execution time was significantly reduced from months to just weeks, resulting in increased efficiency and productivity for our team.
With this advanced feed ranking, Meesho continues to deliver tailored shopping experiences, adding more value to its customers and fulfilling its mission to democratize e-commerce for everyone.
The team is grateful for the continued support and guidance of Ravindra YadavDirector of Data Science at Meesho, and Debdoot MukherjeeHead of AI at Meesho, who played a key role in this success.
To learn more about SageMaker, see Amazon SageMaker Developer Guide.
about the authors
Utkarsh Agrawal currently working as a Senior Data Scientist at Meesho. He has previously worked with Fractal Analytics and Trell on various areas including recommender systems, time series, NLP, etc. He holds a master’s degree in mathematics and computer science from the Indian Institute of Technology Kharagpur (IIT), India.
Rama Badrinath currently working as a Senior Data Scientist at Meesho. He has previously worked with Microsoft and ShareChat on various areas including recommendation systems, image AI, NLP, etc. He holds a master’s degree in machine learning from the Indian Institute of Science (IISc), India. He has also published papers in renowned conferences such as KDD and ECIR.
Divay Jindal currently working as a Lead Data Scientist at Meesho. He previously worked with Bookmyshow on various areas including recommendation systems and dynamic pricing.
Venugopal Pai is a solutions architect at AWS. He lives in Bangalore, India, and helps digitally native customers scale and optimize their applications on AWS.
