New Recommendation Model Demonstrates the Value of Data

A Fortune 500 vehicle retailer wanted a better way for its marketing analysts to conduct experiments and build models, beginning with generating customer product recommendations on its website. Working with CapTech, the team designed, developed, and deployed a data science and machine learning platform using Microsoft Azure. This platform transformed the previously slow and cumbersome recommendation workflow into a streamlined process that runs multiple times a day and delivers personalized results to customers. While enhancing customer engagement is a positive outcome, the platform also served as a demonstration of the system’s ability to perform big data analytics at scale, thus enabling the team to explore new insights.

The vehicle retailer’s team of analysts faced challenges in their exploration work due to limited internal IT resources. The retailer also lacked the necessary infrastructure to store and process large datasets, which prevented the analyst team from conducting real-time analysis. In the case of the product recommendation generator, the team was unable to access information about customer product views or inventory availability, making it difficult to create actionable recommendations. Additionally, the existing physical infrastructure was unable to support the algorithms.

Recognizing these obstacles, CapTech proposed a new cloud platform that could address real-time business needs and support a variety of analyst experiments. CapTech’s cloud-computing specialists leveraged their proven experience deploying systems in the cloud, establishing reliable frameworks for data ingestion, performing data engineering at scale, and following best practices in data science. The success of the platform was measured by its ability to provide the desired capabilities and be highly available and scalable – without requiring additional IT oversight.

CapTech delivered the solution in two phases.  During the first five-month phase, the team established the core services for data storage, processing, and orchestration. They also developed an ingestion framework to load information from five sources. In this phase, data scientists redesigned the product recommendation algorithm to run on the new platform.

In the following four-month phase, the team deployed services to ingest real-time streaming data and made additional enhancements to the platform’s model capabilities. Over this nine-month period, the team proved that ingesting new datasets could be done quickly and easily, provided the capability for iterative model deployments, and trained analysts on migrating their code to the new platform.  Overall, the new platform empowered the retailer’s marketing team to effortlessly execute their work.

At the end of Phase II, the platform contained over 30TB of data and reduced the product recommendation algorithm time by 80%. Other key impacts included:

• By co-locating the data, latency issues were virtually removed. Analysts found they could complete 15 modeling experiments per week, while only a few were previously possible.
• Analysts could combine the platform’s managed data with their own ingested datasets via a lab area, thus creating autonomy from IT.
• The platform enabled faster iteration on models and A/B testing – both activities the team cited as limitations before the project. 
• IT realized the value of using a modern data architecture.

By leveraging Azure’s billing dashboards, the team is now able to quantify the costs associated with the platform, which were previously untracked. This new capability paves the way for the team to introduce use cases based on a reduction of cost or a realization of new value.