Strange Loop

Realtime Hybrid Reinforcement Learning at Scale

Next best action (NBA) is a technique that takes unique user history and characteristics into consideration and recommends the next actions that help the customer progressing towards business goals as quickly and smoothly as possible. It is not easy to design such a AI powered NBA engine. Ideally a hand-free NBA engine needs to handle the following problems. a) It should deal with incomplete historical feedback that are skewed towards a small set of actions; b) It should adapt to dynamic actions, which can be added or removed frequently due to seasonal changes or shifts in business strategies; c) It needs to optimize for multiple complex business objectives, which usually consist of reaching a set of target events or moving users to next more preferred stage; d) Most importantly, it has to learn and make decisions in realtime and at massive scale. Most of the solutions in the market only addresses a few of those challenges, due to model or technical challenges. In this presentation, we will show how we address all those issues at Salesforce Marketing Cloud Einstein. We will present a hybrid model based on reinforcement learning that balances both online and offline learning. We will show how we utilize distributed big data processing technologies and services to train and make predictions at massive scale. We will also discuss an offline evaluation mechanism to provide bounded expected performance, which has been a hard problem for reinforcement machine learning in general.

Kexin Xie

Salesforce

Kexin Xie is a Senior Director of Engineering at Salesforce, responsible for data science research, practices and architecture for Marketing Cloud Einstein. He leads the team for push forward the AI initiative at very large scale, the data system processes over 7 billion monthly unique users making tens of trillions of weekly predictions. Before Salesforce, he has worked in different sectors building large-scale data science platforms that perform machine learning, data mining and online/offline analytics in the space of data management, real-time bidding, intelligent marketing, anti-fraud and anti-money laundering. He has a PhD in computer science, and has publications in top tier journals and conferences like ACM TODS and VLDB.

Yuxi Zhang

Salesforce

Yuxi Zhang is a Senior Data Science Engineer at Salesforce Marketing Cloud Einstein. She has designed and developed several AI-driven features and recommendation systems that are currently served to Marketing Cloud customers. Prior to Salesforce, she has worked on music recommendation systems, audio fingerprinting and A/B testing at Pandora for a few years. She received a Master's in Music Technology and Robotics.