C is for... control groups

C is for… control groups. These two words can fill decisioning architects with dread on customer decisioning projects, but everyone wants to use them! To be clear, when I say ‘control groups’ I am talking about a set of customers who are excluded from receiving a piece of marketing. I will refer to them as ‘hold-out groups’ as control groups can mean different things to different technology solutions – for example, Pega uses control group functionality (which doesn’t hold out customers) within its Impact Analyzer experimentation (an approach I can get fully on board with!) 

And I get it, traditionally hold-out groups have been a really great way of showing uplift for marketing campaigns. They were easy to implement, simple to track and are a very business friendly way to show value to stakeholders. “So why do you not like them?” I hear you ask. Well, to start explaining that, I will first set the scene and delve back into the business transformation from old-school batch and blast (bit harsh I know!) campaigns to the new age of omnichannel always-on marketing.

The paradigm shift

Most marketers will know how it goes: campaigns are sent on scheduled intervals consisting of many segments and delivering one or sometimes multiple offers. The campaign can sometimes be as simple as a query that generates a list/audience of qualifying customers. With this approach, the offer is the starting point, and it is very easy to know which customers will receive that offer from the campaign before sending out the email / SMS / Mobile Push comms etc. Thumbs up for hold-out groups!

Let me introduce the “new” kid on the block, omnichannel customer decisioning. Now we still have offers/service messages (known as Next Best Actions or NBAs for short) that will be sent to customers, but the starting point has changed. Instead of the offer being the primary context, we start with individual customers. We then look at all the available offers they could receive and apply business logic, AI models and scoring (to name a few features) before settling on the best one(s) that we will want to present. This could be triggered via a scheduled approach where we look at batches of customers to send NBAs in an outbound channel, or via inbound requests from channels where we only look at a single customer. The ‘trigger’ for making the decision may differ, but the overall approach in making the decision stays the same.

Some of you will already start to see the challenge; at any given time, we cannot be certain which offer a particular customer will get. With real-time data streaming, constant behavioural learning from closed loop feedback and mindful of the underlying customer behaviour (e.g. visiting the website, clicking on an application form, or calling the call centre, etc.), we would ideally be presenting the most relevant NBA to every customer at that exact individual moment in time. Putting a hold-out group on every NBA would become a mammoth task, each interaction must be fully explained, which cannot be done at scale. Was this NBA not shown to the customer because they were in a hold-out group, or would it never have been shown anyway as it wasn’t the most relevant thing to talk to the customer about at this time? What if the customer is in many hold-out groups for competing NBAs? What if a customer is only in a hold-out group for a single NBA within a group of similar NBAs? Is the hold-out group only applied in outbound channels, what if they see the NBA on an inbound channel? How can we measure the uplift or value of any single NBA?

Tackling testing complexity

Obviously, this is a complex area, but luckily leading platforms have tools that help. Here are the main areas I think are important in a mature experimentation and value reporting strategy within the world of customer decisioning:

• Test and measure high level components of your end-to-end decisioning framework instead of only an NBA’s “targeting”. Measuring components such as overall strategy or business levers can help you identify underperforming elements of your decisioning solution; allowing you to experiment and finetune individual or larger groups of actions for better engagement.  Keeping on the Pega theme, Impact Analyser is the perfect place to perform that testing, below highlights the different type of experiments:

1. Performance of the Next-Best-Action Strategy : Highlights lift of NBA vs random relevant action
2. Performance of Empathy : Highlights missed opportunities because of propensity (P) x business value (V) x context weighting (W) x business lever (L) vs propensity only.
3. Performance of Business Levers : Highlights missed opportunities with business levers vs no levers.
4. Performance of Engagement Policies : Highlights missed opportunities gained by reducing engagement policies.
5. Performance of AI : Highlights lift of AI-driven propensity vs randomly generated propensity.

With the decreased cost in getting an NBA to market (through tools such as 1:1 Operations Manager and GenAI), “experimentation” not “elimination” should be the name of the game!

• Measuring NBA performance to make timely updates to keep up with changing market behaviour. Consider:

a. Measurement that covers a range of granularity: individual NBAs, groups of NBAs, business issues (e.g. retention or sales, etc.), specific products, product groups (e.g. mortgage, savings, loans, etc. in financial services).

b. What time frames should be measured when a new NBA is released? As an example, we would want to monitor new NBAs over shorter time frames, whereas more established NBAs could be monitored over longer time frames (e.g. every quarter).

c. Setting up NBA alerts to trigger notifications whenever there is a drastic change in pattern of behaviour (e.g. drop in clicks/acceptance, drop in website views, etc.) to allow for speedy investigation and resolution if needed.

These should allow for constant gathering of insight and applying that insight into increasing engagement across all touchpoints. This can also be used to show success of NBAs at any point in time, and hopefully even a trend of increased engagement as changes are made.

• Build up key customer audiences which are meaningful to the business, such as millennials working in tech or first time buyers, instead of measuring audiences that happen to be eligible for a certain offers or campaigns. These audiences aren’t specifically used as targeting for specific campaigns but are the groups that marketers or business stakeholders are interested in or are most influential in achieving KPIs. Key audiences can be used to determine:

a. How well engagement has been over the past weeks/months.

b. The NBAs that are the most or least effective at achieving business goals.

c. Insights into what changes should be made to current NBAs, or what new NBAs can be created to further achieve those goals.

• For those that are brave enough and want to take the next step into building a mature reporting framework, consider using the reports and data provided by the decisioning platform to build multi-touch attribution models which can also account for external factors that influence customer behaviour.

I love to be proven wrong and am constantly looking for learning opportunities, in fact I was provided one recently during a Pega Customer Decision Hub (CDH) Community Event in Amsterdam. The event discussed a client use case for hold-out groups that I can get behind, which involved sending out a newsletter which contains multiple articles, with each article being a separate NBA in Pega. The hold out group is used to provide a value metric for the newsletter as a whole, rather than for every single NBA. The newsletter is sent to a known ‘fixed’ audience (good for hold-out groups), and they are utilising always-on decisioning to determine the specific articles each customer would see within the newsletter. In this case I can see the benefit of having a hold-out group. I also acknowledge that sometimes we ‘have’ to build hold-out groups, because that is what clients are used to, or how they currently define success. In these cases I would always try to urge businesses to stay away, or if we must do them, have them at a high level like a fallow cell to exclude customers from seeing any communication from customer decisioning (so have a generic message on inbound and nothing in outbound). This would provide an uplift for decisioning, but I still believe it should form a very small part of a wider value reporting framework that can provide insight into every aspect of the decisioning framework. I am more interested in constantly learning, adapting and fine tuning the overall decisioning engine to maximise overall business value and create the very best customer experience; and I am not convinced hold-out groups provide the insight to enable this.

Our contribution to customer-centric excellence

Customer decisioning is key in delivering exceptional customer experiences. By embracing a 'segment of one' approach and data-driven insights, Merkle empowers precise, timely, and personalised interactions that build meaningful moments and experiences. With Merkle's help, businesses can deliver exceptional customer engagement in this competitive landscape, fostering enduring relationships. Merkle's commitment to customer decisioning marks a vital step in achieving customer-centric excellence.

Want to read more? Our previous article ('Customer Decisioning: Why it’s Crucial to an Unbound Experience') highlights the significance of customer decisioning in enhancing the customer experience. More details on the mechanics of customer decisioning, including a worked example, are available in our ‘Customer Decisioning: Lifting the Bonnet to Explore how a Decisioning Engine Works’ article.

If you missed our Decisioning Congress in October, you can access all the valuable content on-demand from across the two days and gain actionable insights on how you can become a world-class decisioning business.

If you’re considering a complete transformation of your customer experience, get in touch with Merkle today to drive real, valuable results.