Testing your message strategy is as important as it is with email and any other form of communication. Here’s how to set up a proper A|B (or A|B|C|D|E even) test.
Outline of a Test
Before delving into the details, here’s a brief outline of your steps. Structure the test in your mind before even getting to the broadcast step.
- Know what you’re testing: typically this means two different ways to phrase a similar call-to-action.
- Know how you’re measuring effectiveness: because SMS/MMS are a very simple medium, the tools that track things like open rates, etc, might not be available to you, so you’ll have to rely on external methods. Use metadata collection, Revere Direction for full analytics on shortened links, or two different Revere Calling hotline numbers if you’re driving people to make calls…or talk with a Revere services rep for ideas if you’re doing something different. Also, ask yourself what you consider “successful.” Are you looking for clicks on a link, completions of a petition signup, calls into a hotline, connections to a legislator?
- Structure the test: out of your whole audience, you’re going to want to send Message A to one small randomly-selected group, and Message B to another group of the same size. Most of your list in the first phase will receive no message. A good rule of thumb is to allot 5% of your list to each test group and 90% to get no message just yet.
- Determine success: based on what you consider a success, determine which message was more effective. The nice thing about SMS is that the vast majority of actions off of a text message take place fifteen minutes or less after delivery, so you can figure this out really quickly.
- Deliver the most effective message: whichever message out of A and B was most effective, deliver to the 90% of your list that got no message in the first phase. This way, you get the most effective message to the vast majority of your supporters without needing to dedicate too much time or resources on potential dead-ends.
Putting it Into Action
- To accomplish this in your broadcast, click the Segmented Messaging icon. The Broadcast section will change to show two separate messaging inputs, though for this test, you’re going to want to create 3 segments.
- Check the Tag Segments with Metadata box; when the metadata field selector appears, pick the Segment field (this is an example field); this will mark the members of your random samples with a value that shows which group they were in.
- Your first two segments will be your test messages; set each to go out to 5% of the list, and change the tag value to A and B, respectively.
- For your third segment, set it to go to 90% of the list and pick C for the tag, but select the “No message” icon. At this point, the bulk of your list will get no message at all.
- Once you send the broadcast, give the action some time to happen. At the very least, 15 minutes for immediate actions (clicks on a link, etc.). If you’re looking for something that’s a higher lift for users (like making a phone call to a legislator), it might be best to wait about an hour or more.
- Once you’ve given enough time, determine which action was most effective; take the exact body of that message and set up a new broadcast. This one will not be segmented, but you’ll want to filter your audience so that you’re only broadcasting to people who have Segment = C (these are the people who got no message the first time). Now, you’re sending your proven best content to the remaining bulk of your audience!