Testing message strategy is as important as it is with email and any other form of communication. Here’s how to set up a proper A|B (or more) testing.
Outline of a Test
Below is a brief outline of the steps. Visualizing the structure prior to building is a good way to build a successful test.
- What is being tested: typically this means two different ways to phrase a similar call-to-action.
- Measuring effectiveness and Defining Success: because SMS/MMS are a very simple medium, tools that track things like open rates, etc, might not be available, so external methods may be needed. Metadata collection, analytics provided by shortened links, or two different Revere Calling hotline numbers if driving phone calls. What would be considered success – calls, responses via text, link clicks?
- Structure: out of the audience, send Message A to one small randomly-selected group, and Message B to another group of the same size. Most of the list in the first phase will receive no message. A good rule of thumb is to allot 5% of the list to each test group and 90% to get no testing message.
- Results: based on the results, determine which message was more effective.
- Broadcast: Take the successful message and broadcast to the 90% of the list that received no message.
A|B Test Setup
- Prior to setup, create Metadata fields to reflect the segments (up to 6) being tested.
- From the Broadcast page, click the Segmented Messaging icon. The content will change to show two separate messaging inputs. In the example provided, 3 segments are created: testing message A, message B, and the 3rd empty segment which will receive no message in this round.
- Check the Tag Segments with Metadata box; select the appropriate metadata field and value to be associated with each message type (see example below). This value will be associated with each subscriber.
- Enter the message type (SMS, MMS or Mobile Flow) to be sent and enter content. Use care when tagging metadata if a mobile flow is used – metadata should be associated with the mobile flow, not the segment.
- In the example provided, the first two segments will be the test messages; set each to go out to 5% of the list, and change the tag value to A and B, respectively.
- For the third segment, enter 90% and select C as the tag, but select the “No message” icon. The bulk of the list will get no message at all. The percentages must total 100%.
- Once the broadcast has been sent, give the action some time to happen. If looking for something that’s a higher lift for users (like making a phone call to a legislator), it might be best to wait longer.
- Determine the most effective action; take that message and set up a new broadcast. This one will not be segmented, but filter the audience so that only those who did not receive the message in the first round get the desired message.