 | ||
Q: What is the most effective way to use the A/B-split testing feature in LISTSERV Maestro? In the field of email marketing, A/B-split testing is a useful tool to find out how certain variations in the message affect the message's success. In its simplest form, A/B-split testing means that for a given job you split the list of recipients into several batches and send a different version of the message to each batch. Then you analyze which version was most successful and use these insights to optimize your next message. A/B-Split Testing with Sampling With A/B sampling variants, LISTSERV Maestro goes beyond this simple A/B-split testing. When you make use of this feature, LISTSERV Maestro splits the mail job into one "main variant" and several "sampling variants". It also splits the total list of recipients into one, usually large, main batch that is assigned to the main variant and several smaller batches that are assigned to the sampling variants.
You then prepare the sampling variants for delivery, with slight variations in the subject line, message content or the delivery time. LISTSERV Maestro delivers these sampling variants to their small recipient batches and starts collecting tracking data for them. After enough tracking data has been collected, you then view the results to decide which of the variants was the most successful. LISTSERV Maestro will help with your decision by presenting the tracking metrics of all variant jobs in a handy chart, from which you can directly select the winning variant.
Simply select the best performing variant, then click the [Use as Main Job] button, and LISTSERV Maestro will automatically copy the message and tracking settings of the winning sampling variant to the main variant. You can then proceed normally with scheduling and authorizing the delivery of the main variant.
Even though this is a useful feature for A/B-split testing, there are a few caveats to keep in mind. Obviously, the jobs should be a certain minimum size so that it's possible to split the job into a large main batch and several smaller sampling batches but still have a sampling batch size that will give you meaningful tracking results. And of course, using sampling variants also means that the message to the main batch of recipients will be delivered later, possibly a few days later, than the versions that were sent to the recipients in the sampling batches because you first have to wait until enough tracking data has been collected to make an informed decision about which sampling variant was most successful. So you should only use this feature if this delay between sampling and main variants is not a problem for the kind of message that you are sending. Automating the Selection of the Best Perfoming Sampling Variant If you want, you can also automate this last step. Simply tell LISTSERV Maestro how long it's supposed to wait for incoming tracking data after the delivery of the sampling variants, which metric to use to determine the most successful or "winning" variant, and when to deliver the main variant. LISTSERV Maestro will then automatically deliver the sampling variants, wait for the specified time, and then determine the winning variant and use its message for the main job, which is then automatically authorized and delivered. To set up this type of automated A/B-split testing, when you define the A/B-split job, make sure that its "Type" is set to "Sampling A/B-Split" and that the main variant is selected automatically by LISTSERV Maestro.
Then set up the sampling variants as desired and define the settings for the "Automatic Main Variant Selection". These settings have two parts, "How To Select The Winner" and "When To Deliver The Main Variant" as shown in the image below. First, select how LISTSERV Maestro should choose the winning variant. Depending on your choice, the winning variant will be the one with the highest view rate, click rate or share count. Note that only those choices are available that are supported by all sampling variants. For example in the image below, the third variant does not support the "Share Count" choice, which is why this choice is not available. In addition, specify the period after delivery of the sampling variants when the winner should be selected. You can specify this period as a number of hours or days. Note that for each sampling variant, this period starts at the moment of delivery and ends the specified number of hours or days later. If all sampling variants are delivered at the same time, then this period also starts and ends at the same time for all variants. But if the sampling variants are delivered at different times, even though the period is the same for each variant, the counting will start and end at a different times. The length of this period also effectively determines the moment at which LISTSERV Maestro actually selects the winning variant, which is exactly at the time when this period ends for the sampling variant that was delivered last.
Now that you have defined how and when LISTSERV Maestro should select the winning variant, you also need to specify when to deliver the main variant. For this you have four choices: Send Immediately: LISTSERV Maestro will start the delivery of the main variant immediately after it has selected the winner. This is usually the choice to use if delivery time is not important and you want the main job to go out as soon as possible. Send at Winner's Delivery Time: After LISTSERV Maestro has selected the winner, it will look at the original delivery time of the winner and will start delivery of the main variant at the next occurrence of this time, which may be either the same or the next day. This is the choice to use if the time of day is important for your delivery but the actual day is not and you want to use the same time as the one used by the winner. It implies an additional delay of the main variant of up to 24 hours. Send at Winner's Delivery Weekday and Time: After LISTSERV Maestro has selected the winner, it will look at the original delivery weekday and time of the winner and will start delivery of the main variant at the next occurrence of this time on the same weekday, which may be today or whenever the next occurrence of the same weekday happens. This is the choice to use if both the time of day and the weekday are important for your delivery and you want to use the same weekday and time as the one that was used by the winner. It implies an additional delay of the main variant of up to 7 days. Send at the Next Occurrence of the Following Time: After LISTSERV Maestro has selected the winner, it will wait until the next occurrence of the specified time, which may be either today or tomorrow, and will then start delivery of the main variant. This is the choice to use if the time of day is important for your delivery but the actual day is not and you want to use a specific time that may differ from the delivery time that was used by the winner. It implies an additional delay of the main variant of up to 24 hours. Once you have prepared these settings and have completed the definition of all sampling variants, you then only have to authorize the delivery of the sampling variants and the rest will be automatically handled by LISTSERV Maestro. The Process of an Automated A/B-Split Job with Sampling The following steps will occur without further input from you: 1. Each authorized sampling variant will wait in the outgoing mail jobs list until its designated delivery time. LISTSERV Maestro will then start its delivery. The finished sampling variants are moved to the completed mail jobs list. 2. After the delivery of the last sampling variant, LISTSERV Maestro will wait for the specified amount of time to allow tracking events to collect. 3. Once this waiting time has elapsed, LISTSERV Maestro will use the tracking data collected to select the winning sampling variant by picking the one with the highest value for the specified metric. 4. When the winning variant has been selected, LISTSERV Maestro will automatically copy its message and tracking definition to the main variant and will then authorize and schedule the delivery of the main variant according to the specified schedule setting. 5. The main variant will then wait in the list of outgoing mail jobs until its designated delivery time, at which point LISTSERV Maestro will start its delivery and move it to the completed mail jobs list after delivery has finished. Subscribe to LISTSERV at Work. | ||
© L-Soft 2014. All Rights Reserved. |
