AI-Powered A-B Testing and CRO Insights

**(00:49)** Thank you, David, and thanks to everyone joining today—I really appreciate it, especially with Passover and Easter preparations. My wife might not be thrilled I'm doing this today, but the show must go on! I'll give a quick overview of using AI to automate A/B testing, but more so, it’s about accelerating CRO. AI is one of the most exciting tools for this. I'll share a bit about my background and then jump into the demo.

**(01:18)** My experience is mostly as a startup founder and product executive. In 2002, I founded Worth1000, a photo editing contest site that gained a lot of traction, and I learned A/B testing there by focusing on ad clicks. Later, I founded Avary, which started as a competitor to Adobe Creative Suite, but was cloud-based. Eventually, it evolved into a software development kit (SDK) that could be embedded into partner apps.

We had about 10,000 partners and 150 million downloads of our standalone app, and A/B tested various features within the SDK. Adobe later acquired Avary, and I helped develop mobile Creative Cloud products there. After Adobe, I was Chief Product Officer at Shapeways before its IPO, and at Shutterstock, a leader in royalty-free stock photography.

**(02:57)** Eventually, I felt the startup itch again and launched AB3. Across all these companies—whether startup, pre-IPO, or public—the same CRO challenges kept arising. The digital ad spend market is projected to reach a trillion dollars by 2027, with customer acquisition costs only increasing.

Yet, we found that the average conversion rate was around 2%, which was startling considering how much was spent. At places like Shutterstock and Shapeways, we’d hire talented A/B testers and CRO experts who would make meaningful improvements, but often only on the pages they focused on. Once they moved to other areas, the initial pages would revert over time. Plus, if these experts left, they’d take valuable knowledge with them.

**(04:02)** I started wondering: what if we could use AI to optimize every page on a site continuously? Early on, this was just an idea. But around 2019, as AI models like OpenAI began advancing, I realized we could potentially create content at near-human quality.

Fast forward to today, and we’ve come a long way—we can change copy, imagery, and even video (with tools like OpenAI's Sora). AI-generated content opens up incredible possibilities for automated testing. It’s an exciting space, and we’re beginning to see competitors, which is a great sign for the field.

**(05:32)** Let's dive into a live demo. I'll use one of our early beta customers, BarkBox. With our platform, we don't make manual changes; instead, we grant AI the ability to make adjustments on the site based on predefined guidelines. This approach removes people from the bottleneck of testing and allows the AI to optimize across potentially millions of pages.

**(07:02)** The AI starts by identifying elements on a page to test, like BarkBox’s humorous headline, “Monthly dog goodies for good doggies.” By clicking on it, I tell the AI it’s allowed to test variations here. The AI then analyzes the brand voice across the site and suggests alternatives with similar humor and tone, like “Treat-filled surprises for happy hounds” or “Pup-approved treats delivered monthly.” While these suggestions might not surpass the original, it’s a good starting point to refine based on engagement data.

**(08:49)** I can also adjust button text, like changing “Get BarkBox” to alternatives like “Claim Your BarkBox” or “Unleash Joy.” For imagery, I can upload BarkBox’s brand images, and the AI will test which works best. For generic imagery, we connect to royalty-free libraries or generate images through AI.

**(12:33)** Next, we define success criteria for the AI—essentially, telling it what counts as a successful action. We can track clicks on specific calls to action, page views within a funnel, or even revenue via e-commerce integrations like Shopify. The AI uses this data to adapt, aiming to drive higher engagement and conversions.

**(14:22)** At the final setup stage, we provide a simple JavaScript snippet to add to the website’s header, allowing AI to make real-time changes securely. You can specify how much traffic sees the experimental content or use UTM codes to test with specific audiences.

**(16:45)** Once deployed, our platform compares the performance of each variant against a baseline. If, say, one headline variant converts at 15% versus the baseline’s 10%, the AI will automatically favor the higher-performing variant. This process continues, with AI generating and testing new variations based on previous winners, creating a “Darwinian evolution” of content on the site.

**(19:32)** For companies wanting control, you can approve new variants or add, edit, and pause campaigns as needed. Our setup is flexible; you can even disable campaigns temporarily if you want a different setup for a special promotion or holiday.

**(20:57)** That covers the core demo. Before moving on, are there any questions?

**(21:30)** We’ve had several! Starting with setup requirements, our script works on any website supporting JavaScript, including platforms like WordPress. Once the script is added, the AI tracks conversions and adjusts content automatically based on what's performing best.

**(23:06)** As for custom UTM codes, you can specify where the script runs within WordPress or use audience segmentation to target specific campaigns. AI generates new content variants autonomously after it learns from initial results on the page.

**(24:03)** Regarding hallucinations or content quality issues, we avoid these by using prompt engineering and leveraging libraries with built-in safeguards. The platform filters numerous AI-generated variants and selects only those that best match the brand's tone and style. 

**(25:37)** We've also integrated third-party image providers like Midjourney. If regulations change regarding image use, we can adapt our image generation methods accordingly. Our goal is to provide legally and ethically sound options for all our clients.

**(27:42)** Video content testing is not currently supported, though recent advancements by OpenAI in video generation show potential for the future. At the rate AI is advancing, it's likely we'll see video optimization in the coming years.

**(28:51)** A question came up about the content optimization process. Our AI doesn’t take a prescriptive approach; it stays true to the page's original voice and lets the statistics guide its adjustments, ensuring gradual improvements rather than drastic changes.

**(30:20)** As for customer demographics, we’ve seen adoption by both small-to-medium businesses (SMBs) and larger companies in D2C, SaaS, fintech, and healthcare. While regulated industries have strict content guidelines, our platform allows these companies to efficiently run tests with oversight, reducing time and staffing needs for CRO tasks.

**(33:35)** Currently, our platform is focused on web and landing page optimization rather than social media. However, any page you control, where JavaScript can be added, is eligible for our tool.

**(34:08)** For customers with rebrands or new guidelines, it's easy to update the AI’s baseline so it aligns with the latest brand standards. Each new campaign re-baselines automatically, so if you’re starting fresh after a rebrand, it will adjust accordingly.

**(35:07)** We’re also working on tools for agencies and multi-domain setups, allowing you to support multiple clients or domains with customized plans.

**(37:09)** While we don't yet integrate with Google Analytics, we capture engagement data through JavaScript click events. If clients want a Google Analytics integration, we may consider it based on demand.

**(38:09)** As for demographic targeting, audience segmentation based on UTM codes is available. Full demographic customization may be added in the future.

**(39:13)** The AI operates entirely on your domain, so your visitors experience the content directly on your site. If you prefer testing traffic offsite, that’s an option too.

**(41:25)** We also ensure AI changes don't deviate too far from brand standards. While the platform makes small, statistically-driven optimizations, you always have control over final approvals.

**(44:21)** To address SEO, our changes are rendered in the user’s browser, so search engines see the original page content. This approach prevents SEO disruption and safeguards site performance. We don’t add latency or affect load times, and in case of outages, your original content displays.

**(46:33)** Each variant’s performance data is viewable in the dashboard. Once a variant has significant data, it can be paused or edited as needed.

**(47:34)** The AI's approach to audience segmentation focuses on testing the content broadly. Custom user data is not a factor unless specifically set up with UTM campaigns.

**(48:40)** Competitors include companies like VWO and Optimizely, as well as Mutiny and the recently acquired Intellimize. There are also newer companies like Kik and Evolv. It’s exciting to see growth in this space