Linio’s A/B Test between Topsort vs. a Legacy Solution

Linio is one of the most popular eCommerce marketplaces in Latin America. Early in 2021, Linio approached Topsort with an issue that they were facing.

Michael Ostrovsky, Co-Founder and Chief Scientist
May 12, 2022

Comparing Topsort to our competition

It’s not everyday we get to be compared side by side against our competitors in real action. Here’s how we stack up!

Linio is one of the most popular ecommerce marketplaces in Latin America, offering a wide variety of items from electronics to home goods to toys to clothing to perfume - over 4 million items in total. Early in 2021, Linio approached Topsort with an issue that they were facing. They were excited about auction-based sponsored listings and banners, and have been working with another SaaS company in the space (let’s call it Legacy Provider M) to operate them on their platform, but were dissatisfied with the results - the take up among merchants was disappointing, and the monetization results were underwhelming. 

Linio wanted to see whether Topsort could provide better results? Challenge accepted!

Setting up the Test

To test this, Linio's Product team conducted an A/B test, putting Topsort directly against the legacy solution. For Peru and Colombia, the Linio team picked a dozen categories across price ranges and conversion rates (including such popular categories as Appliances, Phone accessories, Shoes, Kitchen, Vitamins, Perfume, and others). For each session, the impression is randomly split into either a Topsort bucket or a Legacy solution bucket.

Given half of the customer traffic was sent to the legacy solution, while the other half was sent to Topsort, everything else was kept the same. Organic results shown to the customers were exactly the same in both cases, but sponsored listings results were different. Differences in the final user experience were drastic: Topsort's integration shows no delay in the sponsored listings rendering at the same time as the organic results, and showed up in the most prominent locations in the search results, instead of a glitchy delay where the sponsored listings would show up after the organic results. Immediately, user experience with Topsort is better.


Engineering

Engineers from Linio and Topsort worked together to run and deploy the A/B test. On the Linio side, the engineers were responsible for “splitting" the traffic accurately. For the “A” part, they then called on the Topsort API to run the auction and retrieve the ads, while the “B” part continued to be sent to the legacy solution. On the Topsort side, the engineers helped answer Linio's questions, and made sure that everything was running smoothly. The whole deployment process took two Linio sprints with one engineer and during this process teams worked closely in Slack channels.

One tricky thing that Linio has encountered is the sync of out-of-stock products. With legacy provider M, sync files were the only way to communicate the availability of SKUs. That of course can be limiting and causes delay in a large scale retail site like Linio. Hence the team was keen on exploring alternative solutions regarding keeping the product catalogue more in sync. With Topsort's nerdy engineering power - here comes the catalogue service!

Catalogue Service is an adapter and an API-first approach to keep the products availability completely in sync with the sponsored results.

After a short 2 weeks set up, Topsort's systems kicked into gear, receiving data from Linio queries, impressions, clicks, and purchases. More excitingly, a new array of technical features that were not previously available via the legacy solution is now running smoothly in production: ranging from Autobidding, Pacing, to Quality Score. The data model Topsort built to predict ad quality scores and conversion rates improved Linio's results on the fly, and improved continuously as time generated more session data. Topsort's data science team worked so hard on this project! About a month into Linio's A/B testas test, Linio's team decided it was sufficient evidence to stop the test and take a look at the results!

Can we say it was a fully fair test? Mathematically, not really, as marketplace solutions get better with time and operating with less amount of context while being compared to a solution that's been in place for a year, required a level of resourcefulness. However, 4 weeks were enough to put us at a “steady state” and obtain statistically reliable comparisons.

Final Results!

The statistical results was unambiguously better on the Topsort's half. Topsort’s results outperformed those of the Legacy Solution in both countries and on all relevant metrics, including click-through rate, conversion, attributed sales.. In one country, the click-through rate (CTR) on ads selected and shown by Topsort was more than double that of the Legacy Solution, and in the other Topsort exceeded that of the Legacy Solution M by over 90%.

Topsort’s conversion rate was 20% higher than the Lagacy Solution's in one country, and more than 80% higher in the other country. During the second half of the experiment when Topsort had enough data to run Autobidding and Quality Score, the performance of CPC and CTR were 5x what the Legacy Solution showed.

The total clicks generated from sponsored listings powered by Topsort's half, combined with higher click-through and conversion rates, in the world of auction-based sponsored listings translate into much better monetization results. From the A/B test, one can say Topsort's technology can generate 5x the revenue for Linio than the Legacy Solution. The Linio business team was delighted with the outcome!

While the main goal of the test was to evaluate the performance of Topsort’s technology, it also gave a chance for Linio to experience the quality and responsiveness of Topsort’s engineering team, as well as the ease of integration was a steep comparison from the previous Legacy Solution. Erick Alanis, a Linio engineering lead on this project, expressed his gratitude to Topsort, saying: