← Back to WorkProduct Analytics

B2B SaaS Onboarding Experiment

A project management company wanted more new customers to finish setting up their first project. I tested a step-by-step checklist and found that it helped smaller teams, but made the experience worse for larger companies.

RoleProduct Analyst
StackPython, pandas, SciPy, statsmodels, SQLite, Plotly
Timeline2026
NotebookOpen ↗
2,000
New Customers Studied
+5%
More Customers Got Started
90
Days of Testing
$97K
Projected First-Year Gain

Overview

The Problem

New customers were signing up, looking around, and leaving before they created their first project. The company wanted to make those first few steps easier without adding unnecessary friction.

What We Tested

For 90 days, half of the 2,000 new customers used the regular setup screen. The other half received a checklist that walked them through creating their first project. I compared how many people got started, how quickly they did it, and which types of customers benefited.

What changed

The checklist helped some customers and slowed down others

Smaller, self-serve teams got started more often with the checklist. Enterprise customers did better with the original, higher-touch experience.

That difference changed the decision: give the checklist to smaller teams, but keep enterprise customers on the experience that already worked for them.

Key Findings

At First, the Checklist Looked Like a Win

With the regular setup, 32 out of every 100 customers got started within seven days. With the checklist, that increased to about 37 out of every 100.

But It Did Not Work for Everyone

Smaller teams benefited because they were setting up the product on their own. Larger companies often had help from a sales or support team, so the extra checklist became another step in their way.

Customers Moved Faster

People who received the checklist created their first project sooner and tried more features during their first week. That showed the checklist was useful, but only for the right customers.

The Answer Was Not One-Size-Fits-All

Instead of giving every customer the same setup, the better decision was to show the checklist to smaller teams and keep the existing experience for larger companies.

How I Reached the Decision

Start
2,000 customers
Half tried each version
Fairness
Fair comparison
Both groups started alike
Overall Result
5 more per 100
Customers got started
The Catch
Different outcomes
Small teams up, large teams down
Decision
Use it selectively
Only where it helped

The Work Behind the Recommendation

1Checking That the Comparison Was Fair
What I didBefore comparing the two experiences, I made sure one group did not start with an unfair advantage. The customers were similar in company size, plan type, and starting behavior.
Why it matteredThat gave me confidence that the checklist caused the difference instead of one group simply being easier to help.
2Finding Out Who It Helped
What I didI compared smaller and larger companies, different plans, and customers working alone versus customers receiving hands-on support. The checklist clearly helped one group while getting in the way of another.
Why it matteredWithout that closer look, the company could have launched the checklist to everyone and made setup harder for its largest customers.
3Choosing the Safest Way to Launch It
What I didI compared three choices: give the checklist to everyone, give it only to smaller teams, or keep the original setup. Then I estimated the first-year value of each option.
Why it matteredGiving it only to the customers it helped was projected to create roughly $97K more value in the first year than giving it to everyone.

Recommendation

Show the walkthrough only to small teams under 20 people. Leave larger companies and sales-led accounts on the old experience. That way the company keeps the upside where the checklist clearly helped, without forcing it onto the customers it slowed down.

Use It Now
Teams Under 20
Test Again
Growing Teams
Keep Current Setup
Large Companies