CRO Agency vs In-House: Which to Pick
What a CRO agency and an in-house tester each actually deliver, what they cost, and the traffic and budget thresholds that decide which one fits.
A CRO agency gives you a full testing team — research, design, development and statistics — for a monthly fee scoped to testing velocity. An in-house hire gives you one person's full attention for a salary, but rarely covers all four skills at once. The right choice depends on your traffic, budget and testing volume.
Both options only matter once you have decided to run a structured A/B testing program rather than shipping changes on instinct. This post assumes that decision is made and compares who runs it.
On this page
What do you actually get from each option?
Four different jobs, whichever way you buy them: research (finding what to test), design (building the variant), development (shipping it without breaking the page) and statistics (reading the result honestly). An agency staffs all four as a team. A single in-house hire is realistically strong in one, competent in a second, and dependent on other people at your company for the rest.
That is not a knock on in-house testers. It is the same arithmetic that applies to any specialist role: nobody is equally good at writing a research plan, designing a variant, shipping frontend code and running a sequential statistics engine. Nielsen Norman Group's own primer on A/B testing treats hypothesis-writing, execution and statistical reading as three distinct skills for a reason — they usually live in different people [2].
| CRO agency | In-house hire | |
|---|---|---|
| Skill coverage | Research, design, dev, statistics as a team | One or two of the four, well |
| Time to first test | Usually 2-4 weeks | Weeks to months, plus hiring time |
| Cost shape | Monthly retainer scoped to velocity | Salary plus tools plus management |
| Institutional memory | Cross-client pattern library | Deep on your product, narrow elsewhere |
| Main failure mode | A generic playbook applied without context | One person, one skill, everything else stalls |
| Best fit | Steady testing volume, no spare engineering time | High volume, or a product too specific for outside pattern-matching |
What does each option actually cost?
An agency retainer is the easier number to get: it is usually quoted as a flat monthly fee tied to a testing velocity, such as two or three tests running at any time. Ask what that fee covers when a test is inconclusive, because a program is inconclusive results as often as wins, and a vendor that only prices around winners is pricing optimistically.
An in-house hire's cost is less visible on a single line. The salary is the obvious part; add a testing tool's licence, a design or research tool if they do not already have one, and the ramp period before they are shipping tests at a real pace. Three to six months of ramp on a salaried role is a real cost even though no invoice ever itemizes it.
How fast can each one actually start producing results?
An agency's advantage here is process, not talent. A team that runs testing programs across multiple clients already has an intake process, a tool already configured, and a way to write a hypothesis that survives statistical review — Evan Miller's widely read piece on the ways an A/B test goes wrong is basically a checklist of mistakes a repeat-practice team has already been burned by once and stopped making [3].
An in-house hire, even a strong one, has to build that process from nothing on your specific stack. That is not wasted time — the process they build will fit your product better than a generic one eventually — but "eventually" is doing real work in that sentence, and it is usually measured in months, not weeks.
What tools does each option need, and who pays for them?
A testing platform, an analytics setup and usually a heatmap or session-replay tool, whichever way you staff the work. The difference is who is already paying for the licence when you start.
An agency typically brings its own testing platform license, already configured across its client base, and folds the cost into the retainer rather than billing it as a line item. That saves you a procurement cycle, but it also means switching agencies later can mean rebuilding your test history on a different platform, since the account and its data may sit under the agency's name, not yours.
An in-house hire needs you to buy or already have a testing tool, an analytics setup they can actually query, and — per heatmaps and session replay — a way to watch what happened inside a test, not just read the topline number. None of this is expensive relative to salary, but it is a real setup cost in the first month that a retainer quote does not carry.
How do you check an agency's past results without being misled?
Ask for one specific account's full test history, not a highlight reel of wins. A real testing program loses more tests than it wins — a healthy hit rate is closer to one meaningful win in every four or five tests run, not the unbroken streak a case-study page tends to show.
Ask three questions directly: how many tests ran in the period the case study covers, how many were inconclusive or negative, and whether the account is still a client. An agency that answers all three without hesitation is showing you a real program. One that redirects to the win alone is showing you marketing, which is a different thing from evidence.
The same check applies if you are hiring in-house and want to weigh a candidate's portfolio from a past role. Ask what they shipped that lost, and what they learned from it. A candidate who can only describe wins has probably been managed away from the losses, not spared them.
Who ends up owning the testing roadmap and the tool choice?
This is where the two options diverge in a way price does not capture. An in-house hire's roadmap is yours by default; they report to you, and the backlog of tests reflects your priorities without translation. An agency's roadmap is a collaboration, and a bad one drifts toward whatever pattern worked for their last client rather than what your data actually shows.
The fix, either way, is the same: whoever runs the tests should not also be the only person who decides what gets tested. A second set of eyes on the hypothesis — whether that is a product manager reviewing an agency's test plan or a manager reviewing an in-house tester's — catches the drift before it costs a quarter.
What is the real failure mode of each option?
For an agency, it is a playbook applied without enough context: a "best practice" changed on your pricing page because it worked somewhere else, tested without understanding why your buyers behave differently. Ask any agency you are evaluating how they choose which tools and methods to use on a new account, and whether their answer changes account to account or stays the same regardless of who they are working with.
For an in-house hire, the failure mode is coverage: a strong designer running statistics they were never trained on, or a strong analyst designing variants nobody would actually want to click. The tell is a testing program that keeps producing "flat" results — not because the ideas are bad, but because the execution or the read on the data is quietly off.
When does hiring in-house clearly win?
At real volume, or with a product too specific for outside pattern-matching to help much. A team running ten or more tests a month, on a product with enough surface area that context matters more than generic best practice, gets more from one person who lives inside the product every day than from a team split across other clients.
The same is true for a product with an unusual statistical shape — very low-frequency, high-value purchases, for instance, where multivariate testing or a longer test window is the norm rather than the exception, and a generalist agency playbook does not transfer cleanly.
What does a realistic first 90 days look like, either way?
Slower at the start than either side usually promises, then faster once the process exists. The first two to four weeks are instrumentation and audit, regardless of who is running it: tagging the funnel correctly, confirming the analytics numbers match reality, and picking the first page or flow worth testing.
Weeks four through eight are usually the first live test, running long enough to reach a real read on your traffic, not a glance at an early trend. By week twelve, a working program has a second or third test either running or already read, and a rhythm for how a new idea gets proposed, prioritized and shipped.
An agency tends to hit that rhythm faster because the audit and intake process already exists; an in-house hire is often still building the intake process alongside the first test itself. Neither is a red flag — a program with zero tests live by week six, whoever is running it, is the actual warning sign worth asking about.
Weeks 1-4: instrument and audit
Confirm the funnel is tracked correctly and pick the first test based on real data, not a guess.
Weeks 4-8: first test live
Launch, then wait for a real statistical read rather than calling it early on a promising trend.
Weeks 8-12: second test plus a working cadence
A repeatable intake and prioritization process should exist by here, whoever built it.
Does a hybrid model actually work?
Yes, and it is the model most testing programs converge on eventually. One in-house person owns the roadmap, the hypotheses and the read on results; an agency, a contractor or a tool handles execution capacity the in-house person does not have time for. This is not a compromise so much as the natural next stage once a program outgrows what any single hire or any single vendor can carry alone.
The order you build it in matters less than being explicit about who owns what. A roadmap with two owners and no clear split produces the same drift as either option running alone, just with twice the meetings.
Does company size or stage change which option fits?
Yes, more than most of the criteria above. An early-stage company with uneven traffic and a product that still changes monthly often gets more from an agency, because the process and statistical discipline it brings compensate for a site that is too small and too unstable for one in-house hire to learn quickly.
A company with steady, high traffic and a product roadmap that moves slowly is the opposite case. Here, an in-house hire's deep familiarity with the product compounds over time in a way an outside team, split across other clients, cannot match test for test. The same testing skill produces better hypotheses once it has sat inside your funnel for a year than it did in month one.
Company size interacts with budget in a way worth naming directly: a small team can often afford either option, but not both, and the choice above should decide which one gets the budget first, not an assumption that in-house is always cheaper. A single salaried hire with tools and ramp time frequently costs more in year one than a scoped agency retainer would have, before that hire has shipped a single test.
How do you actually decide, this quarter?
Three questions, in order. Do you have engineering and design time to spare right now, or is every hour already spoken for? Is your testing need steady — a program running indefinitely — or a fixed project with an end date? And do you already know exactly which skill is missing, or are you still guessing at what a test program even requires?
No spare capacity, steady need
An agency retainer is the fastest path to tests actually running this quarter.
Spare capacity, one clear skill gap
Hire for the specific gap — usually statistics or research — and lean on existing design and dev time for the rest.
High testing volume already
Move toward the hybrid: an in-house owner directing agency or contractor execution scales further than either alone.
Still not sure what a program needs
Start with an agency for one quarter. You will learn what to hire for faster by watching a real program run than by guessing from a job description.
Whichever you pick, the choice is reversible. Plenty of teams start with one option and move to the other, or to the hybrid, once they know more than they did on day one — that knowledge is the actual product of the first few months, whoever is running the tests.
Questions people also ask
Is a CRO agency worth it for a small site?
Usually not below a few thousand monthly visitors to the pages you'd test, because no agency or in-house tester can reach statistical significance on traffic that thin. Below that threshold, spend on traffic or on fixing obvious usability problems first.
Can one in-house person run a full testing program?
One person can run a program, but not the full stack behind it: they will be strong in one of research, statistics, design and development, and thinner in the rest. That is fine if the gaps are covered by existing teammates; it is a bottleneck if they are not.
What does a CRO agency typically cost?
Engagements are usually a flat monthly retainer scoped to a testing velocity — a number of tests running per month — rather than a per-test fee. Ask what happens to the price if a test is inconclusive, because a program that only bills for wins is rare and worth being wary of.
How long before either option shows a result?
Expect the first real read in six to ten weeks: two to four weeks to instrument and launch the first test properly, then however long that test needs to reach significance on your traffic. Anyone who promises a result faster than a valid test can run is guessing.
Can we start in-house and add an agency later, or the reverse?
Both directions work. Starting in-house and adding an agency for overflow or specialist statistics is common once volume outgrows one person. Starting with an agency and hiring in-house once you know exactly what skill you're missing is the lower-risk order, because the job description writes itself.
Sources
- Conversion rate optimization — Optimizely (checked 22 September 2026)
- A/B Testing 101 — Nielsen Norman Group (checked 22 September 2026)
- How Not To Run an A/B Test — Evan Miller (checked 22 September 2026)
- GrowthBook's Statistics — GrowthBook (checked 22 September 2026)
Tell us your traffic and team, we'll tell you what fits
Book a free 30-minute consultation with Ibrahem and Jeremy. We look at your site while we talk, and you leave with a prioritised plan.
Who wrote this
Jeremy ZavaCo-founder
Leads growth and client partnerships.
Works on: CRO, SEO
Reviewed before publishing by Ibrahem Helal, Co-founder.
- A/B Testing: The Complete GuideWhat A/B testing is, when your traffic can support it, how to run a split test that holds up, and how to read the result without fooling yourself.
- Best CRO Tools for a Real Testing StackThe CRO tools that actually matter, grouped by what each one is for — testing, session replay, feedback and analytics — with honest cost ranges.
- A/B vs Multivariate Testing: The DifferenceA/B testing compares whole page versions; multivariate testing isolates which element drives the result — and needs far more traffic to prove it.
- How to Run a CRO Audit (Free Checklist)A CRO audit is a structured review of analytics, heuristics and real user sessions that finds the specific reasons visitors leave before converting.