Measure Aptitude Before You Measure Productivity

The first metric that matters is not output.

If you measure productivity during the adoption dip, you will kill the initiative before it pays off.

The dip is real

Every change management playbook includes the same graph: performance drops before it rises. New tools slow people down before they speed them up. New workflows create friction before they create flow.

You know this. Your executives know this. But when the quarterly review comes and the productivity numbers are flat or down, the pressure lands anyway.

"Any change that you go through, you can't measure the impact of the change until after that classic change management dip."

JB Brown, Roo Cast S01E19

This is where AI adoption initiatives die. Not because the tool failed. Because the measurement came too early.

Smartsheet's sequence

Smartsheet took a different approach. Instead of measuring AI-driven productivity from day one, they measured learning velocity.

Engineers were rated on a four-point scale: red, yellow, orange, green. Not based on output. Based on their aptitude toward using AI in their job. Were they experimenting? Were they building familiarity? Were they moving from skeptical to curious?

"We tried to drive learning velocity at first... rate for one of those and then there was no negative outcome of that, no punishment out of that. But then that gave us a baseline."

JB Brown, Roo Cast S01E19

Managers got goals to improve their team's distribution over six months. Move more people from red to yellow. Move yellow to orange. The goal was adoption, not acceleration.

Only after the organization reached baseline adoption did AI usage become an expectation.

The mechanics of protected time

Aptitude does not develop in the margins. If your engineers are fully loaded with sprint commitments and you tell them to "also try AI," they will not try AI. They will ship their tickets and mention the tool in retros.

Protected time is not a perk. It is infrastructure.

"Oftentimes it's just giving them time. I'm going to stop loading you up with all these assignments... I want you to go do a week of just using this tool in this way."

JB Brown, Roo Cast S01E19

One week of dedicated experimentation produces more adoption signal than three months of "use it when you can."

The tradeoff

This approach costs time. You are explicitly slowing down short-term output to build long-term capability. That is a real tradeoff, and it requires executive buy-in.

If your leadership expects AI to show ROI in the first quarter, this sequence will feel too slow. If your leadership understands that adoption curves have a dip and you cannot skip the dip, this sequence protects the initiative long enough to pay off.

Why this matters for your organization

For a 50-person engineering team, the difference between "measure productivity now" and "measure aptitude first" is the difference between killing the initiative in Q2 and having a transformed workflow by Q4.

The teams that survive the dip are the ones that planned for it. They measured learning velocity while the skeptics were experimenting. They gave protected time while the early adopters were building muscle memory. They waited to measure productivity until the baseline existed.

The sequence is: positive environment first, measurement pressure second.

If you are planning an AI adoption rollout, start by asking: what does "green" look like for aptitude? Define that before you define the productivity targets.

How Roo Code accelerates the aptitude phase

AI coding agents like Roo Code reduce the learning curve during the aptitude phase because they close the loop: proposing changes, running tests, and iterating based on results without requiring engineers to context-switch between tools.

When engineers experiment with Roo Code during protected time, they build aptitude faster because the agent handles the tedious parts of the workflow. The BYOK (bring your own key) model means teams can start experimenting immediately without procurement delays or vendor negotiations blocking the learning period.

Roo Code transforms the aptitude phase from "learn a new tool" to "learn a new workflow" by handling execution while developers focus on intent and review.

Measuring adoption: old approach vs. new approach