Defining Success Criteria Upfront

Overview

If you define success after seeing the results, you’re not measuring—you’re rationalizing. Success criteria must be defined before you launch your pilot, agreed upon by stakeholders, and documented in writing. This is how you make objective decisions about whether to expand, adjust, or course-correct.

What You’ll Learn

• Metric Types: Adoption, productivity, satisfaction, and mission metrics
• Realistic Targets: Baselines, benchmarks, and phased expectations
• Stakeholder Alignment: Matching metrics to what each stakeholder cares about
• Measurement Plan: A documented framework for tracking and reporting

Script

Hook: define success before you start

Before you enable a single Copilot license for your pilot, answer this question: what does success look like?

If you wait until after the pilot to define success, you’ll unconsciously set the bar to match whatever happened. High adoption? We’re successful. Low adoption? We learned valuable lessons. That’s not measurement—that’s storytelling.

Define your criteria now. Get agreement. Write them down.

Types of success metrics

Success metrics for Copilot fall into four categories.

Adoption metrics tell you who’s using Copilot and how. Active users over 30 days is the baseline—what percentage of licensed users actually used Copilot? Feature usage tells you which apps are getting traction. Teams? Outlook? Word? Usage frequency tells you whether people are building habits. Someone who uses Copilot daily is getting more value than someone who tried it once.

Productivity metrics tell you whether Copilot is saving time or improving output. This is harder to measure but more meaningful. Survey users about time saved on specific tasks: meeting preparation, email triage, document drafting, data analysis. The Forrester Total Economic Impact study found that Copilot users saved an average of 11 hours per month. Your numbers will vary, but that provides a benchmark.

Satisfaction metrics capture how users feel about the tool. A simple survey at the two-week and four-week marks asking “How useful is Copilot for your daily work?” on a five-point scale gives you actionable data. Net Promoter Score—”Would you recommend Copilot to a colleague?”—provides a single number that tracks sentiment over time.

Mission metrics are specific to your organization. If your agency’s priority is faster FOIA response times, measure whether Copilot users process FOIA requests faster. If your priority is better briefing materials, measure quality ratings on briefings produced with Copilot assistance. These metrics connect Copilot directly to the outcomes leadership cares about.

Setting realistic targets

You need baselines and benchmarks to set realistic targets.

Baseline your current state before enabling Copilot. How much time do your employees spend on email? How long does it take to prepare a typical briefing document? How many meetings per week and how much preparation time? These baselines let you measure improvement, not just activity.

For adoption targets, use Microsoft’s own benchmarks as a starting point. For a well-managed pilot, 60 to 70 percent weekly active usage is a reasonable target. For broad deployment at the three-month mark, 50 to 60 percent active usage indicates healthy adoption. Below 40 percent signals an adoption problem that needs intervention.

Set phased targets. Pilot targets are different from broad deployment targets. During a pilot with selected, motivated users, you should see higher adoption rates. During broad deployment with the general population, rates will be lower. Adjust your expectations accordingly.

Avoid vanity metrics. Total prompts submitted sounds impressive but doesn’t tell you much. One user submitting 500 prompts isn’t the same as 50 users each submitting 10. Focus on metrics that reflect breadth of adoption and depth of engagement, not just volume.

Aligning metrics with stakeholders

Different stakeholders care about different things. Align your metrics to their concerns.

Executives care about ROI and mission impact. Report to them in terms of hours saved, productivity improvements, and strategic goal advancement. “Our pilot users saved an average of 8 hours per month on routine tasks” speaks their language.

IT leadership cares about technical health. Report on support ticket volume, system stability, and security posture. “Copilot generated zero security incidents during the pilot and support volume was manageable with existing staff” gives them confidence.

End users care about time saved and ease of use. Report back to users on the aggregate results: “Pilot users rated Copilot 4.2 out of 5 for usefulness and reported saving an average of 2 hours per week.” Users who see that their peers find value are more likely to engage.

The critical step: get agreement on which metrics matter and what targets are acceptable before the pilot starts. Hold a brief meeting with your executive sponsor, IT lead, and change management lead. Present the proposed metrics and targets. Adjust based on their input. Document the agreement. This prevents the post-pilot debate about whether the results were “good enough.”

Close: your measurement plan

Document your success criteria in a measurement plan.

For each metric, record four things: what you’re measuring, the target value, the data source, and the reporting frequency.

Example: Active 30-day usage. Target: 65 percent of pilot users. Source: Microsoft 365 Admin Center Copilot usage report. Frequency: weekly during pilot.

Example: User satisfaction. Target: 4.0 or higher on five-point scale. Source: survey at week 2 and week 4. Frequency: biweekly during pilot.

Review your criteria quarterly after the pilot. As you move from pilot to broad deployment, your targets and metrics should evolve. What mattered in a 100-person pilot may not be the right measure for an organization-wide deployment.

Use your criteria to make decisions. If your pilot meets the adoption and satisfaction targets, advance to the next phase. If it doesn’t, investigate why and adjust before expanding. Success criteria aren’t just for reporting—they’re your decision-making framework.

Sources & References

• Microsoft Copilot adoption resources — Adoption measurement guidance
• Viva Insights Copilot Dashboard — Metrics available for Copilot
• Forrester TEI study on Microsoft Copilot — Benchmark productivity data