Calibration feature in QMS
Year
2023
My Role
Product Designer
Type
B2B Web App
Summary
Meta’s Quality Management System (QMS) enables support teams to review agent conversation with customer, provide feedback, and score performance. At the time, calibration was handled manually by each program, leading to slow, inconsistent, and bias-prone processes that didn’t scale. I led end-to-end product design to introduce an automated calibration workflow to deliver a unified, scalable solution within Meta’s support ecosystem.
41% reduction in calibration time for senior agents
5 support programs scaled onto a unified calibration workflow
8,000+ agents enabled through the new QMS experience
The Challenge | Redesigning for Fairness at Scale
The existing calibration process at Meta was a manual, fragmented operation that struggled to keep pace with the organisation’s scale. Because each program (e.g., WhatsApp, Instagram, Facebook) managed reviews independently, the lack of a unified workflow led to three critical bottlenecks:
Operational Inefficiency
Manual data entry and cross-referencing were time-consuming
Subjective Bias
Reviewers often knew when they were being "calibrated," leading to human bias
Hard to scale
The manual process made it hard to scale standards across Meta’s support ecosystem.
The Solution | A Unified Standard for Meta’s Global Support
To solve the fragmentation across different programs, I designed a centralized calibration feature directly within Meta’s Quality Management System (QMS). The goal was to replace isolated spreadsheets and manual workflows with a single, automated source of truth that could be adopted by every program.
Global Oversight for Program Managers
The Calibrations Dashboard provides a unified command centre for program managers to monitor their global operations. By aggregating complex data into a single, intuitive view, managers can easily track the real-time progress of each calibration cycle.
Calibration dashboard for program managers
UI highlights
I transitioned the calibration process from external spreadsheets to a built-in function within the QMS. Users can now initiate and manage review cycles directly where they handle their daily support operations.
The status bars allow program managers to immediately see the progress of each calibration cycle. This view simplifies the oversight of multiple global queues, ensuring deadlines are met across the entire organisation.
I designed a "Smart Reminder" feature that automatically detects and flags if a selected case has already been reviewed. This preventative UI detail ensures data integrity and the consistency of the calibration results.
The Senior Agent Experience: A Unified Audit Environment
By integrating the full audit workflow into QMS, we eliminated the need for agents to toggle between multiple tools and spreadsheets. This streamlined environment was the primary driver for the 41% reduction in calibration time, allowing senior agents to focus on the nuance of the conversation rather than administrative overhead.
The “Create new audit” screen for senior agents
UI highlights
The system automatically assigns cases to auditors, removing the ability to "cherry-pick" easy or familiar interactions. This ensure a truly random and representative sample of agent performance, which is critical for maintaining high quality standards at Meta's scale.
By hosting the audit forms within QMS, we empowered individual programs to manage their own criteria while maintaining a unified technical framework. This ensured that every agent is evaluated through a consistent process that can scale as new programs are onboarded.
To eliminate "observer bias," the UI hides the fact that an audit is a calibration until the final submission dialog. By revealing the calibration status only at the end, we ensure senior agents perform an independent, honest evaluation before being notified that their score will be used for program-wide alignment.
Key Takeaways
The Meta QMS calibration project demonstrated how to scale fairness and efficiency across a global organisation. By replacing fragmented, manual spreadsheets with a unified workflow, we scaled to 5 major programs and reduced calibration time by 41%.