Dashboard
The Agent Metrics Dashboard is the live data layer for the MSP AI Platform agent performance program. It replaces manual data collection by pulling four of the six benchmark metrics directly from ConnectWise Manage and Microsoft Graph, then packaging them for import into the Agent Benchmark Log.
This tool does not replace the Agent Benchmark Log — it feeds it. The benchmark log is the governance record. This dashboard is the data source.
- Resolution rate from CW Manage ticket disposition
- Escalation rate from post-AI assignment history
- Time saved from CW open-to-close timestamps
- Cost per interaction from Copilot seat licensing and Graph interaction count
- Accuracy — requires human spot-check of 20-30 ticket sample
- User satisfaction — requires survey from end users
- Both shown as badges in the dashboard: "Spot-check required" and "Survey pending"
In ConnectWise Manage go to Setup Tables → Custom Fields → Service Ticket. Create a Text field (100 chars) with this exact caption:
Field caption: AI_Agent
Field type: Text (100 characters)When an AI agent handles a ticket, the agent writes its identifier to this field. Values must match the cwTag in the AGENTS config array exactly — case-sensitive.
Grant the Function Managed Identity Reports.Read.All application permission in Azure AD. Admin consent required.
az ad app permission add \
--id \
--api 00000003-0000-0000-c000-000000000000 \
--api-permissions 230c1aed-a721-4c5d-9cb4-a90514e508ef=Role
az ad app permission admin-consent --id Update the config block at the top of the script before setting DEMO_MODE to false:
var DEMO_MODE = false; // change last, after all else is validated
var PROXY_BASE = 'https://.azurewebsites.net/api';
var COPILOT_SEATS = 60; // your licensed Copilot seat count
var COPILOT_SEAT_COST = 30; // monthly cost per seat in USD Collapsible yellow strip containing the four setup steps with exact field names and Azure CLI commands. Collapse once setup is complete.
Four-cell table documenting exact proxy routes, target endpoints, auth methods, and pass-through parameters. Build spec for the Azure Function configuration.
One tab per agent plus "All Agents". Each tab has a RAG dot. Selecting a tab filters all panels to that agent's data.
Left panel. Bar chart rows for each metric with current value, RAG color, and baseline delta. Accuracy and satisfaction shown as manual badges — not calculated.
Right panel. Active users, total interactions, seat cost, and cost per interaction. Feature usage breakdown (Teams, Outlook, Word, Excel, etc.). Cost formula callout at the bottom.
Live table of AI-tagged tickets: ID, summary, agent tag, status chip (resolved/escalated/open), AI close time, date. Filtered to selected agent. 10 rows max.
"Sync to Benchmark Log" button and last-synced timestamp. Packages the four auto-calculated metrics into a JSON payload and writes to localStorage.
The sync operation bridges live data with the governance record. It pre-fills the log entry form — you review before saving.
- Click "Sync to Benchmark Log" in the sync bar
- JSON payload written to localStorage["msp-benchmark-sync"]
- Open tools/tl-agent-benchmark-log.html in the same browser
- Green banner appears: "Metrics from API Dashboard Ready"
- Click "Import and Pre-fill Entry" — entry modal opens with four fields pre-filled
- Enter accuracy (spot-check result) and satisfaction (survey average) manually
- Notes field auto-records sync source and date
- Click Save — entry logged with full attribution
// Payload structure written to localStorage
{
agentHint: "IVR-Triage",
date: "2025-05-18",
source: "API Dashboard — CW Manage + M365 Copilot",
metrics: { res: 82.3, esc: 17.7, time: 3.1, cost: 0.021, acc: null, sat: null },
notes: "Auto-synced from Agent Metrics Dashboard on May 18, 2025..."
}DEMO_MODE at the top of the script block controls the data source. The "Switch to Live" button in the header toggles it at runtime.
Edit the AGENTS array to match your deployed agents. The cwTag must exactly match the CW custom field value — case-sensitive.
var AGENTS = [
{ id: 'all', label: 'All Agents', cwTag: null },
{ id: 'IVR-Triage', label: 'IVR Triage', cwTag: 'IVR-Triage' },
{ id: 'NOC-Summarizer', label: 'NOC Summarizer', cwTag: 'NOC-Summarizer' },
{ id: 'Ticket-Classifier', label: 'Ticket Classifier', cwTag: 'Ticket-Classifier' },
];