[smoke-detector] 🔍 Smoke Test Investigation - Smoke Copilot Run #18778382550: Safe-Outputs MCP Treats Config as Character Array

[github-actions]

🔍 Smoke Test Investigation - Run #18778382550

Summary

The Smoke Copilot workflow failed because the safe-outputs MCP server has a bug in its config parsing logic - it treats the GH_AW_SAFE_OUTPUTS_CONFIG JSON string as a character array instead of parsing it as JSON. This caused the MCP server to register 52 character-based tools (named '0' through '51') instead of the actual safe-outputs tools (create_issue, missing_tool), making them unavailable to the agent.

Failure Details

• Run: #18778382550
• Commit: fe2b5db
• Branch: copilot/update-copilot-agent-engine
• Trigger: workflow_dispatch
• Duration: 1.6 minutes
• Failed Jobs: create_issue (3s duration)

Root Cause Analysis

The Bug in Safe-Outputs MCP Server

From /tmp/gh-aw/aw-mcp/logs/run-18778382550/session-d1483906-329c-4e4d-8b77-3b0c3dcb099f.log:

[safe-outputs-mcp-server] Successfully parsed config from environment: "{\"create_issue\":{\"max\":1,\"min\":1},\"missing_tool\":{}}"

[safe-outputs-mcp-server] Final processed config: {"0":"{","1":"\"","2":"c","3":"r","4":"e","5":"a","6":"t","7":"e","8":"_","9":"i","10":"s","11":"s","12":"u","13":"e","14":"\"","15":":","16":"{","17":"\"","18":"m","19":"a","20":"x","21":"\"","22":":","23":"1","24":",","25":"\"","26":"m","27":"i","28":"n","29":"\"","30":":","31":"1","32":"}","33":",","34":"\"","35":"m","36":"i","37":"s","38":"s","39":"i","40":"n","41":"g","42":"_","43":"t","44":"o","45":"o","46":"l","47":"\"","48":":","49":"{","50":"}","51":"}"}

[safe-outputs-mcp-server] tools: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51

What's Happening:

1. MCP server successfully reads GH_AW_SAFE_OUTPUTS_CONFIG environment variable
2. The config string is valid JSON: {"create_issue":{"max":1,"min":1},"missing_tool":{}}
3. BUG: Instead of JSON.parse(configString), the server uses spread operator or similar: {...configString}
4. JavaScript spreads the string into an object where each character becomes a key: {"0": "{", "1": "\"", ...}
5. MCP server registers 52 tools (one per character) instead of 2 actual tools

Impact Chain

Safe-Outputs MCP Bug
    ↓
Agent sees tools: safe-outputs-0, safe-outputs-1, ..., safe-outputs-51
    ↓
Agent cannot find create_issue tool
    ↓
Agent completes successfully but creates no safe-outputs
    ↓
create_issue job fails: agent_output.json not found

Failed Jobs and Errors

Job Sequence

1. ✅ activation - succeeded (3s)
2. ✅ agent - succeeded (28s) - Agent ran but had no access to real safe-outputs tools
3. ✅ detection - succeeded (25s)
4. ❌ create_issue - failed (3s)
5. ⏭️ missing_tool - skipped

Error Details

Location: create_issue job

Error reading agent output file: ENOENT: no such file or directory, 
open '/tmp/gh-aw/safe-outputs/agent_output.json'

The agent never created this file because it couldn't use the safe-outputs MCP tools (they were registered with wrong names).

Investigation Findings

Commit Analysis

Commit: fe2b5db
Message: "Inline secrets directly in MCP config for Copilot engine"

Changes:

• Removed env var passthrough pattern for MCP configs
• Inlined GITHUB_PERSONAL_ACCESS_TOKEN directly in GitHub MCP config
• Inlined GH_AW_SAFE_OUTPUTS_CONFIG in safe-outputs MCP config
• Removed various env vars from execution step

Effect: This change likely exposed a pre-existing bug in the safe-outputs MCP server's config parsing logic. The config is now passed differently, triggering the incorrect parsing behavior.

Comparison with Issue #2280

Issue	Problem	Config Status	MCP Server Status
#2280	Malformed JSON (backslash escaping)	Invalid JSON syntax	Crashed during startup
This Issue	Character array parsing	Valid JSON	Started successfully but wrong tools

This is a DIFFERENT bug - the JSON is valid, but the MCP server's config parsing logic is broken.

Code Location

Likely Bug Location: /tmp/gh-aw/safe-outputs/mcp-server.cjs

Look for code like:

// WRONG - spreads string into character array
const config = {...process.env.GH_AW_SAFE_OUTPUTS_CONFIG};

// or
const config = Object.assign({}, process.env.GH_AW_SAFE_OUTPUTS_CONFIG);

Should be:

const config = JSON.parse(process.env.GH_AW_SAFE_OUTPUTS_CONFIG);

Recommended Actions

Critical Priority ⚠️

• Fix safe-outputs MCP server config parsing

  • Location: /tmp/gh-aw/safe-outputs/mcp-server.cjs (likely around config initialization)
  • Replace spread/Object.assign with JSON.parse()
  • Why: This is the root cause

• Add config parsing validation

  // Validate config after parsing
  const config = JSON.parse(process.env.GH_AW_SAFE_OUTPUTS_CONFIG || '{}');
  
  if (typeof config !== 'object' || Array.isArray(config)) {
    throw new Error('Config must be a JSON object');
  }
  
  // Log for debugging
  console.error('[safe-outputs-mcp-server] Parsed config keys:', Object.keys(config));

  • Why: Catch parsing issues early with clear errors

• Search codebase for the bug pattern

  # Find the broken code
  grep -r "{\.\.\..*GH_AW_SAFE_OUTPUTS_CONFIG" .
  grep -r "Object\.assign.*GH_AW_SAFE_OUTPUTS_CONFIG" .

  • Why: Identify exact location of the bug

High Priority

• Add integration test for MCP server config parsing

  test('safe-outputs MCP server parses config correctly', () => {
    process.env.GH_AW_SAFE_OUTPUTS_CONFIG = '{"create_issue":{"max":1}}';
    const server = new SafeOutputsMCPServer();
    const tools = server.listTools();
    
    expect(tools).toContain('create_issue');
    expect(tools).not.toContain('0');
    expect(tools).not.toContain('1');
  });

  • Why: Prevent regression

• Add startup diagnostics

  console.error('[safe-outputs-mcp-server] Config parsing debug:');
  console.error('  Raw value:', process.env.GH_AW_SAFE_OUTPUTS_CONFIG);
  console.error('  Parsed type:', typeof config);
  console.error('  Is array?', Array.isArray(config));
  console.error('  Keys:', Object.keys(config));
  console.error('  First key type:', typeof Object.keys(config)[0]);

  • Why: Better debugging for future issues

Medium Priority

• Review all MCP server config parsing

  • Check if other MCP servers have similar bugs
  • Standardize config parsing across all MCP servers
  • Why: This could affect other servers

• Document expected config format

  • Add schema validation for MCP configs
  • Document in safe-outputs README
  • Why: Prevent similar issues

Prevention Strategies

1. Config Validation: Add JSON schema validation for all MCP configs

   const Ajv = require('ajv');
   const ajv = new Ajv();
   const schema = {
     type: 'object',
     properties: {
       create_issue: { type: 'object' },
       missing_tool: { type: 'object' }
     }
   };
   const validate = ajv.compile(schema);
   if (!validate(config)) {
     throw new Error('Invalid config: ' + JSON.stringify(validate.errors));
   }

2. Type Checking: Ensure config is parsed as object, not spread as string

   if (typeof config !== 'object' || typeof config[0] === 'string') {
     throw new Error('Config was spread as string - use JSON.parse()');
   }

3. Integration Tests: Test MCP server startup with actual environment variables

4. Code Review: Add checklist item: "Are config values properly JSON.parse()'d?"

Technical Details

Environment Context

• Node.js: v24.10.0
• Copilot CLI: 0.0.349
• Safe-Outputs MCP Server: v1.0.0
• Staged Mode: true

Config Details

• Variable: GH_AW_SAFE_OUTPUTS_CONFIG
• Length: 62 characters
• Expected Value: {"create_issue":{"max":1,"min":1},"missing_tool":{}}
• Parsing Result: String spread into 52 character-based keys

Tool Registration Evidence

Expected Tools:

• safe-outputs-create_issue
• safe-outputs-missing_tool

Actual Tools (from logs):

safe-outputs-0, safe-outputs-1, safe-outputs-2, ..., safe-outputs-51

Each tool description: "Custom safe-job: N" where N is 0-51.

Historical Context

Related Issues:

• [smoke-detector] 🔍 Smoke Test Investigation - Smoke Copilot: Safe-Outputs MCP Crashes Due to Malformed Config JSON #2280: Safe-outputs config malformed JSON (different issue - JSON syntax error)
• [smoke-detector] 🔍 Smoke Test Investigation - Smoke OpenCode Run #18722224746: Agent Does Not Use Safe-Outputs MCP Tools #2143: OpenCode agent didn't use safe-outputs (different engine)
• [smoke-detector] 🔍 Smoke Test Investigation - Smoke Copilot: Base64 MCP Config vs JSON Expected #2262: Copilot base64 vs JSON config (different issue - format mismatch)

Pattern Classification:

• Pattern ID: COPILOT_SAFE_OUTPUTS_CONFIG_AS_STRING
• Category: MCP Server Bug
• Severity: Critical
• Is Flaky: No - deterministic bug
• First Occurrence: 2025-10-24T11:28:02Z

Investigation saved to: /tmp/gh-aw/cache-memory/investigations/2025-10-24-18778382550.json

---

Investigation Metadata:

• Investigator: Smoke Detector
• Investigation Run: #18778421165
• Pattern ID: COPILOT_SAFE_OUTPUTS_CONFIG_AS_STRING
• Severity: Critical
• Is Flaky: No
• Related Pattern: COPILOT_SAFE_OUTPUTS_MALFORMED_CONFIG ([smoke-detector] 🔍 Smoke Test Investigation - Smoke Copilot: Safe-Outputs MCP Crashes Due to Malformed Config JSON #2280)

Labels: smoke-test, investigation, copilot, safe-outputs, mcp, critical, bug

AI generated by Smoke Detector - Smoke Test Failure Investigator

AI generated by Smoke Detector - Smoke Test Failure Investigator