[Schema Consistency] 🔍 Schema Consistency Check - Required Fields & Error Documentation Gap (Nov 2, 2025)

[github-actions[bot]]

🔍 Schema Consistency Check - November 2, 2025

Overview

Today's analysis using Strategy-003 (Required Field Enforcement + Error Message Documentation) uncovered a critical gap between schema-level validation and runtime enforcement. The headline finding: all three schemas have zero required fields, yet the compiler enforces multiple implicit requirements through 328 error messages that are completely undocumented for end users.

Key Statistics:

• Inconsistencies Found: 6 critical issues
• Error Messages: 328 (0% documented)
• Workflows Analyzed: 67 (100% have 'on' field despite it not being required)
• Schemas Checked: 3 (main, included, mcp_config)

Full Report Details

Critical Issues

1. Schema Has NO Required Fields Despite Implicit Requirements ⚠️

Finding: All three schemas have no "required" array defined:

• main_workflow_schema.json: No required fields
• included_file_schema.json: No required fields
• mcp_config_schema.json: No required fields

Reality: The compiler implicitly requires several fields. For example, all 67 production workflows have the on: field (100% usage), yet the schema doesn't mark it as required.

Impact:

• Users can create workflows without 'on' triggers (passes schema validation)
• Workflow will fail at runtime with unclear behavior
• No early validation feedback from IDEs or schema validators

File Reference: pkg/parser/schemas/main_workflow_schema.json:1 (missing "required": ["on"])

---

2. MCP HTTP Tool URL Required Only at Runtime ⚠️

Finding: The MCP config schema doesn't enforce the url field as required via JSON Schema.

Code Enforcement (pkg/workflow/mcp-config.go:827):

return nil, fmt.Errorf("http MCP tool '%s' missing required 'url' field", toolName)

Impact:

• Schema validation passes without url
• Error only appears at compile time (not schema validation time)
• Users don't get IDE validation or early warning

Recommendation: Add conditional required array to http_mcp_tool definition in schema.

---

3. 328 Error Messages - ZERO Documentation 🔴

Critical Finding: The codebase contains 328 error messages (via errors.New and fmt.Errorf), but there is:

• ❌ No docs/src/content/docs/troubleshooting/ directory
• ❌ No error reference documentation
• ❌ No searchable error message index

Sample Undocumented Errors:

Error Message	File	Impact
"cannot use 'command' with 'issues' in the same workflow"	compiler.go	Command trigger conflicts - NOT in triggers.md
"strict mode: 'network' configuration is required"	strict_mode.go	Strict mode requirement - NOT in frontmatter.md
"job name cannot be empty"	jobs.go	Validation rule - NOT in docs
"safe-outputs.{feature} configuration is required"	15+ files	Repeated pattern - NOT explained
"http MCP tool '%s' missing required 'url' field"	mcp-config.go	MCP requirement - NOT in mcps.md

Impact: Users encountering errors cannot:

• Search documentation for solutions
• Understand error context
• Find migration paths (except stop-time which has EXCELLENT error message)

---

4. Conditional Requirements NOT in Schema 🔴

Finding: Many requirements are conditional but not encoded in JSON Schema.

Examples:

1. Command Trigger Conflicts (pkg/workflow/compiler.go:1221):

   conflictingEvents := []string{"issues", "issue_comment", "pull_request", "pull_request_review_comment"}
   // If command is used, these 4 events are forbidden

   • ❌ NOT in schema using JSON Schema not or if/then/else
   • ❌ NOT documented with anti-examples

2. MCP Command XOR Container (pkg/workflow/mcp-config.go):

   return fmt.Errorf("tool '%s' mcp configuration must specify either 'command' or 'container'", toolName)

   • ✅ GOOD: MCP tool schemas DO use anyOf, not, allOf correctly
   • ❌ BAD: Main schema doesn't leverage this pattern for other fields

3. Strict Mode Network Requirement (pkg/workflow/strict_mode.go):

   return fmt.Errorf("strict mode: 'network' configuration is required")

   • ❌ NOT documented in strict mode docs
   • ❌ No schema conditional for strict mode implications

Positive Example: The MCP tool schemas are GOLD STANDARD with proper use of JSON Schema conditionals.

Recommendation: Apply MCP schema conditional pattern to main schema for command trigger conflicts, strict mode requirements, and feature co-dependencies.

---

5. Default Values: Code vs Schema Mismatch ⚠️

Finding: Code implements default values that are NOT documented in schema.

Code Defaults:

1. Engine: Defaults to "claude" if not specified (pkg/workflow/compiler.go:716)
2. Timeout: Uses engine defaults if not set (compiler.go:829)
3. Reaction: Auto-enables "eyes" for command triggers (compiler.go:1241)

Schema Defaults: NONE defined using "default" keyword

Impact:

• IDE cannot show default values
• Users don't know what happens when fields are omitted
• Schema doesn't reflect actual runtime behavior

---

6. Excellent Error Example: stop-time Deprecation ✅

Positive Finding: The stop-time deprecation error is a GOLD STANDARD:

return nil, fmt.Errorf("'stop-time' is no longer supported at the root level. Please move it under the 'on:' section and rename to 'stop-after:'.\n\nExample:\n---\non:\n  schedule:\n    - cron: \"0 9 * * 1\"\n  stop-after: \"%s\"\n---", stopTimeValue)

Why This is Excellent:

• ✅ Clear explanation of what's wrong
• ✅ Migration path provided
• ✅ Working example included
• ✅ Preserves user's value in example

Recommendation: Apply this pattern to ALL deprecation and validation errors.

---

Documentation Gaps

Missing Documentation Categories

1. No Troubleshooting Section:

   • Directory docs/src/content/docs/troubleshooting/ does NOT exist
   • Should include: errors.md, common-issues.md, migration-guides.md

2. Error Messages NOT Documented:

   • 328 error messages found in code
   • 0 searchable in documentation
   • Users must read source code or use trial-and-error

3. Conditional Requirements NOT Explained:

   • Command + event conflicts: documented behavior but no anti-examples
   • Strict mode requirements: 6 requirements found, only 4 fully documented
   • MCP network/container dependency: not documented

4. Validation Timing NOT Clarified:

   • Schema validation time (JSON Schema)
   • Compile time (Go compiler)
   • Runtime (during workflow execution)
   • Users don't know WHEN errors will occur

---

Schema Improvements Needed

1. Add Required Fields Array

{
  "$schema": "(redacted)",
  "title": "Agentic Workflow",
  "type": "object",
  "required": ["on"],  // ← ADD THIS
  "properties": { ... }
}

2. Add Default Values

{
  "engine": {
    "type": "object",
    "default": {"id": "claude"},  // ← ADD THIS
    "description": "AI engine configuration (defaults to Claude if not specified)"
  }
}

3. Add Examples Field

{
  "tools": {
    "type": "object",
    "examples": [  // ← ADD THIS
      {"bash": ["ls", "cat"], "edit": null},
      {"github": {"allowed": ["issues:*"]}}
    ]
  }
}

4. Add Conditionals for Field Relationships

{
  "if": {
    "properties": {
      "on": {"properties": {"command": {"type": "string"}}}
    }
  },
  "then": {
    "not": {
      "properties": {
        "on": {
          "anyOf": [
            {"required": ["issues"]},
            {"required": ["issue_comment"]},
            {"required": ["pull_request"]},
            {"required": ["pull_request_review_comment"]}
          ]
        }
      }
    }
  }
}

---

Positive Findings ✅

1. All 67 Workflows Comply: Every production workflow has 'on:' field despite it not being required
2. MCP Schemas Excellent: MCP tool schemas use JSON Schema conditionals correctly
3. Consistent Error Patterns: safe-outputs errors follow consistent naming
4. Stop-time Migration: Exemplary deprecation error with migration path
5. Implicit Requirements Work: Despite lack of schema enforcement, runtime validation prevents bad configs

---

Recommendations

High Priority

1. Add "required": ["on"] to main_workflow_schema.json
2. Create troubleshooting documentation:
   • docs/src/content/docs/troubleshooting/errors.md
   • docs/src/content/docs/troubleshooting/common-issues.md
3. Document conditional requirements with anti-examples
4. Add default values to schema definitions

Medium Priority

1. Extract error messages into constants for documentation generation
2. Add JSON Schema conditionals for field relationships (following MCP pattern)
3. Document validation timing (schema vs compile vs runtime)
4. Create error message style guide (follow stop-time example)

Low Priority

1. Add schema examples field for IDE autocomplete
2. Create searchable error index in documentation
3. Add troubleshooting links to error messages in code

---

Strategy Performance

Strategy Used: Required Field Enforcement + Error Message Documentation
Effectiveness: Very High
Findings: 6 critical issues, 328 undocumented error messages
Unique Discoveries:

• No required fields in any schema
• 328 error messages with zero documentation
• MCP schemas as gold standard for conditionals
• stop-time as exemplary error message pattern

Should Reuse: YES - This strategy reveals runtime vs schema gaps that other strategies miss.

---

Files Analyzed

Schemas: pkg/parser/schemas/*.json (all 3)
Compiler/Parser: pkg/workflow/*.go, pkg/parser/*.go (all files)
Documentation: docs/src/content/docs/**/*.md (all reference docs)
Workflows: .github/workflows/*.md (all 67 files)

---

Next Steps

• Add "required": ["on"] to main schema
• Create troubleshooting documentation directory
• Document top 20 most common error messages
• Add anti-examples for command trigger conflicts
• Create error message extraction script
• Add JSON Schema conditionals to main schema
• Document default value behavior
• Create validation timing reference

AI generated by Schema Consistency Checker

[pelikhan]

/plan

[github-actions[bot]]

✅ Agentic Plan Command completed successfully.