Refactor Tavily MCP configuration to shared workflow and add mcp-servers import support (#1280)

* Initial plan

* Refactor Tavily MCP configuration to shared workflow and fix import merging

- Created shared/tavily-mcp.md for reusable Tavily MCP configuration
- Updated scout.md to use imports field instead of inline mcp-servers
- Fixed bug in mergeTools where newline-separated JSON objects weren't being parsed correctly
- Import functionality now properly handles multiple JSON objects from different imports

Co-authored-by: pelikhan <[email protected]>

* Add integration tests for imports functionality

Added comprehensive tests to verify that imports work correctly with single and multiple imported files. Tests validate that MCP configurations are properly merged from imported files.

Co-authored-by: pelikhan <[email protected]>

* Complete: Tavily MCP refactoring with imports and tests

Co-authored-by: pelikhan <[email protected]>

* Add support for importing mcp-servers from shared workflows

- Updated shared/tavily-mcp.md to use mcp-servers instead of tools
- Added mcp-servers to included_file_schema.json
- Added MergedMCPServers field to ImportsResult struct
- Implemented extractMCPServersFromContent and mergeMCPServers functions
- Updated ProcessImportsFromFrontmatterWithManifest to extract and merge mcp-servers
- Updated compiler to merge imported mcp-servers with top-level mcp-servers
- Added test for importing mcp-servers (TestCompileWorkflowWithMCPServersImport)

Co-authored-by: pelikhan <[email protected]>

* Address PR feedback: improve MCP servers schema and documentation

- Copied full MCP servers schema from main workflow to included file schema with $defs for stdio_mcp_tool and http_mcp_tool
- Refactored mergeMCPServers to ensure clean map creation instead of modifying input
- Updated documentation to mention MCP servers can be imported:
  - include-directives.md: Added frontmatter merging section for mcp-servers with example
  - packaging-imports.md: Added MCP server import examples and updated existing examples
- All tests passing

Co-authored-by: pelikhan <[email protected]>

* Merge main and recompile all workflows

- Merged latest changes from main branch
- Recompiled all workflows to update lock files
- All tests passing including import-related tests
- Tavily MCP server still correctly configured in scout workflow

Co-authored-by: pelikhan <[email protected]>

* Refactor discussion and issue creation scripts for improved readability and error handling

- Simplified the parsing and validation of agent output in create_discussion.js and create_issue.js.
- Enhanced error messages for better debugging.
- Improved the handling of discussion categories and issue labels.
- Added checks for empty titles and bodies, ensuring defaults are set appropriately.
- Streamlined the creation process for discussions and issues, including better logging of created items.
- Updated summary writing to include details of created discussions and issues.

* Refactor: Move import/merge code to imports.go and fix linting

- Created pkg/workflow/imports.go with mergeTools and mergeMCPServers functions
- Removed these functions from compiler.go for better code organization
- Fixed linting issue: removed unnecessary nil check in mergeMCPServers (S1031)
- All tests passing including import-related tests
- Scout workflow still compiles correctly with Tavily MCP server

Co-authored-by: pelikhan <[email protected]>

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: pelikhan <[email protected]>
Co-authored-by: Peli de Halleux <[email protected]>

Author: Copilot

.github/workflows/scout.lock.yml

@@ -13,13 +13,11 @@ permissions:
   actions: read
 roles: [admin, maintainer, write]
 engine: copilot
+imports:
+  - shared/tavily-mcp.md
 tools:
   cache-memory:
     retention-days: 7
-mcp-servers:
-  tavily:
-    url: "https://mcp.tavily.com/mcp/?tavilyApiKey=${{ secrets.TAVILY_API_KEY }}"
-    allowed: ["*"]
 safe-outputs:
   add-comment:
     max: 1

.github/workflows/scout.md

@@ -0,0 +1,6 @@
+---
+mcp-servers:
+  tavily:
+    url: "https://mcp.tavily.com/mcp/?tavilyApiKey=${{ secrets.TAVILY_API_KEY }}"
+    allowed: ["*"]
+---

.github/workflows/shared/tavily-mcp.md

@@ -298,8 +298,9 @@ Imports only a specific section from a markdown file using the section header.
 
 ### Frontmatter Merging
 
-- **Only `tools:` frontmatter** is allowed in imported files; other entries give a warning
+- **Only `tools:` and `mcp-servers:` frontmatter** is allowed in imported files; other entries give a warning
 - **Tool merging**: `allowed:` tools are merged across all imported files
+- **MCP server merging**: MCP servers defined in imported files are merged with the main workflow
 
 **Example Tool Merging:**
 
@@ -326,6 +327,30 @@ tools:
 
 **Result:** Final workflow has `github.allowed: [get_issue, add_issue_comment, update_issue]` and Claude Edit tool.
 
+**Example MCP Server Merging:**
+
+```aw wrap
+# Base workflow
+---
+on: issues
+engine: copilot
+---
+
+{{#import: shared/tavily-mcp.md}}  # Adds Tavily MCP server
+```
+
+```aw wrap
+# shared/tavily-mcp.md
+---
+mcp-servers:
+  tavily:
+    url: "https://mcp.tavily.com/mcp/?tavilyApiKey=${{ secrets.TAVILY_API_KEY }}"
+    allowed: ["*"]
+---
+```
+
+**Result:** Final workflow has the Tavily MCP server configured and available to the AI engine.
+
 ### Import Processing During Add
 
 When adding a workflow with the `add` command, local file references in import directives are automatically converted to workflow specifications:
@@ -353,9 +378,12 @@ The `imports:` field in frontmatter provides an alternative way to import files,
 imports:
   - shared/common-tools.md
   - shared/security-setup.md
+  - shared/tavily-mcp.md
 ---
 ```
 
+Imports can include both tool configurations and MCP server definitions from shared files.
+
 ### Import Processing
 
 Like `@include` directives, the imports field is processed during the `add` command:
@@ -427,6 +455,44 @@ gh aw update experimental
 
 ### Example 3: Using Imports for Modular Workflows
 
+Create a shared MCP server configuration:
+
+```aw wrap
+# .github/workflows/shared/tavily-mcp.md
+---
+mcp-servers:
+  tavily:
+    url: "https://mcp.tavily.com/mcp/?tavilyApiKey=${{ secrets.TAVILY_API_KEY }}"
+    allowed: ["*"]
+---
+```
+
+Create a workflow that imports the MCP server:
+
+```aw wrap
+# .github/workflows/research-agent.md
+---
+on:
+  issues:
+    types: [opened]
+
+imports:
+  - shared/tavily-mcp.md
+
+tools:
+  github:
+    allowed: [add_issue_comment]
+---
+
+# Research Agent
+
+Perform web research using Tavily and respond to issues.
+```
+
+The final workflow will have both the Tavily MCP server and GitHub tools configured.
+
+### Example 4: Using Imports for Common Tools
+
 Create a base workflow with common tools:
 
 ```aw wrap
@@ -466,7 +532,7 @@ Handle new issues with automated responses.
 
 The final workflow will have all three GitHub tools: `get_issue`, `get_pull_request`, and `add_issue_comment`.
 
-### Example 4: Creating Reusable Components
+### Example 5: Creating Reusable Components
 
 **Shared security notice:**
 

docs/src/content/docs/guides/packaging-imports.md

@@ -37,8 +37,9 @@ Imports only a specific section from a markdown file using the section header.
 
 ## Frontmatter Merging
 
-- **Only `tools:` frontmatter** is allowed in imported files, other entries give a warning.
+- **Only `tools:` and `mcp-servers:` frontmatter** is allowed in imported files, other entries give a warning.
 - **Tool merging**: `allowed:` tools are merged across all imported files
+- **MCP server merging**: MCP servers defined in imported files are merged with the main workflow
 
 ### Example Tool Merging
 ```aw wrap
@@ -64,6 +65,30 @@ tools:
 
 **Result**: Final workflow has `github.allowed: [get_issue, add_issue_comment, update_issue]` and Claude Edit tool.
 
+### Example MCP Server Merging
+
+```aw wrap
+# Base workflow
+---
+on: issues
+engine: copilot
+---
+
+@import shared/tavily-mcp.md  # Adds Tavily MCP server
+```
+
+```aw wrap
+# shared/tavily-mcp.md
+---
+mcp-servers:
+  tavily:
+    url: "https://mcp.tavily.com/mcp/?tavilyApiKey=${{ secrets.TAVILY_API_KEY }}"
+    allowed: ["*"]
+---
+```
+
+**Result**: Final workflow has the Tavily MCP server configured and available to the AI engine.
+
 ## Import Path Resolution
 
 - **Relative paths**: Resolved relative to the importing file

docs/src/content/docs/reference/include-directives.md

@@ -87,10 +87,11 @@ type FrontmatterResult struct {
 
 // ImportsResult holds the result of processing imports from frontmatter
 type ImportsResult struct {
-	MergedTools    string   // Merged tools configuration from all imports
-	MergedEngines  []string // Merged engine configurations from all imports
-	MergedMarkdown string   // Merged markdown content from all imports
-	ImportedFiles  []string // List of imported file paths (for manifest)
+	MergedTools      string   // Merged tools configuration from all imports
+	MergedMCPServers string   // Merged mcp-servers configuration from all imports
+	MergedEngines    []string // Merged engine configurations from all imports
+	MergedMarkdown   string   // Merged markdown content from all imports
+	ImportedFiles    []string // List of imported file paths (for manifest)
 }
 
 // ExtractFrontmatterFromContent parses YAML frontmatter from markdown content string
@@ -387,6 +388,7 @@ func ProcessImportsFromFrontmatterWithManifest(frontmatter map[string]any, baseD
 
 	// Process each import
 	var toolsBuilder strings.Builder
+	var mcpServersBuilder strings.Builder
 	var markdownBuilder strings.Builder
 	var engines []string
 	var processedFiles []string
@@ -451,13 +453,20 @@ func ProcessImportsFromFrontmatterWithManifest(frontmatter map[string]any, baseD
 		if err == nil && engineContent != "" {
 			engines = append(engines, engineContent)
 		}
+
+		// Extract mcp-servers from imported file
+		mcpServersContent, err := extractMCPServersFromContent(string(content))
+		if err == nil && mcpServersContent != "" && mcpServersContent != "{}" {
+			mcpServersBuilder.WriteString(mcpServersContent + "\n")
+		}
 	}
 
 	return &ImportsResult{
-		MergedTools:    toolsBuilder.String(),
-		MergedEngines:  engines,
-		MergedMarkdown: markdownBuilder.String(),
-		ImportedFiles:  processedFiles,
+		MergedTools:      toolsBuilder.String(),
+		MergedMCPServers: mcpServersBuilder.String(),
+		MergedEngines:    engines,
+		MergedMarkdown:   markdownBuilder.String(),
+		ImportedFiles:    processedFiles,
 	}, nil
 }
 
@@ -810,6 +819,28 @@ func extractToolsFromContent(content string) (string, error) {
 	return strings.TrimSpace(string(toolsJSON)), nil
 }
 
+// extractMCPServersFromContent extracts mcp-servers section from frontmatter as JSON string
+func extractMCPServersFromContent(content string) (string, error) {
+	result, err := ExtractFrontmatterFromContent(content)
+	if err != nil {
+		return "{}", nil // Return empty object on error
+	}
+
+	// Extract mcp-servers section
+	mcpServers, exists := result.Frontmatter["mcp-servers"]
+	if !exists {
+		return "{}", nil
+	}
+
+	// Convert to JSON string
+	mcpServersJSON, err := json.Marshal(mcpServers)
+	if err != nil {
+		return "{}", nil
+	}
+
+	return strings.TrimSpace(string(mcpServersJSON)), nil
+}
+
 // extractEngineFromContent extracts engine section from frontmatter as JSON string
 func extractEngineFromContent(content string) (string, error) {
 	result, err := ExtractFrontmatterFromContent(content)