FEAT: Add cross-repo CI support for BCP development

.github/PULL_REQUEST_TEMPLATE.MD

@@ -12,6 +12,14 @@ Only one reference is required - either GitHub issue OR ADO Work Item.
 <!-- External contributors: GitHub Issue -->
 > GitHub Issue: #<ISSUE_NUMBER>
 
+<!-- 
+### Cross-Repo Dependency (BCP Development Only)
+Uncomment and fill in if this PR depends on changes in mssql-tds repo.
+The CI pipeline will automatically download mssql-py-core artifacts from the linked PR.
+
+Depends-On: mssql-tds#<PR_NUMBER>
+-->
+
 -------------------------------------------------------------------
 ### Summary   
 <!-- Insert your summary of changes below. Minimum 10 characters required. -->  

docs/plan-crossRepoDevBcp.prompt.md

@@ -0,0 +1,58 @@
+## Plan: Cross-Repo Development for mssql-python + mssql-tds BCP
+
+Enable parallel development across `mssql-python` (GitHub, `public` ADO project) and `mssql-tds` (ADO repo, `mssql-rs` project) with linked PRs for BCP feature implementation using Rust bindings.
+
+### Steps
+
+1. **Add artifact publishing to `mssql-tds` PR pipeline**: Modify [validation-pipeline.yml](../../mssql-tds/.pipeline/validation-pipeline.yml) to publish `mssql_py_core` `.so` files as named pipeline artifacts with PR metadata, e.g., `mssql-py-core-$(Build.BuildId)`. Ensure artifacts are published for PR builds (currently may be filtered).
+
+2. **Create cross-repo artifact download in `mssql-python` PR pipeline**: Add a conditional stage to [pr-validation-pipeline.yml](../eng/pipelines/pr-validation-pipeline.yml) that: parses PR description for `Depends-On: mssql-tds#<PR>`, uses `DownloadPipelineArtifact@2` with `project: mssql-rs` to fetch `.so` artifacts for all platforms (Linux/macOS/Alpine), places them in `mssql_python/` folder. Skip BCP tests if no `Depends-On` found.
+
+3. **Verify cross-project service connection**: Confirm `Public Artifact Access` connection in `public` project can access `mssql-rs` pipelines. If not, create via Project Settings → Service Connections → Azure DevOps pointing to `mssql-rs`.
+
+4. **Update PR template with Depends-On section**: Add commented section to [PULL_REQUEST_TEMPLATE.MD](../.github/PULL_REQUEST_TEMPLATE.MD) for BCP developers to specify cross-repo dependency. Add after the GitHub Issue section:
+   ```markdown
+   <!-- 
+   ### Cross-Repo Dependency (BCP Development Only)
+   If this PR depends on changes in mssql-tds repo, uncomment and specify the PR number:
+   Depends-On: mssql-tds#<PR_NUMBER>
+   -->
+   ```
+
+5. **Create local dev script `scripts/build-py-core.sh`**: Add to `mssql-python` repo—expects `mssql-tds` as sibling folder, runs `maturin develop` or `maturin build` in `../mssql-tds/mssql-py-core/`, copies `.so` to `mssql_python/` with platform-appropriate naming.
+
+6. **Create devcontainer for dual-repo development**: Add `.devcontainer/` to `mssql-python` with Rust + Python + maturin, expects both repos mounted as siblings, auto-runs `build-py-core.sh` on start.
+
+7. **Document the workflow**: Add section to `CONTRIBUTING.md` or `docs/cross-repo-dev.md` explaining: sibling folder structure, PR template usage, local dev script.
+
+### Folder Structure (Local Development)
+
+```
+parent-folder/
+├── mssql-python/     # GitHub repo
+└── mssql-tds/        # ADO repo (cloned alongside)
+```
+
+### CI Flow Diagram
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│  mssql-tds repo (mssql-rs project)                              │
+│                                                                 │
+│  PR #123: "Add BCP bindings"                                    │
+│     └──► Pipeline builds mssql_py_core .so (all platforms)      │
+│              └──► Publishes artifact: mssql-py-core-{BuildId}   │
+└─────────────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────────┐
+│  mssql-python repo (public project)                             │
+│                                                                 │
+│  PR #456: "Add Cursor.bulkcopy() API"                           │
+│     │  PR Description: "Depends-On: mssql-tds#123"              │
+│     │                                                           │
+│     └──► Pipeline parses PR desc → extracts mssql-tds#123       │
+│              └──► Downloads all platform .so artifacts          │
+│              └──► Runs tests on each platform with .so          │
+└─────────────────────────────────────────────────────────────────┘
+```

eng/pipelines/pr-validation-pipeline.yml

@@ -434,6 +434,12 @@ jobs:
         sqlVersion: 'SQL2025'
 
   steps:
+  # Download mssql-py-core artifacts from mssql-tds if Depends-On is specified in PR
+  - template: templates/download-mssql-tds-artifacts.yml
+    parameters:
+      osType: 'MacOS'
+      architecture: 'x64'
+
   - task: UsePythonVersion@0
     inputs:
       versionSpec: '3.13'
@@ -546,6 +552,12 @@ jobs:
         useAzureSQL: 'false'
 
   steps:
+  # Download mssql-py-core artifacts from mssql-tds if Depends-On is specified in PR
+  - template: templates/download-mssql-tds-artifacts.yml
+    parameters:
+      osType: 'Linux'
+      architecture: 'x64'
+
   - script: |
       # Create a Docker container for testing
       docker run -d --name test-container-$(distroName) \
@@ -846,6 +858,12 @@ jobs:
         archName: 'arm64'
 
   steps:
+  # Download mssql-py-core artifacts from mssql-tds if Depends-On is specified in PR
+  - template: templates/download-mssql-tds-artifacts.yml
+    parameters:
+      osType: 'Linux'
+      architecture: 'ARM64'
+
   - script: |
       # Set up Docker buildx for multi-architecture support
       docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
@@ -1474,6 +1492,12 @@ jobs:
     vmImage: 'ubuntu-latest'
 
   steps:
+  # Download mssql-py-core artifacts from mssql-tds if Depends-On is specified in PR
+  - template: templates/download-mssql-tds-artifacts.yml
+    parameters:
+      osType: 'Alpine'
+      architecture: 'x64'
+
   - script: |
       # Set up Docker buildx for multi-architecture support
       docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
@@ -1716,6 +1740,12 @@ jobs:
     vmImage: 'ubuntu-latest'
 
   steps:
+  # Download mssql-py-core artifacts from mssql-tds if Depends-On is specified in PR
+  - template: templates/download-mssql-tds-artifacts.yml
+    parameters:
+      osType: 'Alpine'
+      architecture: 'ARM64'
+
   - script: |
       # Set up Docker buildx for multi-architecture support
       docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

eng/pipelines/templates/download-mssql-tds-artifacts.yml

@@ -0,0 +1,112 @@
+# Template to download mssql-py-core artifacts from mssql-tds pipeline
+# Used for cross-repo BCP development - downloads .so/.whl files built by mssql-tds
+#
+# Usage in PR description:
+#   Depends-On: mssql-tds#<PR_NUMBER>
+#
+# Parameters:
+#   osType: Linux, Alpine, MacOS
+#   architecture: x64, ARM64
+
+parameters:
+- name: osType
+  type: string
+  default: 'Linux'
+  values:
+  - Linux
+  - Alpine
+  - MacOS
+- name: architecture
+  type: string
+  default: 'x64'
+  values:
+  - x64
+  - ARM64
+
+steps:
+# Parse PR description for Depends-On: mssql-tds#<PR_NUMBER>
+- bash: |
+    echo "Checking for cross-repo dependency in PR description..."
+
+    if [ "$(Build.Reason)" = "PullRequest" ]; then
+      PR_NUMBER=$(System.PullRequest.PullRequestNumber)
+      echo "PR Number: $PR_NUMBER"
+
+      # Fetch PR description from GitHub API (no auth needed for public repo)
+      API_RESPONSE=$(curl -s "https://api.github.com/repos/microsoft/mssql-python/pulls/$PR_NUMBER")
+
+      # Debug: show API response status
+      echo "API Response (first 300 chars):"
+      echo "$API_RESPONSE" | head -c 300
+      echo ""
+
+      # Extract body using jq
+      PR_BODY=$(echo "$API_RESPONSE" | jq -r '.body // ""')
+
+      echo "PR Body length: ${#PR_BODY}"
+      echo "PR Body preview:"
+      echo "$PR_BODY" | head -c 300
+      echo ""
+
+      # Extract mssql-tds PR number using sed (works on both Linux and macOS)
+      MSSQL_TDS_PR=$(echo "$PR_BODY" | sed -n 's/.*Depends-On:[[:space:]]*mssql-tds#\([0-9][0-9]*\).*/\1/p' | head -1)
+
+      if [ -n "$MSSQL_TDS_PR" ]; then
+        echo "Found cross-repo dependency: mssql-tds#$MSSQL_TDS_PR"
+        echo "##vso[task.setvariable variable=MSSQL_TDS_PR_NUMBER]$MSSQL_TDS_PR"
+        echo "##vso[task.setvariable variable=HAS_CROSS_REPO_DEPENDENCY]true"
+      else
+        echo "No cross-repo dependency found in PR description"
+        echo "##vso[task.setvariable variable=HAS_CROSS_REPO_DEPENDENCY]false"
+      fi
+    else
+      echo "Not a PR build, skipping cross-repo dependency check"
+      echo "##vso[task.setvariable variable=HAS_CROSS_REPO_DEPENDENCY]false"
+    fi
+  displayName: 'Parse PR for mssql-tds dependency'
+
+# Download artifacts from mssql-tds pipeline if dependency found
+- task: DownloadPipelineArtifact@2
+  condition: eq(variables['HAS_CROSS_REPO_DEPENDENCY'], 'true')
+  displayName: 'Download mssql-py-core artifacts from mssql-tds'
+  inputs:
+    buildType: 'specific'
+    project: 'mssql-rs'
+    definition: '2204'  # mssql-tds build and test pipeline
+    buildVersionToDownload: 'latestFromBranch'
+    branchName: 'refs/pull/$(MSSQL_TDS_PR_NUMBER)/merge'
+    artifactName: 'python-wheels-${{ parameters.osType }}-${{ parameters.architecture }}'
+    targetPath: '$(Pipeline.Workspace)/mssql-py-core-artifacts'
+
+# Extract and place .so files in mssql_python directory
+- bash: |
+    echo "Processing mssql-py-core artifacts..."
+
+    ARTIFACT_DIR="$(Pipeline.Workspace)/mssql-py-core-artifacts"
+    TARGET_DIR="$(Build.SourcesDirectory)/mssql_python"
+
+    if [ -d "$ARTIFACT_DIR" ]; then
+      echo "Artifacts found in $ARTIFACT_DIR"
+      ls -la "$ARTIFACT_DIR"
+
+      # Extract .so files from wheels or copy directly
+      for whl in "$ARTIFACT_DIR"/*.whl; do
+        if [ -f "$whl" ]; then
+          echo "Extracting wheel: $whl"
+          unzip -o "$whl" "*.so" -d "$TARGET_DIR" || true
+          unzip -o "$whl" "*.pyd" -d "$TARGET_DIR" || true
+        fi
+      done
+
+      # Also copy any loose .so files
+      cp -f "$ARTIFACT_DIR"/*.so "$TARGET_DIR"/ 2>/dev/null || true
+      cp -f "$ARTIFACT_DIR"/*.pyd "$TARGET_DIR"/ 2>/dev/null || true
+
+      echo "mssql_python directory contents:"
+      ls -la "$TARGET_DIR"/*.so 2>/dev/null || echo "No .so files found"
+      ls -la "$TARGET_DIR"/*.pyd 2>/dev/null || echo "No .pyd files found"
+    else
+      echo "No artifacts directory found - cross-repo dependency not configured"
+    fi
+  condition: eq(variables['HAS_CROSS_REPO_DEPENDENCY'], 'true')
+  displayName: 'Extract mssql-py-core .so files'