native service execution (container-free) - working

[ahundt]

Proposal: The Harbor Hybrid Runtime - An Architectural Roadmap

Proposed Harbor CLI update to allow directly installed services to run without a container. This is to solve critical performance bottlenecks (especially for Apple Silicon users) and harbor command line functionality gaps while preserving the existing stability and ease of use of the container-based system as much as possible.

Upate 2025-06-27 native services are now working! (ollama and speaches)

This pull request is not yet working: I'm checking if the conceptual idea is acceptable before I make too many changes.

On macos in particular there is no support for gpu-passthrough of containers, services that need gpus have to run natively to get the best performance (often ~1000x speedup), and existing hacks like setting the uri actually lead a container to still be launched which can lead to conflicts.

Key Advantages:

1. run natively installed services via harbor for key commands and the issues look straightforward to work around
2. gain portability to macos, as setting URIs to a non-container ollama, for example, means you still can't control ollama via harbor
3. harbor might be able to help configure locally installed commands, saving time / effort for those not using containers

1. Core Feature: Native Service Execution

This architecture will empower Harbor to manage services as native host processes, running directly on your machine alongside traditional Docker containers.

• The Goal: To unlock full, bare-metal hardware acceleration. For GPU-intensive services like ollama on Apple Silicon Macs, this means a potential 10x+ performance increase by bypassing Docker's virtualization limitations.
• Proposed Mechanism: A service opts into this system by providing two files:
  1. <service_name>_native.yml: A "contract" file that declares the service's native properties (e.g., the command to run, the port to use).
  2. <service_name>_native.sh: A "bootstrap" script that handles the logic of starting the native application, ensuring a clean and configurable launch.

2. Planned Capability: Intelligent Execution & User Control

The new architecture will feature a smart, multi-layered system for choosing how services run, providing a seamless "it just works" experience with options for expert control.

• The Goal: To deliver maximum performance out-of-the-box without requiring manual configuration, while still allowing full customization.
• Proposed Features:
  • "auto" Mode (Default): Harbor will intelligently default to stable containers but will automatically switch to a native execution for services that require a GPU on platforms with poor Docker support (i.e., Apple Silicon).
  • User Overrides: A user can easily override this default for any service via a simple config command, giving them ultimate control over their environment.

    # Planned Command: Force a service to run as a container, even on a Mac
    harbor config set ollama.execution_preference container

3. Planned Capability: Robust Hybrid Dependency Management

The architecture's cornerstone is a system that makes dependencies between native and containerized services reliable and automatic.

• The Goal: To ensure complex application stacks start up in the correct order, every time, eliminating race conditions.
• Proposed Features:
  • Containers Depending on Native Services: A containerized app (like webui) can depend_on a native service (ollama). The system will guarantee the webui container waits until the native ollama process is fully healthy before starting.
  • Native Services Depending on Containers: A native app can declare a dependency on a containerized database. The system will start the database container and wait for it to be healthy before launching the native application.

4. High-Impact Command Enhancements

The most frequently used commands will be upgraded to be fully "hybrid-aware," creating a unified and intuitive user experience.

• harbor up & harbor down: These orchestrators are the most significant change. They will manage the complete lifecycle of the hybrid stack, correctly starting, stopping, and waiting for both native processes and containers in the proper dependency order.
• harbor ps: The output will be transformed into a unified status dashboard, showing all active services at a glance and clearly labeling them as (native) or (container).
• harbor logs <service>: This will become a universal tool for debugging. It will seamlessly stream logs from a container or tail the log file of a native process with the same, simple command.
• harbor run <service> & harbor shell <service>: These commands will be enhanced to provide the most logical experience based on the service's configuration. A run on a native service will use the local toolchain; a shell on a native service will launch a new host shell, turning potential errors into helpful shortcuts.

5. Expected Limitations and Trade-offs

To set clear expectations, this proposed architecture has the following inherent limitations:

• Host System Dependency: The native execution feature requires the user to have independently installed the native application on their host machine (e.g., via brew install ollama). Harbor will manage the process, but not its installation. The harbor doctor command will be enhanced to check for these dependencies.
• No Resource Isolation for Native Services: Unlike Docker containers, native processes are not resource-constrained. They will consume as much CPU, RAM, or GPU as they need, directly from the host's resources.
• Port Conflicts: Native services will claim ports directly on the host (localhost), which could conflict with other non-Harbor applications the user is running.
• "Ejected" Configurations are Container-Only: The harbor eject command, which produces a portable docker-compose.yml file, will by design only include the definitions for container-based services. The ejected configuration will not be able to manage native processes.

If someone else is interested to help get this running please lmk!

[ahundt]

progress!

native harbor up / down

this code as of 14b9093 can now bring native ollama up and down on macos with:

harbor up -n ollama
harbor down -n ollama

goal

The goal is basically stick as close as possible to how docker and harbor works, while keeping the fact that code is running natively as transparent as possible through the harbor api. That should give a big speedup on a bunch of apps and allow some flexibility when installing and testing!

work to do, any suggestions?

still lots of details to work out though. Right now I'm planning each native service to have a corresponding proxy service (a small lightweight proxy container) to try and get the dependencies filled and bridge between the local host and docker. If there are alternative design suggestions lmk. Here are the current drafts of config/setup files:

<service>_native.yml compose file

I'm currently making a <service>_native.yml compose file such as ollama_native.yml with a different name and using some of the embedded parameters starting with x that docker doesn't use, there is a description here:

#!/usr/bin/env bash
# ollama/ollama_native.sh
#
# Native Service Entrypoint for Ollama in Harbor
#
# This script serves as the entrypoint for running Ollama natively via Harbor's
# hybrid orchestration system. It follows the Docker-style ENTRYPOINT + CMD
# pattern where Harbor passes the complete command to execute.
#
# DESIGN PATTERN:
# This script uses the simple "exec "$@"" pattern, which works well for services
# like Ollama that have straightforward execution requirements. Other services
# may need customized scripts for setup, validation, or special handling.
#
# USAGE PATTERNS:
# Harbor calls this script with the full command to execute:
#
# Daemon Startup:
#   ollama_native.sh "ollama" "serve"
#
# User Commands (via `harbor run ollama`):
#   ollama_native.sh "ollama" "list"
#   ollama_native.sh "ollama" "pull" "llama2"
#
# WHEN TO CUSTOMIZE THIS SCRIPT:
# The simple "exec "$@"" pattern works for Ollama, but other services may need:
# - Pre-execution setup (environment variables, config files)
# - Binary validation and error handling
# - Post-execution cleanup
# - Service-specific argument processing
# - Health checks or initialization steps
#
# Examples of services that might need custom scripts:
# - Services requiring specific environment setup
# - Services with complex initialization sequences
# - Services needing configuration file generation
# - Services with multiple binaries or wrapper scripts
#
# EXECUTABLE NAME SCENARIOS:
# 1. Same as service handle: "ollama" service uses "ollama" executable
# 2. Different name: Service uses different executable name
# 3. Custom path: Service uses executable at specific path
# 4. Complex args: Executable with multiple flags and parameters
#
# ERROR HANDLING:
# This script provides helpful validation and error messages:
# - Validates executable exists (in PATH or as full path)
# - Checks file permissions for full paths
# - Provides clear, actionable error messages
# - Falls back to shell error handling for other issues
# This gives much better user experience than raw "command not found" errors.
#
# LOGGING:
# This script's stdout/stderr are redirected to Harbor's log directory:
# $HARBOR_HOME/app/backend/data/logs/harbor-ollama-native.log

set -euo pipefail # Strict mode for robustness.

# Validate that we have arguments to execute
if [[ $# -eq 0 ]]; then
    echo "ERROR: No command provided to execute." >&2
    echo "Usage: $0 <executable> [args...]" >&2
    exit 1
fi

# Extract the executable name (first argument) for validation
EXECUTABLE="$1"

# Attempt to find the executable in the system's PATH or validate if it's a full path
if [[ "$EXECUTABLE" == /* ]]; then
    # Full path provided - check if file exists and is executable
    if [[ ! -f "$EXECUTABLE" ]]; then
        echo "ERROR: Executable '$EXECUTABLE' not found." >&2
        exit 1
    elif [[ ! -x "$EXECUTABLE" ]]; then
        echo "ERROR: File '$EXECUTABLE' is not executable." >&2
        echo "Try: chmod +x '$EXECUTABLE'" >&2
        exit 1
    fi
else
    # Binary name provided - check if it's in PATH
    if ! command -v "$EXECUTABLE" &>/dev/null; then
        echo "ERROR: Executable '$EXECUTABLE' not found in system's PATH." >&2
        echo "Please ensure $EXECUTABLE is installed and accessible." >&2
        echo "You can check with: command -v $EXECUTABLE" >&2
        exit 1
    fi
fi

# Execute whatever command Harbor passed to us.
# This follows the standard Docker entrypoint pattern of "exec "$@""
# which means "execute all the arguments as a command".
#
# Harbor calls this script like: ollama_native.sh "ollama" "serve"
# So "$@" becomes: "ollama" "serve"
# And this executes: exec ollama serve
exec "$@"

<service>_native.sh init file

Then there is also a <service>_native.sh such as ollama_native.sh to help with launching.

# =============================================================================
# Harbor Native Service Contract: Ollama
# =============================================================================
#
# This file defines the native execution configuration for the Ollama service
# in Harbor. When Harbor is configured to run Olloma natively (instead of in
# a container), this contract specifies how to start the service and integrate
# it with Harbor's hybrid orchestration system.
#
# NATIVE SCRIPT INTEGRATION:
# This YAML file works in conjunction with 'ollama_native.sh', which serves
# as the entrypoint script for this specific service. The script uses the
# Docker-style pattern where Harbor passes the complete command to execute.
#
# SCRIPT DESIGN APPROACH:
# This particular script uses a simple "exec "$@"" pattern that works well
# for Ollama's straightforward execution model. Different services may need
# customized scripts for setup, validation, or special handling requirements.
#
# EXECUTION FLOW:
# 1. Harbor reads this YAML to understand native service configuration
# 2. Harbor calls: ollama_native.sh "ollama" "serve"
# 3. Script executes: exec "$@" → exec ollama serve
# 4. Process runs natively with logs redirected to Harbor's log directory
#
# WHEN SCRIPTS NEED CUSTOMIZATION:
# While this script is simple, other services might require:
# - Environment variable setup before execution
# - Configuration file generation or validation
# - Binary path resolution and error checking
# - Service-specific initialization steps
# - Health checks or readiness verification
#
# REQUIREMENTS:
# - The 'ollama' binary must be installed and available in the system's PATH
# - The native script (ollama_native.sh) must be executable
# - Harbor manages the process lifecycle via PID files and log redirection
#
# TROUBLESHOOTING:
# - Logs are written to: $HARBOR_HOME/app/backend/data/logs/harbor-ollama-native.log
# - PID file location: $HARBOR_HOME/app/backend/data/pids/ollama.pid
# - Check binary availability: command -v ollama
# - Verify script permissions: ls -la ollama_native.sh
#
# =============================================================================
# ========================================================================
# == Harbor Unified Native Contract for 'ollama'
# ========================================================================
#
# This file serves a dual purpose and is the heart of Harbor's hybrid runtime:
#
# 1. A Docker Compose Override File: This file defines a lightweight "proxy"
#    service. When 'ollama' is run in NATIVE mode, Harbor's composer
#    will EXCLUDE the standard 'compose.ollama.yml' (the container definition)
#    and INCLUDE this file instead. This effectively REPLACES the real service
#    with this proxy in the eyes of Docker Compose.
#
# 2. A Native Metadata Contract: The 'x-harbor-native' block contains all
#    the information Harbor needs to start, stop, manage, and configure the
#    actual native 'ollama' process running on the host machine. Docker
#    Compose safely ignores fields starting with 'x-'.
#
services:
  # The top-level service key MUST match the Harbor service handle ('ollama').
  ollama:
    # --------------------------------------------------------------------
    # -- Section 1: Proxy Container Definition
    #    This section defines the lightweight container that will represent
    #    the native 'ollama' service within the Docker network.
    # --------------------------------------------------------------------

    # The Docker image to use for the proxy container. A minimal image with
    # networking tools is ideal. 'alpine/socat' is a common, robust choice.
    image: alpine/socat:latest
    container_name: ${HARBOR_CONTAINER_PREFIX:-harbor}.ollama

    # The command for the proxy container. This uses 'socat' to create a TCP
    # listener on the specified port inside the container and forward all
    # traffic to the native service running on the host. 'host.docker.internal'
    # is the special DNS name that Docker provides for containers to connect
    # back to the host machine.
    command: tcp-listen:11434,fork,reuseaddr tcp-connect:host.docker.internal:11434

    # The healthcheck for the proxy container. This is the KEY to making hybrid
    # dependencies work. This test does NOT check the proxy itself; it checks
    # the readiness of the ACTUAL NATIVE SERVICE on the host. This ensures that
    # any container with `depends_on: ollama` will correctly wait until the
    # native Ollama daemon is fully initialized and ready to accept connections.
    healthcheck:
      test: ["CMD-SHELL", "nc -z host.docker.internal 11434 || exit 1"]
      interval: 2s
      timeout: 5s
      retries: 30
      start_period: 5s

    # Ensures the proxy can be discovered by other services in the stack.
    networks:
      - harbor-network

    # --------------------------------------------------------------------
    # -- Section 2: Native Process Metadata (The Harbor Contract)
    #    This custom block is ignored by Docker Compose but is parsed by
    #    Harbor's scripts to manage the native process lifecycle.
    # --------------------------------------------------------------------
    x-harbor-native:
      # ================================================================
      # == Native Execution Configuration (Docker-Style Pattern)
      # ================================================================
      #
      # Harbor uses a Docker-style ENTRYPOINT + CMD pattern for native services:
      # - 'executable': Like Docker's ENTRYPOINT - the binary to execute
      # - 'daemon_args': Like Docker's CMD - the arguments for daemon startup
      #
      # DESIGN PHILOSOPHY:
      # This pattern separates the "what to run" (executable) from the "how to run it"
      # (arguments), enabling maximum flexibility for different service architectures.
      #
      # EXECUTABLE vs SERVICE NAME VARIATIONS:
      # The 'executable' field can differ from the Harbor service handle:
      #
      # Case 1 - Same name (like ollama):
      #   - Service handle: "ollama"
      #   - Executable: "ollama"
      #   - Daemon startup: ollama serve
      #   - User commands: ollama list, ollama pull model
      #
      # Case 2 - Different executable name:
      #   - Service handle: "textgen"
      #   - Executable: "text-generation-server"
      #   - Daemon startup: text-generation-server --port 8080
      #   - User commands: text-generation-server --help
      #
      # Case 3 - Custom installation path:
      #   - Service handle: "ollama"
      #   - Executable: "/opt/custom/ollama-enterprise"
      #   - Daemon startup: /opt/custom/ollama-enterprise serve
      #   - User commands: /opt/custom/ollama-enterprise list
      #
      # Case 4 - Separate daemon and client executables:
      #   - Service handle: "myservice"
      #   - Executable: "myservice-daemon" (for daemon startup)
      #   - Client tool: "myservice-cli" (separate binary for user commands)
      #   - Note: Harbor currently uses 'executable' for both. For separate
      #     binaries, you'd typically set executable to the daemon binary.
      #
      # EXECUTION FLOW:
      # 1. Daemon startup: Harbor calls: native_script.sh "ollama" "serve"
      # 2. Native script executes: exec "$@" (i.e., exec ollama serve)
      # 3. User commands: Harbor calls: executable + user_args directly
      #    Example: `harbor run ollama list` -> ollama list
      #
      # SCRIPT DESIGN:
      # The native script (ollama_native.sh) uses a simple pattern:
      #   exec "$@"
      # This works well for Ollama but other services may need custom scripts.
      #
      # CONFIGURATION EXAMPLES:
      #
      # 1. SIMPLE DAEMON (like Ollama):
      # executable: "ollama"
      # daemon_args: ["serve"]
      # Results in: ollama serve
      #
      # 2. DAEMON WITH OPTIONS:
      # executable: "ollama"
      # daemon_args: ["serve", "--host", "0.0.0.0", "--port", "11434"]
      # Results in: ollama serve --host 0.0.0.0 --port 11434
      #
      # 3. CUSTOM INSTALLATION PATH:
      # executable: "/usr/local/bin/ollama"
      # daemon_args: ["serve"]
      # Results in: /usr/local/bin/ollama serve
      #
      # 4. DIFFERENT EXECUTABLE NAME:
      # executable: "text-generation-server"
      # daemon_args: ["--port", "8080", "--model-id", "model"]
      # Results in: text-generation-server --port 8080 --model-id model
      #
      # SERVICES THAT NEED CUSTOM SCRIPTS:
      # - Services requiring environment setup (PATH, LD_LIBRARY_PATH, etc.)
      # - Services needing configuration file generation
      # - Services with complex initialization or health checks
      # - Services with multiple binaries or wrapper requirements
      # - Services needing specific user/permission handling
      #
      # ERROR HANDLING APPROACH:
      # This simple script relies on shell and executable error messages.
      # For production deployments, you may want custom error handling,
      # binary validation, or setup verification in the script.

      # The executable binary name or path for this service.
      # Used for both daemon startup and user commands (e.g., `harbor run ollama list`).
      # Can be just the binary name (if in PATH) or a full path.
      # Examples: "ollama", "/usr/local/bin/ollama", "/opt/custom/ollama-v2"
      executable: "ollama"

      # The arguments to pass to the executable when starting the daemon.
      # This replaces the old 'daemon_command' field and supports complex arguments.
      # Examples: ["serve"], ["serve", "--debug"], ["start", "--port", "11434"]
      daemon_args: ["serve"]

      # The TCP port that the native daemon process will listen on. This is used
      # by the proxy's command and healthcheck.
      port: 11434

      # A boolean (true/false) flag that informs Harbor's "auto" execution preference.
      # If true, `auto` mode will prefer to run this service natively on platforms
      # where Docker has poor GPU support (i.e., Apple Silicon) to ensure max performance.
      requires_gpu_passthrough: true

      # A list of environment variables from Harbor's main .env file that should be
      # exported into the environment of the native process before it is started.
      # This allows for consistent configuration of both native and container services.
      env_vars:
        - "OLLAMA_DEBUG"
        - "OLLAMA_KEEP_ALIVE"

      # A map of environment variables that Harbor will inject into OTHER, DEPENDENT
      # containers during `harbor up`. This allows containerized apps (like a web UI)
      # to correctly configure themselves to find and connect to the native service.
      # Example: This sets HARBOR_OLLAMA_INTERNAL_URL for the 'webui' container.
      env_overrides:
        HARBOR_OLLAMA_INTERNAL_URL: "http://host.docker.internal:11434"

That's basically where things are for now, I'd appreciate any feedback/contributions!

[ahundt]

I found some more bugs I plan to continue addressing them.