For a detailed history of the project's evolution, see CHANGELOG.md.

• HomeLab Infrastructure
  • Overview
  • 🏗️ Architecture Overview
  • 📂 Repository Structure
  • 🚀 Components
    • 1. Infrastructure & Provisioning (/ansible, /machineConfigs)
  • 🖥️ Hardware Inventory
    • Kubernetes
      • Machines
        • Storage
      • Kubernetes Nodes
        • Ceph Performance Tests
        • GPU
  • 🌐 Network Architecture
    • Physical Topology
    • Networking Machines
    • VLAN Map
    • 2. Kubernetes Cluster (/okd)
    • 3. GitOps & Applications (/kubernetes)
    • 4. Tooling (/containers, main.bash)
    • 5. Virtualization & Cloud (/vms, /terraform)
  • 🛠️ Usage
    • Bootstrapping
    • Deploying a New App
      • Kubernetes Commands
        • Delete Pod Using Graceful Termination Eviction Request

This repository contains the complete Infrastructure-as-Code (IaC) and GitOps configuration for my HomeLab. It manages everything from bare-metal server provisioning and virtualization to the Kubernetes (OKD) cluster and application deployment.

The infrastructure is built in layers, starting from physical hardware management with Ansible, up to application deployment managed by ArgoCD.

graph TD
    User[User / Admin] -->|manages| MainScript[main.bash]

    subgraph "Provisioning Layer"
        MainScript -->|triggers| Ansible[Ansible Playbooks]
        Ansible -->|configures| BareMetal[Physical Servers]
        Ansible -->|configures| KVM[KVM Hypervisors]
        Ansible -->|configures| Desktops[Workstations]
    end

    subgraph "Cluster Layer (OKD)"
        AgentConfig[Agent Config] -->|bootstraps| OKD[OKD Cluster]
        InstallConfig[Install Config] -->|defines| OKD
        BareMetal -->|hosts| OKD
    end

    subgraph "GitOps Layer"
        OKD -->|runs| ArgoCD[ArgoCD]
        ArgoCD -->|syncs| Apps[Applications]

        Apps -->|source| K8sDir[./kubernetes/]
    end

    subgraph "Applications"
        K8sDir --> Monitoring[Observability]
        K8sDir --> Storage[Storage]
        K8sDir --> HomeAuto[Home Automation]
        K8sDir --> DevTools[DevOps & Security]
        K8sDir --> Media[Media]
    end

├── ansible/            # Ansible playbooks for node configuration (KVM, Servers, Desktops)
│   ├── playbooks/      # Reusable Ansible playbooks
│   └── inventory       # Host inventory
├── containers/         # Custom container images (Toolbox, Utilities)
├── kubernetes/         # Kubernetes manifests for all applications (GitOps source)
│   ├── argocd/         # ArgoCD bootstrap configuration
│   ├── ceph/           # Rook-Ceph storage configuration
│   ├── ...             # Individual application folders (Traefik, Vault, etc.)
├── machineConfigs/     # Ignition/Preseed configs for machine provisioning
├── okd/                # OpenShift/OKD specific installation configs
│   ├── agent-config.yaml   # Node definitions for Agent Installer
│   └── install-config.yaml # Cluster configuration
├── tekton/             # Tekton Pipelines and Tasks
│   ├── base/           # Base Tekton configurations
│   ├── overlays/       # Environment-specific overlays
│   └── tasks/          # Reusable Tekton Tasks
├── terraform/          # Terraform configurations for infrastructure
│   ├── agent/          # Infrastructure for the Agent Based Terraform KVM
│   ├── homelab/        # Main HomeLab infrastructure
│   └── sandbox/        # Infrastructure for the UPI Based Terraform KVM
├── vms/                # Virtual Machine definitions (KubeVirt)
├── main.bash           # Central entrypoint script for management tasks
└── notes/              # Documentation, scratchpads, and manual scripts

• Ansible: Used to configure the base operating system (Fedora/CentOS Stream), set up KVM, and manage networking (NMState).
• Networking:
  • Subnets: Machine Network (10.101.10.0/24), Cluster Network (10.101.32.0/19).
  • Bonding: Nodes utilize bonded interfaces for redundancy and performance.
    • active-backup: Used for split-switch topologies or where LACP is unavailable.
    • 802.3ad (LACP): Used for bandwidth aggregation on supported switches.
  • MTU: Jumbo frames (9000) are enabled for storage and cluster traffic.
• Virtualization: KVM is used to host virtualized control plane or worker nodes where applicable.

https://www.okd.io/

CPU Benchmark

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1080        Off |   00000000:2B:00.0 Off |                  N/A |
| 27%   32C    P8             13W /  180W |       0MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

• Node: gpu-1
• Card: NVIDIA GeForce GTX 1080 (8GB)
• Driver: 580.95.05 (CUDA 13.0)
• Workload: Transcoding (Plex/Immich), AI/ML experiments.

The network is designed for high throughput (10GbE core) and segmentation via VLANs.

• Router: Ubiquiti UDM-SE (Gateway 10.0.0.1)
  • WAN 1: 1Gbps Fiber
  • WAN 2: LTE Backup
• Core Switching:
  • USW-Pro-HD-24: 10GbE/2.5GbE Core Switch.
  • USW-Aggregation (x2): 10GbE Fiber/DAC aggregation layer.
  • USW-Pro-Max-16: 2.5GbE/1GbE distribution layer.
• Wireless: U7 Pro Max & U7 Pro Wall (WiFi 7).

graph TD
 linkStyle default interpolate basis

subgraph Networking
    wan1[<center>WAN<br>192.168.100.1</center>]---|1000/50 Mb|router{<center>UDM Pro SE<br>10.0.0.1</center>}
    wan2[<center>LTE<br>192.168.1.1</center>]---|750/25 Mb|router

    router---|10GbE|switch_hd[<center>USW Pro HD 24</center>]

    switch_hd---|10GbE|switch_agg_bot[<center>Agg Switch Bottom</center>]
    switch_hd---|10GbE|switch_max[<center>USW Pro Max 16</center>]

    switch_agg_bot---|10GbE|switch_agg_top[<center>Agg Switch Top</center>]
    switch_agg_top---|10GbE|switch_max

    router---|2.5GbE|ap1{<center>U7 Pro Wall<br></center>}
    switch_hd---|2.5GbE|ap2{<center>U7 Pro Max<br></center>}
end

subgraph Storage
    switch_hd---|10GbE|unas(<center>UNAS<br>10.101.2.6</center>)
    switch_hd---|2.5GbE|truenas(<center>TrueNas<br>10.101.1.6</center>)
    router---|1GbE|truenas
end

subgraph "OpenShift/OKD"
    %% Control Plane
    switch_agg_bot---|2x10GbE|server1(<center>server-1<br>10.101.10.101</center>)
    switch_agg_top---|2x10GbE|server1
    switch_max-.-|4x 1GbE|server1

    switch_agg_bot---|2x10GbE|server2(<center>server-2<br>10.101.10.102</center>)
    switch_agg_top---|2x10GbE|server2
    switch_max-.-|4x 1GbE|server2

    switch_agg_bot---|2x10GbE|server3(<center>server-3<br>10.101.10.103</center>)
    switch_agg_top---|2x10GbE|server3
    switch_max-.-|4x 1GbE|server3

    %% Workers & GPU
    switch_hd---|2.5GbE|worker1(<center>worker-1<br>10.101.10.104</center>)
    switch_max---|2.5GbE|worker1

    switch_hd---|2.5GbE|worker2(<center>worker-2<br>10.101.10.105</center>)
    switch_max---|2.5GbE|worker2

    switch_hd---|2.5GbE|worker3(<center>worker-3<br>10.101.10.106</center>)
    switch_max---|2.5GbE|worker3

    switch_hd---|2.5GbE|gpu1(<center>gpu-1<br>10.101.10.107</center>)
    switch_max---|2.5GbE|gpu1
end

subgraph Edge
    router---|1GbE|microshift[<center>MicroShift / PiHole<br>10.101.1.7</center>]
end

• Distribution: OKD (OpenShift Kubernetes Distribution).
• Installation: Uses the Agent-based Installer for bare-metal deployment.
• Configuration:
  • agent-config.yaml: Defines the physical hosts, network interfaces, and static IP assignments.
  • install-config.yaml: Defines the cluster name, base domain, and networking CIDRs.

• Controller: ArgoCD watches this repository and syncs the state of the cluster.
• Storage Strategy:
  • Rook-Ceph: Block, Object, and File storage for the cluster.
  • MinIO: S3-compatible object storage for applications.
  • Longhorn: Distributed block storage (alternative/backup).
• Observability Stack:
  • Prometheus & Grafana: Metrics collection and visualization.
  • Loki: Log aggregation.
  • StackRox (ACS): Kubernetes security and compliance.
  • Network Observability: Flow logs and traffic analysis.
• Security:
  • Vault: Secret management.
  • Cert-Manager: Automated certificate management.
  • External Secrets: Syncing secrets from external providers.
• Home Automation:
  • Home Assistant: Central automation hub.
  • ESPHome: Microcontroller management.
• DevOps:
  • Gitea: Self-hosted Git service.
  • Tekton: CI/CD pipelines.
  • Woodpecker: CI/CD automation.

• Toolbox: A custom container image (containers/toolbox) pre-loaded with CLI tools (oc, kubectl, ansible, tofu) for managing the environment.
• main.bash: A wrapper script to execute common tasks like running Ansible playbooks, fixing Kustomize files, or managing VMs.
  • ansible: Runs the Ansible playbooks for server configuration.
  • kustomize_fix: Fixes Kustomize files by running kustomize edit fix.
  • test_overlays: Validates Kubernetes manifests using kubeconform and argocd-vault-plugin.
  • stateful_workload_stop: Suspends CronJobs and scales down monitoring/stateful workloads (useful for maintenance).

• KubeVirt (/vms): Defines Virtual Machines (e.g., Fedora) running on top of the OKD cluster, managed via GitOps.
• Terraform (/terraform): Manages external cloud resources (GCP) and integrates with Vault for secret management.

The environment is typically bootstrapped using the main.bash script which orchestrates the Ansible runs and prepares the OKD installer.

./main.bash

1. Create a new folder in kubernetes/<app-name>.
2. Add Kubernetes manifests or a Kustomization file.
3. Register the application in ArgoCD (usually via kubernetes/argocd/applications).
4. Commit and push. ArgoCD will sync the changes automatically.

# Watch ALl Pods
watch kubectl get pods -A -o wide --sort-by=.metadata.creationTimestamp
# Watch non Health Pods
watch -n10 $(echo "kubectl get pods  -A | grep -vE 'Running|Completed'")
# Delete Pods that Have a Restart
kubectl get pods -A | awk '$5>0' | awk '{print "kubectl delete pod -n " $1 " " $2}' | bash -
# Drain Node
 oc adm drain server-3 --delete-emptydir-data --ignore-daemonsets --force
# Nextcloud
kubectl exec -it nextcloud-0 -n nextcloud -- runuser -u www-data -- php -f /var/www/html/occ

NAMESPACE=homelab
POD=el-webhook-6b56cc5f84-clfc6

curl --header "Authorization: Bearer $(oc whoami -t)" -H 'Content-type: application/json' \
"$(oc whoami --show-server)/api/v1/namespaces/{$NAMESPACE}/pods/{$POD}/eviction" \
-d '{"apiVersion": "policy/v1","kind": "Eviction","metadata": {"name": "'"${POD}"'","namespace": "'"${NAMESPACE}"'"}}'

• https://docs.okd.io/latest/rest_api/policy_apis/eviction-policy-v1.html#eviction-policy-v1
• https://unofficial-kubernetes.readthedocs.io/en/latest/tasks/configure-pod-container/configure-pod-disruption-budget/