---
name: deploying-new-k8s-service
description: Use when deploying a new service to Zoe's homelab k3s cluster (ansiblestack). Covers scaffolding Helm charts, writing ArgoCD app manifests, wiring ExternalSecrets via OpenBao, configuring Traefik IngressRoutes with cert-manager TLS, and watching GitOps sync to completion.
---

# Deploying a New k3s Service (ansiblestack)

## Overview

All services deploy via GitOps: Helm chart in `ansiblestack` repo → ArgoCD syncs → k3s cluster. Never `kubectl apply` workload manifests directly. Always commit and let ArgoCD drive.

## Cluster Quick Reference

| Thing | Value |
|---|---|
| Cluster | k3s at `10.0.6.10:6443` |
| GitOps repo | `git@git.ctz.fyi:zoe/ansiblestack.git` (GitHub mirror: `ZoesDev/ansiblestack`) |
| ArgoCD | `argocd.ctz.fyi` |
| Secrets | External Secrets Operator → OpenBao (`bao.ctz.fyi`); ClusterSecretStore: `openbao` |
| Ingress | Traefik IngressRoute CRDs |
| TLS | cert-manager, ClusterIssuer: `letsencrypt-production` |
| DNS | external-dns via annotation |
| Registry | Harbor at `registry.ctz.fyi`, project `library` |
| Storage | `ssd` (NFS-SSD, preferred for stateful), `local-path` (node-local) |
| Hostname convention | Public: `<svc>.ctz.fyi` · Internal: `<svc>.i.ctz.fyi` |
| OpenBao KV path | `secret/production/<namespace>/<secret-name>` |

---

## Workflow

### 1. Research the app

Before touching any file:
- Read the upstream GitHub repo or Docker Hub page
- Identify: **ports**, **required env vars**, **config file mounts**, **volume paths**, **default user/UID**
- Wrong env vars = silent failure. Don't skip this.

### 2. Check existing charts for patterns

```
helm/charts/
  jellyfin/    ← stateful reference
  tandoor/     ← stateful with DB reference
  crucix/      ← simple stateless reference
  convertx/    ← simple stateless reference
```

Match the pattern to your app type before scaffolding.

### 3. Scaffold chart files

Path: `helm/charts/<name>/`

```
Chart.yaml
values.yaml
templates/
  _helpers.tpl
  deployment.yaml
  service.yaml
  ingressroute.yaml
  external-secrets.yaml   # only if secrets needed
```

#### Chart.yaml

```yaml
apiVersion: v2
name: <name>
description: <one-liner>
version: 0.1.0
appVersion: "latest"
```

#### values.yaml (minimum)

```yaml
image:
  repository: registry.ctz.fyi/library/<name>  # or upstream image
  tag: latest
  pullPolicy: IfNotPresent

service:
  hostname: <name>.ctz.fyi

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    memory: 512Mi

# persistence:           # include for stateful apps
#   enabled: true
#   storageClass: ssd
#   size: 10Gi
#   mountPath: /data
```

#### templates/_helpers.tpl

```
{{- define "<name>.fullname" -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- end }}
```

#### templates/deployment.yaml

Standard Deployment. Key points:
- `namespace: {{ .Release.Namespace }}`
- Use `{{ include "<name>.fullname" . }}` for all name references
- Mount secrets from ExternalSecret-created Secret if needed
- For stateful: use `PersistentVolumeClaim` via `volumes` + `volumeMounts`, storageClass `ssd`

#### templates/service.yaml

```yaml
apiVersion: v1
kind: Service
metadata:
  name: {{ include "<name>.fullname" . }}
  namespace: {{ .Release.Namespace }}
spec:
  type: ClusterIP
  selector:
    app: {{ include "<name>.fullname" . }}
  ports:
    - port: <port>
      targetPort: <port>
```

#### templates/ingressroute.yaml

**CRITICAL: You need BOTH objects. Do not omit either.**

```yaml
# 1. Traefik IngressRoute — actual routing
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: {{ include "<name>.fullname" . }}
  namespace: {{ .Release.Namespace }}
  annotations:
    external-dns.alpha.kubernetes.io/hostname: {{ .Values.service.hostname }}
spec:
  entryPoints: [websecure]
  routes:
    - match: Host(`{{ .Values.service.hostname }}`)
      kind: Rule
      services:
        - name: {{ include "<name>.fullname" . }}
          port: <port>
  tls:
    secretName: {{ include "<name>.fullname" . }}-tls

---
# 2. Companion Ingress — cert-manager TLS + external-dns ONLY (Traefik ignores this)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ include "<name>.fullname" . }}-cm
  namespace: {{ .Release.Namespace }}
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    external-dns.alpha.kubernetes.io/hostname: {{ .Values.service.hostname }}
    # Add this only for Pangolin/externally-tunneled services:
    # external-dns.alpha.kubernetes.io/target: "external"
spec:
  ingressClassName: traefik
  rules:
    - host: {{ .Values.service.hostname }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: placeholder
                port:
                  number: 80
  tls:
    - hosts: [{{ .Values.service.hostname }}]
      secretName: {{ include "<name>.fullname" . }}-tls
```

#### templates/external-secrets.yaml (only if secrets needed)

```yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: {{ include "<name>.fullname" . }}-secret
  namespace: {{ .Release.Namespace }}
  annotations:
    argocd.argoproj.io/sync-wave: "-1"   # ← REQUIRED — must exist before Deployment
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: openbao
    kind: ClusterSecretStore
  target:
    name: {{ include "<name>.fullname" . }}-secret
    creationPolicy: Owner
  data:
    - secretKey: <key>
      remoteRef:
        key: secret/production/{{ .Release.Namespace }}/{{ include "<name>.fullname" . }}
        property: <key>
```

### 4. Write ArgoCD app manifest

Path: `helm/argocd/<name>-app.yaml`

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: <name>
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "10"
spec:
  project: default
  source:
    repoURL: https://git.ctz.fyi/zoe/ansiblestack
    targetRevision: main
    path: helm/charts/<name>
    helm:
      valueFiles: [values.yaml]
  destination:
    server: https://kubernetes.default.svc
    namespace: <name>
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions: [CreateNamespace=true]
```

### 5. Write secrets to OpenBao (if needed)

```bash
bao kv put secret/production/<namespace>/<name> \
  key1=value1 \
  key2=value2
```

Do this **before** applying the ArgoCD app. ExternalSecret will pull on first sync.

### 6. Commit and push

```bash
cd ansiblestack
git add helm/charts/<name>/ helm/argocd/<name>-app.yaml
git commit -m "feat: add <name> service"
git push
```

### 7. Apply the ArgoCD Application

```bash
kubectl apply -f helm/argocd/<name>-app.yaml
```

ArgoCD picks up the app and begins syncing.

### 8. Verify

```bash
# Watch sync status
kubectl get applications -n argocd <name>

# Check pods
kubectl get pods -n <name>

# Check logs
kubectl logs -n <name> -l app=<name>

# Smoke test
curl -I https://<name>.ctz.fyi
```

Or check the ArgoCD UI at `argocd.ctz.fyi`.

---

## Pangolin (external tunnel) services

Add these to the IngressRoute metadata annotations:
```yaml
annotations:
  pangolin.fossorial.io/enabled: "true"
  pangolin.fossorial.io/target-port: "<port>"
```

And add to the companion Ingress:
```yaml
  external-dns.alpha.kubernetes.io/target: "external"
```

---

## Common Gotchas

| Gotcha | Fix |
|---|---|
| Deployment crashes on startup, missing secret | `sync-wave: "-1"` on ExternalSecret is required — it must exist before Deployment syncs |
| TLS cert never issues | Companion Ingress is missing — cert-manager needs it even though Traefik doesn't route through it |
| Service unreachable despite pod running | Check env vars against upstream docs; wrong vars often cause silent failure at startup |
| PVC stuck in Pending | Use `ssd` storageClass for NFS-backed volumes; `local-path` won't schedule if node is wrong |
| Harbor pull fails | Private Harbor projects need `imagePullSecrets` on the Deployment |
| DNS not registering | Check `external-dns.alpha.kubernetes.io/hostname` annotation is on both IngressRoute and companion Ingress |
| StatefulSet data not persisting | Use `volumeClaimTemplates` in StatefulSet spec, not a standalone PVC manifest |
