autojanet/skills/deploying-new-k8s-service/SKILL.md

name	description
deploying-new-k8s-service	Use when deploying a new service to Zoe's homelab k3s cluster (ansiblestack). Covers scaffolding Helm charts, writing ArgoCD app manifests, wiring ExternalSecrets via OpenBao, configuring Traefik IngressRoutes with cert-manager TLS, and watching GitOps sync to completion.

Deploying a New k3s Service (ansiblestack)

Overview

All services deploy via GitOps: Helm chart in ansiblestack repo → ArgoCD syncs → k3s cluster. Never kubectl apply workload manifests directly. Always commit and let ArgoCD drive.

Cluster Quick Reference

Thing	Value
Cluster	k3s at 10.0.6.10:6443
GitOps repo	git@git.ctz.fyi:zoe/ansiblestack.git (GitHub mirror: ZoesDev/ansiblestack)
ArgoCD	argocd.ctz.fyi
Secrets	External Secrets Operator → OpenBao (bao.ctz.fyi); ClusterSecretStore: openbao
Ingress	Traefik IngressRoute CRDs
TLS	cert-manager, ClusterIssuer: letsencrypt-production
DNS	external-dns via annotation
Registry	Harbor at registry.ctz.fyi, project library
Storage	ssd (NFS-SSD, preferred for stateful), local-path (node-local)
Hostname convention	Public: <svc>.ctz.fyi · Internal: <svc>.i.ctz.fyi
OpenBao KV path	secret/production/<namespace>/<secret-name>

---

Workflow

1. Research the app

Before touching any file:

• Read the upstream GitHub repo or Docker Hub page
• Identify: ports, required env vars, config file mounts, volume paths, default user/UID
• Wrong env vars = silent failure. Don't skip this.

2. Check existing charts for patterns

helm/charts/
  jellyfin/    ← stateful reference
  tandoor/     ← stateful with DB reference
  crucix/      ← simple stateless reference
  convertx/    ← simple stateless reference

Match the pattern to your app type before scaffolding.

3. Scaffold chart files

Path: helm/charts/<name>/

Chart.yaml
values.yaml
templates/
  _helpers.tpl
  deployment.yaml
  service.yaml
  ingressroute.yaml
  external-secrets.yaml   # only if secrets needed

Chart.yaml

apiVersion: v2
name: <name>
description: <one-liner>
version: 0.1.0
appVersion: "latest"

values.yaml (minimum)

image:
  repository: registry.ctz.fyi/library/<name>  # or upstream image
  tag: latest
  pullPolicy: IfNotPresent

service:
  hostname: <name>.ctz.fyi

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    memory: 512Mi

# persistence:           # include for stateful apps
#   enabled: true
#   storageClass: ssd
#   size: 10Gi
#   mountPath: /data

templates/_helpers.tpl

{{- define "<name>.fullname" -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- end }}

templates/deployment.yaml

Standard Deployment. Key points:

• namespace: {{ .Release.Namespace }}
• Use {{ include "<name>.fullname" . }} for all name references
• Mount secrets from ExternalSecret-created Secret if needed
• For stateful: use PersistentVolumeClaim via volumes + volumeMounts, storageClass ssd

templates/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: {{ include "<name>.fullname" . }}
  namespace: {{ .Release.Namespace }}
spec:
  type: ClusterIP
  selector:
    app: {{ include "<name>.fullname" . }}
  ports:
    - port: <port>
      targetPort: <port>

templates/ingressroute.yaml

CRITICAL: You need BOTH objects. Do not omit either.

# 1. Traefik IngressRoute — actual routing
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: {{ include "<name>.fullname" . }}
  namespace: {{ .Release.Namespace }}
  annotations:
    external-dns.alpha.kubernetes.io/hostname: {{ .Values.service.hostname }}
spec:
  entryPoints: [websecure]
  routes:
    - match: Host(`{{ .Values.service.hostname }}`)
      kind: Rule
      services:
        - name: {{ include "<name>.fullname" . }}
          port: <port>
  tls:
    secretName: {{ include "<name>.fullname" . }}-tls

---
# 2. Companion Ingress — cert-manager TLS + external-dns ONLY (Traefik ignores this)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ include "<name>.fullname" . }}-cm
  namespace: {{ .Release.Namespace }}
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    external-dns.alpha.kubernetes.io/hostname: {{ .Values.service.hostname }}
    # Add this only for Pangolin/externally-tunneled services:
    # external-dns.alpha.kubernetes.io/target: "external"
spec:
  ingressClassName: traefik
  rules:
    - host: {{ .Values.service.hostname }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: placeholder
                port:
                  number: 80
  tls:
    - hosts: [{{ .Values.service.hostname }}]
      secretName: {{ include "<name>.fullname" . }}-tls

templates/external-secrets.yaml (only if secrets needed)

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: {{ include "<name>.fullname" . }}-secret
  namespace: {{ .Release.Namespace }}
  annotations:
    argocd.argoproj.io/sync-wave: "-1"   # ← REQUIRED — must exist before Deployment
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: openbao
    kind: ClusterSecretStore
  target:
    name: {{ include "<name>.fullname" . }}-secret
    creationPolicy: Owner
  data:
    - secretKey: <key>
      remoteRef:
        key: secret/production/{{ .Release.Namespace }}/{{ include "<name>.fullname" . }}
        property: <key>

4. Write ArgoCD app manifest

Path: helm/argocd/<name>-app.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: <name>
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "10"
spec:
  project: default
  source:
    repoURL: https://git.ctz.fyi/zoe/ansiblestack
    targetRevision: main
    path: helm/charts/<name>
    helm:
      valueFiles: [values.yaml]
  destination:
    server: https://kubernetes.default.svc
    namespace: <name>
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions: [CreateNamespace=true]

5. Write secrets to OpenBao (if needed)

bao kv put secret/production/<namespace>/<name> \
  key1=value1 \
  key2=value2

Do this before applying the ArgoCD app. ExternalSecret will pull on first sync.

6. Commit and push

cd ansiblestack
git add helm/charts/<name>/ helm/argocd/<name>-app.yaml
git commit -m "feat: add <name> service"
git push

7. Apply the ArgoCD Application

kubectl apply -f helm/argocd/<name>-app.yaml

ArgoCD picks up the app and begins syncing.

8. Verify

# Watch sync status
kubectl get applications -n argocd <name>

# Check pods
kubectl get pods -n <name>

# Check logs
kubectl logs -n <name> -l app=<name>

# Smoke test
curl -I https://<name>.ctz.fyi

Or check the ArgoCD UI at argocd.ctz.fyi.

---

Pangolin (external tunnel) services

Add these to the IngressRoute metadata annotations:

annotations:
  pangolin.fossorial.io/enabled: "true"
  pangolin.fossorial.io/target-port: "<port>"

And add to the companion Ingress:

  external-dns.alpha.kubernetes.io/target: "external"

---

Common Gotchas

Gotcha	Fix
Deployment crashes on startup, missing secret	sync-wave: "-1" on ExternalSecret is required — it must exist before Deployment syncs
TLS cert never issues	Companion Ingress is missing — cert-manager needs it even though Traefik doesn't route through it
Service unreachable despite pod running	Check env vars against upstream docs; wrong vars often cause silent failure at startup
PVC stuck in Pending	Use ssd storageClass for NFS-backed volumes; local-path won't schedule if node is wrong
Harbor pull fails	Private Harbor projects need imagePullSecrets on the Deployment
DNS not registering	Check external-dns.alpha.kubernetes.io/hostname annotation is on both IngressRoute and companion Ingress
StatefulSet data not persisting	Use volumeClaimTemplates in StatefulSet spec, not a standalone PVC manifest