Multi-Cluster Argo CD: One Control Plane, Many Destinations, Fewer Foot-Guns

One Argo CD talking to twenty clusters is appealing: one UI, one SSO, one set of AppProjects. It is also how a mis-labeled destination.name ships staging manifests to prod.

This is the topology and guardrail set that keeps the hub model honest.

Two topologies

Hub-and-spoke. Argo CD (HA) in a management cluster. Other clusters are registered destinations. Applications live in the hub and set spec.destination.

Spoke-local. Each cluster runs its own Argo CD. Git is the only shared control plane.

Use hub-and-spoke when you want centralized RBAC and one place to see OutOfSync. Use spoke-local when clusters are in isolated networks, or when a compromised hub must not have kubeconfig to prod.

A hybrid is common: hub for non-prod, local Argo CD for prod (or the reverse, if prod is the only place you allow kubeconfig).

Registering a cluster

argocd cluster add prod-eks \
  --kubeconfig ./prod.kubeconfig \
  --name prod \
  --label env=prod

This stores a Secret in argocd with the remote credentials. Treat that Secret like prod admin. Rotate the remote ServiceAccount token; do not use a human cloud-admin kubeconfig.

The remote RBAC for Argo CD’s manager SA should be namespace-scoped where you can. Cluster-admin on every spoke is convenient and indefensible.

Destination is a hard-coded contract

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-prod
  namespace: argocd
spec:
  project: prod
  source:
    repoURL: https://github.com/acme/apps.git
    path: api/overlays/prod
    targetRevision: main
  destination:
    name: prod          # cluster name, not an in-cluster default
    namespace: api
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Never leave destination.server: https://kubernetes.default.svc on an app that was copied from a local-cluster example. That syncs the hub cluster.

AppProject must allow-list destinations:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: prod
  namespace: argocd
spec:
  destinations:
    - name: prod
      namespace: "api"
    - name: prod
      namespace: "api-*"
  sourceRepos:
    - "https://github.com/acme/apps.git"
  clusterResourceWhitelist:
    - group: ""
      kind: Namespace

A * destination on the prod project is how staging YAML lands on prod. Split projects: nonprod may list dev and staging. prod lists only prod.

ApplicationSet for the matrix

Twenty apps × five clusters should not be twenty hand-copied Applications.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: api
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster: dev
            path: api/overlays/dev
          - cluster: staging
            path: api/overlays/staging
          - cluster: prod
            path: api/overlays/prod
  template:
    metadata:
      name: "api-{{cluster}}"
    spec:
      project: "{{cluster}}"   # project per env, not one project
      source:
        repoURL: https://github.com/acme/apps.git
        path: "{{path}}"
        targetRevision: main
      destination:
        name: "{{cluster}}"
        namespace: api

Keep project: "{{cluster}}" so the prod element cannot use the nonprod project even if someone edits the path.

Sync waves do not cross clusters

Ordering “CRDs on all clusters, then apps” is not a wave on one Application. Each destination is a separate sync. If you need “cert-manager Healthy on prod before api-prod syncs,” use:

• two Applications and waves on the parent app-of-apps in the hub, or
• a promotion pipeline that writes the prod path after a hook.

Do not assume ApplicationSet list order is a sync order.

Secrets and the hub

Cluster credentials, repo deploy keys, and SSO client secrets live on the hub. Back them with External Secrets or sealed machinery that is not the same Git repo as app manifests.

A repo that contains both apps/ and argocd-values-with-oauth-secret.yaml is one leak away from every cluster.

Common pitfalls

1. One AppProject with destinations: [{name: '*', namespace: '*'}].
2. Copy-paste Application with in-cluster destination.
3. Hub Argo CD on Spot — see Spot node pools. The control plane should be boring on-demand.
4. Automated sync + prune on prod from main with no folder protection. Use targetRevision pins or a prod branch that only updates via promotion PR.
5. Same Git path for all clusters. Overlay per env, or you will “fix staging” in prod.

Decision checklist

• Prod project cannot target non-prod clusters (and the reverse)
• Cluster Secrets use least-privilege remote SAs
• ApplicationSet elements set both path and project per env
• Hub is HA, on-demand, backed up (argocd-cm, projects, cluster secrets)
• Prod auto-sync is a conscious choice, not the template default