GitOps is an operating model, not a folder of YAML
A repository full of Kubernetes manifests is useful, but it is not sufficient to establish GitOps. The OpenGitOps project defines four principles for the desired state of a managed system. It must be declarative. It must be versioned and immutable, with complete history. Software agents must pull declarations automatically. Those agents must continuously observe actual state and attempt to reconcile it with desired state.
Each word matters. Declarative configuration says what should exist instead of encoding only a one-time procedure. Version history provides review, authorship, comparison, and a rollback point. Pull-based agents reduce the need for an external deployment job to hold broad inbound cluster credentials. Continuous reconciliation detects a manual edit, deletion, failed rollout, stale source, or changed artifact and reports or corrects the difference according to policy.
GitOps also creates responsibilities that a diagram can hide. Who may change production declarations? Which source and artifact identities are trusted? What fields belong to an autoscaler rather than the GitOps controller? What happens when a removed file causes pruning? Can a tenant create a cluster-wide role? Where do decrypted secrets exist? How is a new image promoted? Which signal proves the controller can still fetch and apply changes? What does Git restore after a database volume is lost?
The five-phase GitOps roadmap develops those responsibilities in order. It begins with OpenGitOps and Kubernetes object management, moves through Kustomize and Helm, then builds one substantial project with Argo CD and one with Flux. The final phase tests supply-chain policy, progressive delivery, observability, rollback, and disaster recovery.
Start with desired state and field ownership
Kubernetes already offers a declarative API. A manifest identifies an object and states selected fields. The API server adds defaults, admission systems may mutate objects, and controllers continuously manage status and other fields. A good GitOps design therefore does not assume the committed YAML will be byte-for-byte identical to the live object. It decides which fields are authoritative, which are defaulted, and which belong to another controller.
Consider a Deployment controlled by a HorizontalPodAutoscaler. If the Git declaration fixes replicas while the HPA changes replicas, two controllers can fight. The safer approach is usually to omit that field from the workload declaration or configure a narrow, documented difference rule. Broad ignore patterns are dangerous because they can hide an attacker or operator changing the image, security context, or service account. Every ignored path should have a known owner and a test.
Self-healing should be similarly deliberate. In Argo CD, an automated sync policy can apply changes, and self-heal can reconcile live drift. Flux Kustomizations repeatedly use server-side apply behavior to detect and correct drift. This is powerful when someone changes an image or deletes a managed Service, but it can surprise an incident responder whose emergency patch disappears. Break-glass procedures should explain how to suspend or bypass reconciliation temporarily, record the action, and create the durable Git change.
Pruning is the other half of desired-state ownership. With pruning enabled, removing a previously managed object from source can remove it from the cluster. That is expected convergence, not a controller bug. Yet pruning a namespace, persistent volume claim, shared custom resource, or CRD may have effects beyond one application. Review needs an inventory-aware diff, ownership information, and a deletion policy. Sensitive data-bearing objects may need explicit protection and a separate retirement workflow.
Design repositories for ownership and promotion
There is no universal rule requiring one repository or many. A monorepository can simplify atomic platform changes and shared review. Separate repositories can create stronger ownership and access boundaries. The decision should consider team authority, environment sensitivity, number of clusters, change frequency, repository scaling, credential scope, and blast radius.
Regardless of layout, make environment desired state explicit. A reusable application base can define the Deployment, Service, probes, labels, and security context. Dev, staging, and production overlays change only what genuinely differs: namespace, resource sizing, route, feature configuration, rollout policy, and image digest. Production paths should have protected ownership. Promotion is then a reviewed commit that moves the same tested artifact digest to the next environment.
Do not rebuild for every environment. If dev uses one image and production rebuilds another from the same source tag, production is no longer promoting tested content. Build once, identify the result by digest, scan and sign it, produce provenance, and update environment declarations to that exact digest. Configuration can differ, but artifact identity remains stable and auditable.
Kustomize works well when teams want plain Kubernetes resources plus overlays and patches. Its base has no knowledge of overlays, so the same base can serve several environments. Keep patches small and targeted. Render each overlay during review because a patch can match the wrong object or combine into invalid output.
Helm works well for versioned reusable packages with templates, default values, dependencies, and optional values schemas. Values precedence and template logic can produce output reviewers will not infer reliably from separate files. Pin the chart version or OCI digest, lint and render the exact environment values, validate the final manifests, and decide how CRDs are installed and upgraded. A successful template render does not prove the workload is safe, authorized, or healthy.
Keep CI strong without making it the cluster deployer
GitOps does not eliminate CI. CI should test application code, build an immutable artifact, scan dependencies and images, produce an SBOM where needed, sign artifacts, generate provenance, and validate proposed desired state. It can render Kustomize and Helm output, validate schemas, run policy tests, detect forbidden namespaces or resource kinds, and show a reviewable diff.
The separation is that ordinary validation does not need a production kubeconfig. The reconciler inside or near the cluster pulls approved desired state. GitHub's secure-use guidance recommends minimum permissions for the workflow token, safe treatment of untrusted pull-request content, secret minimization, and pinning third-party actions to full commit SHAs. Those controls matter because a compromised validation workflow can change the very declarations that the controller trusts.
Artifact verification belongs at multiple boundaries. Sigstore Cosign can verify that an image signature matches the image digest and, in a keyless flow, constrain certificate identity and OIDC issuer. SLSA defines tracks and increasing supply-chain guarantees and recommends attestation formats such as provenance. Neither signature nor provenance is automatically authorization. A policy must decide which source repository, workflow identity, builder, parameters, and subject digest are accepted.
Argo CD: applications, generation, and controlled order
Argo CD describes delivery through Application resources. An Application identifies a source, a destination cluster and namespace, a project, and synchronization behavior. Argo compares rendered desired state with live state and reports sync and health. Automated sync can apply a new revision. Prune removes obsolete managed resources. Self-heal can restore live drift.
AppProjects are a core multi-tenancy boundary. A project can constrain trusted source repositories, allowed destination clusters and namespaces, and resource kinds. Argo CD RBAC controls what users can do through Argo, while Kubernetes RBAC controls what the runtime identity can do at the API server. Strong designs test negative cases: an unauthorized repository, another team's namespace, a ClusterRole, or an unexpected destination should fail visibly.
The app-of-apps pattern uses a root Application to manage child Application manifests. It is convenient for bootstrapping, but the root can create high-impact child applications. Keep it small, platform-owned, protected by code owners, and separate from normal tenant contribution paths.
ApplicationSet addresses repetition. Generators can derive Applications from a cluster inventory, Git directories, lists, or other supported inputs, while a template defines the common Application. This reduces copied manifests, but it also multiplies template impact. Constrain the project, destination, source, and generated fields, and test removal behavior before enabling pruning across a fleet.
Sync phases and waves model ordering. A PreSync hook can run a database migration before the main sync; PostSync can run a verification after resources become healthy. Waves order resources numerically within a phase. Hooks are Kubernetes resources, often Jobs, so they need service accounts, timeouts, failure handling, and cleanup. Migrations must be idempotent or record schema versions because a controller or operator may retry after an uncertain result.
Argo CD applies desired state, but gradual traffic or replica progression is a separate concern. Argo Rollouts adds canary and blue-green strategies, pauses, analysis, and abort behavior. Git selects the immutable version and rollout policy; the Rollouts controller performs the progressive transition. A synthetic project should prove both a successful progression and an analysis failure that returns to the stable version.
Flux: sources and composable reconciliation
Flux is built from specialized controllers and Kubernetes custom resources. A Source such as GitRepository, OCIRepository, HelmRepository, or Bucket describes an origin and how to select content. Source-controller fetches it and produces an addressable artifact with revision or digest status. This separates acquisition failures from application failures.
A Flux Kustomization references a Source artifact and path, builds the content, validates it, applies it, tracks inventory, performs health checks, prunes when configured, and repeats reconciliation at an interval or when a new revision arrives. It is important to distinguish this custom resource from the kustomization.yaml file that Kustomize consumes.
Dependencies form a reconciliation graph. A platform Kustomization can create namespaces and tenant service accounts. A policy Kustomization can install CRDs and controllers. Tenant Kustomizations can depend on both and wait for their health before applying workloads. This is stronger than filename ordering. Circular dependencies never become ready, so model and inspect the graph.
HelmRelease brings Helm lifecycle into reconciliation. It can obtain a chart, combine values, install or upgrade, run Helm tests, report Conditions and history, detect drift, and remediate failure. For a stateless release that tolerates rollback, bounded retries with rollback can restore the last known-good release. A stateful system may need retry-on-failure or a carefully designed forward recovery. Failure injection—not assumptions—should determine the policy.
Flux supports multi-tenancy through Kubernetes RBAC and service-account impersonation. Set serviceAccountName on tenant Kustomizations and HelmReleases and bind that account only to its namespace and required resources. Administrators can enforce a default service account and disable cross-namespace references so an omitted field or shared Source does not silently expand authority. A test manifest that tries to create a ClusterRole or another tenant's Deployment should be denied.
Secrets require encryption and an authority boundary
A Kubernetes Secret with base64 data is not encrypted. Repository privacy does not change that. Git history, clones, caches, pull-request artifacts, logs, and compromised accounts can preserve the value indefinitely. Secret design begins by deciding where plaintext may exist, who can decrypt, which namespaces may receive it, and how it rotates.
Flux Kustomization supports SOPS decryption. A practical workflow encrypts data or stringData fields with age, OpenPGP, or an approved KMS and commits ciphertext plus the required metadata. The private age key or KMS authorization remains outside Git. The reconciler receives narrowly scoped authority and decrypts during reconciliation. Workload identity is preferable to static cloud credentials where supported.
CI should validate encrypted structure and policy without printing plaintext or receiving the production decryption key. Controller logs, Events, debug commands, artifacts, and support bundles must be reviewed for accidental disclosure. Rotation should add the new recipient, re-encrypt, verify reconciliation, and only then remove old authority. A recovery plan must explain how the decryption identity is rehydrated after cluster loss.
Image automation and promotion must preserve review
Flux image automation can scan a registry with ImageRepository, select an allowed version with ImagePolicy, update marked YAML with ImageUpdateAutomation, commit the result, and let normal reconciliation deploy it. This retains Git history instead of mutating only the cluster.
Production still needs policy. Restrict semantic-version ranges, exclude pre-releases, pin or reflect digests, verify artifact identity, and use a narrowly scoped repository writer. ImageUpdateAutomation can push to a different branch so GitHub Actions opens a pull request. This preserves human approval and policy checks before main changes. During an incident, suspend image automation so a new registry event does not race the rollback.
Avoid mutable latest tags. If an operational need requires following a fixed tag, reflect and commit its digest so restarts cannot silently pull different content. The desired state should always identify what is actually intended to run.
Observe the delivery system, not only the workload
A healthy old Deployment can hide a broken delivery system. The Git credential may have expired, the Source may be stale, an overlay path may be invalid, the reconciler may be suspended, admission may reject every apply, or notification delivery may be down. Monitor source and reconciliation health independently from application health.
Useful signals include controller availability; Source Ready status and artifact age; last attempted and applied revision; reconciliation duration and failure count; Kustomization or Application sync and health; HelmRelease action, history, tests, and remediation; drift; suspension; image policy selection; automation commits; and notification success. Alerts should name the cluster, namespace, object kind and name, revision or digest, and safe error category without copying secret data.
Troubleshooting should move through layers. Start with the Application, Kustomization, HelmRelease, or Source Conditions and Events. Confirm the referenced source revision and artifact. Render the exact path and values locally with pinned tools. Check authorization and admission denial. Inspect workload rollout and health. Finally filter controller logs to the object and correlation data. Reinstalling controllers first destroys evidence and rarely fixes an invalid declaration.
Rollback and disaster recovery are different
If an application revision is bad but the cluster is healthy, rollback should update authoritative desired state. Revert the promotion commit or make a reviewed forward commit to a known-good digest. The reconciler then applies the change and records the resulting revision. A live kubectl patch may be needed during a severe incident, but it should be reflected in Git quickly or self-healing may reverse it.
Controller-level remediation can complement Git rollback. Argo Rollouts can abort a canary. Flux HelmRelease can roll back a failed Helm upgrade. These mechanisms recover a release operation, while Git still records the desired version that should persist.
Disaster recovery begins when the cluster or control plane is gone. Git can rebuild declared namespaces, RBAC, controllers, applications, and configuration references. It does not automatically contain database contents, persistent volumes, external DNS state, cloud-managed resources, KMS keys, repository deploy keys, or identity trust. Back up stateful data separately and document recovery order.
A credible exercise creates a new disposable cluster, installs pinned GitOps components, re-establishes minimum repository and registry access, rehydrates decryption authority outside Git, reconciles platform prerequisites, restores data, then reconciles applications. Measure the time to reach the expected source revision, artifact digest, and health. Test this process before an incident.
Two projects that demonstrate practical skill
Argo CD multi-environment platform
The first GitOps project uses three disposable clusters. A protected repository contains platform bootstrap, AppProjects, a small app-of-apps root, an ApplicationSet, a reusable Kustomize base, environment overlays, policies, a synthetic migration, and an Argo Rollout. Pull-request CI renders everything, rejects cross-namespace or cluster-scoped tenant objects, requires immutable image identity, and verifies synthetic signature or provenance policy without holding a cluster credential.
The project promotes one digest through dev, staging, and production. A failing migration stops sync. A bad canary metric aborts progression. A manual image patch self-heals. A Git revert restores the known-good digest. Finally, the learner deletes and rebuilds one cluster, restores synthetic data and secret references, records recovery time, measures cost surfaces, and removes the full lab.
Flux multi-tenant cluster fleet
The second project bootstraps pinned Flux controllers across three disposable clusters. Sources produce artifacts. A dependency graph applies namespaces, RBAC, policy, and controllers before tenants. Each tenant Kustomization and HelmRelease impersonates a namespace-scoped service account. Cross-tenant and cluster-scope attempts fail.
A SOPS-style workflow commits only synthetic ciphertext and keeps the age or KMS authority outside Git. ImageRepository and ImagePolicy select allowed signed releases. ImageUpdateAutomation writes to a review branch. A canary controller provides gradual rollout. Conditions, Events, metrics, and notification-controller alerts expose source, build, authorization, decryption, Helm, health, image, and rollout failures. The project ends with failure injection, complete cluster restore, cost notes, and verified key, credential, cluster, image, artifact, and telemetry cleanup.
An eight-week implementation plan
- Week 1: Study OpenGitOps principles and Kubernetes declarative object management. Build a small desired state, preview changes, create drift, and document field ownership.
- Week 2: Create Kustomize bases and overlays, package or consume a pinned Helm chart, render all environments, and add schema and policy validation in secure CI.
- Week 3: Install Argo CD in disposable clusters. Define Applications, AppProjects, RBAC, automated sync, prune, self-heal, health, and notifications.
- Week 4: Add the root bootstrap app, ApplicationSet, sync waves, replay-safe migration, immutable promotion, Argo Rollouts, and failure tests.
- Week 5: Bootstrap Flux. Build Sources and dependent Kustomizations for platform and tenants, then enforce service-account impersonation and denied cross-tenant tests.
- Week 6: Add HelmRelease remediation, SOPS-style decryption and rotation, image scanning, bounded policy, digest updates, and a production review branch.
- Week 7: Add admission and supply-chain verification, canary delivery, revision-aware metrics, Events, alerts, and runbooks. Inject source, render, authorization, secret, health, and controller failures.
- Week 8: Complete 25 original knowledge checks, perform Git-native rollback, destroy and restore a cluster, document costs and limitations, and verify cleanup.
Common GitOps mistakes
- Calling a Git repository GitOps. Prove automated pull and continuous reconciliation.
- Giving CI cluster-admin. Keep validation and artifact creation separate from in-cluster reconciliation.
- Using mutable tags. Promote immutable digests and verify trusted artifact identity.
- Ignoring broad drift paths. Assign each ignored field a legitimate controller and narrow selector.
- Enabling prune without inventory review. Test deletion behavior for storage, namespaces, CRDs, and shared resources.
- Sharing a controller's cluster-admin authority with tenants. Use project constraints and service-account impersonation.
- Committing base64 Secrets. Base64 is not encryption; keep decryption authority outside Git.
- Ordering by filenames or sleeps. Use sync waves, hooks, dependencies, and health checks.
- Retrying migrations blindly. Make hooks idempotent or schema-version aware.
- Watching only Pods. Alert on stale Sources, failed reconciliation, suspension, drift, and notification health.
- Rolling back only in the cluster. Restore authoritative Git desired state.
- Assuming Git is a database backup. Protect stateful data, external services, identities, and keys separately.
Present practical evidence honestly
A useful portfolio does not need a production cloud estate. Use disposable clusters and synthetic workloads. Publish a sanitized architecture, repository tree, RBAC matrix, rendering and policy report, Application or Flux reconciliation graph, sample Conditions and Events, canary result, failure-injection table, rollback commit, restore timeline, cost model, and cleanup checklist.
State the limits. A local fleet does not prove experience with thousands of clusters. A synthetic canary does not prove production SLO design. An encrypted test Secret does not prove enterprise key governance. It does demonstrate that the learner can reason about authority, desired state, retries, deletion, provenance, telemetry, failure, and recovery in a reproducible environment.
Roles that use these skills include platform engineer, Kubernetes engineer, DevOps engineer, cloud infrastructure engineer, SRE, release engineer, and software supply-chain engineer. Explore related cloud and platform roles, but do not treat completion as a job guarantee. Hiring depends on broader experience, communication, systems knowledge, and market conditions.
Official sources
- OpenGitOps principles
- CNCF Argo project page
- CNCF Flux project page
- Argo CD documentation
- Argo CD automated sync policy
- Argo CD sync phases and waves
- Argo CD ApplicationSet
- Argo Rollouts documentation
- Flux documentation
- Flux Source controllers
- Flux Kustomization API
- Flux HelmRelease API
- Flux SOPS guide
- Flux image automation guide
- Kubernetes declarative object management
- Kubernetes Kustomize documentation
- Helm charts documentation
- GitHub Actions secure use
- Sigstore Cosign verification
- SLSA specification v1.2
Continue the practical path
- GitOps with Argo CD and Flux five-phase roadmap
- GitOps original knowledge checks
- GitOps flashcards
- GitOps hands-on projects
- Cloud and platform engineering jobs
- Infrastructure as Code for Beginners
- DevOps Engineer Tech Stack in 2026
- Cloud Lab Cost Control
- PrepKloud editorial policy
Frequently asked questions
Is GitOps with Argo CD and Flux a certification?
No. This is an independent practical skill path with original knowledge checks and projects. It is not an exam, credential, passing-score program, or guarantee.
What is the difference between CI and GitOps continuous delivery?
CI builds, tests, scans, signs, and publishes artifacts and validates proposed configuration. A GitOps reconciler pulls approved desired state and continuously applies it. Keeping those roles separate reduces the need for production cluster credentials in ordinary CI.
How do Argo CD and Flux differ?
Argo CD centers delivery around Application resources and provides a strong application operations experience. Flux uses composable Kubernetes custom resources and specialized controllers for sources, Kustomize, Helm, notifications, and image automation. Both can implement pull reconciliation; choose through requirements and operate both in disposable labs before standardizing.
How should secrets be managed in GitOps?
Never rely on base64 or repository privacy. Commit only approved ciphertext or nonsecret references, keep decryption authority outside Git, scope it narrowly, prevent plaintext from reaching CI and logs, and test rotation, revocation, and recovery.
What proves practical GitOps skill?
Strong evidence includes deterministic rendering, constrained reconciliation identities, denied cross-tenant access, immutable promotion, failure injection, revision-aware telemetry, Git-native rollback, cluster restore, cost notes, and verified cleanup.