The idea
On a big shared EKS cluster, the platform team wants a say in how every workload runs. Which Karpenter nodepool it lands on, which tolerations it carries, whether it gets a PodDisruptionBudget, how it spreads across zones. Product teams own their own Helm charts and manifests, and asking every team to copy the same block of YAML never works for long.
So the team I worked with built the deployment manifest mutator (DMM). Every deploy sends its rendered manifests through a small Go service. The service asks LaunchDarkly what the platform rules are for this cluster, namespace and app, rewrites the manifests, and the pipeline applies the result.
Why feature flags? A platform engineer can move one namespace to a new nodepool, or roll a topology spread rule out to 10% of apps, from the LaunchDarkly UI. Nobody opens a pull request in every app repo.
How it works
There are two repos. deployment-manifest-packages is a Go library that holds the mutation logic, one package per resource type. deployment-manifest-mutator is the service around it: an HTTP API, a CLI called dmm-client, a Helm chart and the CI workflows.
Developers never call DMM themselves. The shared deploy action that every app pipeline uses runs dmm-client right before kubectl apply, so it was invisible to most teams. The command inside the template looks like this:
dmm-client k8s --cluster <cluster> --namespace <ns> --application <app> kubectl apply -f .cicd/k8s/final/
The API runs as a normal Deployment behind an internal ingress. It never talks to the cluster during a mutation, so it needs no RBAC. Its only secret is the LaunchDarkly server SDK key.
Where LaunchDarkly fits
Both the client and the API connect straight to LaunchDarkly's public cloud with the SDK defaults: stream.launchdarkly.com for live updates, app.launchdarkly.com as the polling fallback and events.launchdarkly.com for analytics. There is no relay proxy, so the pods need outbound HTTPS to *.launchdarkly.com.
The client reads one master switch, ConfigMutator. If it is off, the client copies the manifests to final/ unchanged. The API reads twelve flags, one per mutation, and sends cluster, namespace, resource name, app and tier as attributes, so a rule in LaunchDarkly can target any of them.
| Flag area | What it changes |
|---|---|
| Node selector | Pins pods to a Karpenter nodepool |
| Tolerations | Lets pods run on tainted nodepools |
| Affinity / anti-affinity | Node affinity and spreading replicas across nodes |
| Topology spread | Spreads pods across zones |
| PDB | Adds or adjusts a PodDisruptionBudget |
| CronJob | Scheduling and eviction settings for jobs |
| HPA to KEDA | Turns a queue consumer's HPA into a KEDA ScaledObject |
What I found when I read the code again
Both repos built. Then I ran the library tests, and the deployment and keda packages did not compile. Function signatures had changed and nobody updated the tests, so CI had been checking much less than everyone assumed. A few other problems were worse.
The client always exited 0
Every error in dmm-client was logged and then ignored. If the API was down, or returned a 500, the client still exited successfully, and the next step ran kubectl apply on an empty or half-written final/ folder. A broken mutator could ship a partial deploy and the pipeline would stay green.
A new LaunchDarkly client on every request
Each /mutate call created a fresh LaunchDarkly client, which waits up to 10 seconds to initialize and downloads every flag. The KEDA path added a fixed 3 second sleep and an HTTP call with no timeout. The server's response limit was 15 seconds, so a slow day meant failed deploys.
Percentage rollouts were random
The LaunchDarkly context key was the flag name plus six random characters. Rules that targeted a cluster or namespace attribute still worked. A "10% of apps" rollout picked different apps on every deploy, and LaunchDarkly counted a new context on every evaluation.
One missing flag broke every deploy
If any of the twelve flags did not exist in an environment, the whole request failed. A flag that was off without an explicit off variation was worse: the node selector code wrote the string "null" as the nodepool name, and pods with that selector cannot schedule anywhere.
Why not a mutating webhook?
My first thought for a fresh cluster was a mutating admission webhook. The API server would call DMM for every create or update, so kubectl, Helm and ArgoCD would all get mutated, and there would be no client to forget.
The HPA to KEDA conversion is what stopped me. A webhook can patch the object it receives. It cannot turn a HorizontalPodAutoscaler into a ScaledObject, because that is a different resource. The CI design handles it easily since the client just writes a different file into final/. I kept the original CI-driven shape and wrote down the webhook as a later option for the mutations that only patch.
What I changed
First I moved both repos to private repos under my GitHub account and renamed the Go module path everywhere. The CLI had been pinned to a January 2024 version of the library while the API used a much newer one, so I put both on the same tag. I also removed 25 MB of stale prebuilt binaries that were committed in the client folder.
dmm-clientexits 1 on any failure: flag error, missingAPIvariable, API unreachable or not 200, bad YAML, no manifests, or a write error.- The API creates one LaunchDarkly client at startup and shares it across requests.
- The context key is
cluster/namespace/app, so a percentage rollout keeps the same app in or out on every deploy. - A missing or null flag skips only its own mutation. If the KEDA min replicas flag is not set, the HPA stays as it is.
- The Karpenter nodepool label comes from
NODEPOOL_LABEL_KEY(defaultkarpenter.sh/nodepool) instead of a hard-coded company label. - The API no longer crashes when it cannot load a kubeconfig.
- The Dockerfile moved to Go 1.23, stopped running
go mod tidyduring the build, passes the GitHub token as a BuildKit secret instead of a build arg, and runs as non-root on a minimal image.
I fixed the tests that did not compile. One CronJob test expected output that was older than the current code, so I updated the expectation to match the real behaviour. Both go test ./... and go test -tags unit ./... pass now.
Installing it on a fresh cluster
The original setup depended on things outside the two repos: a shared actions repo, an S3 bucket for the client binary, an ECR registry and a secret created by hand. On a new cluster with no GitHub Actions at all, each of those needs a replacement.
| Before | Fresh install |
|---|---|
| Client binary in S3 | GitHub Release asset |
| API image in ECR | GHCR |
| Company shared deploy action | gh-actions/kubernetes/apply composite action in my org |
| Secret created by a separate process | dmm-secret with LD_SDK_KEY, created once by hand |
| Datadog sidecar, SLO Terraform | Left out of the first version |
In the LaunchDarkly trial I create one project and all thirteen flags, and I give every JSON flag an explicit off variation so the "null" nodepool bug cannot come back through configuration. The API goes in with helm upgrade --install, two replicas and resource limits. The sample app's deploy workflow only calls gh-actions/kubernetes/apply@v1, the same way product teams never saw DMM before.
My acceptance test has three checks:
- With every flag off, the manifests come through unchanged.
- Turning on the tolerations flag adds tolerations on the next deploy.
- Stopping the API makes the deploy job fail.
HPA to KEDA stays out of the first version. It called an internal service to look up RabbitMQ queues, and it needs KEDA and a message broker on the cluster before it means anything.
What I learned
A step that sits between build and kubectl apply has to fail loudly. The exit code bug was one line of behaviour, and it could have shipped a half-empty deploy on any day the API had problems.
If a tool rewrites every deploy, its tests and its exit code matter more than its features.
Feature flags work well for platform rules, as long as the flag context is stable and every flag has a safe off value. Random context keys quietly turned percentage rollouts into a coin flip, and nobody noticed because the targeted rules still worked.
Next steps
- Create the LaunchDarkly trial project and run the mutation end to end with real flags.
- Write the GitHub Actions: API image to GHCR, client to GitHub Releases, and the shared apply action.
- Add an annotation to each mutated object that lists which flags changed it, so "why is my pod on that nodepool" has an answer.
- Bring back HPA to KEDA with a Kubernetes secret instead of the internal lookup service.
