kustomization/flux-system/0788b29c reconciliation succeeded
kustomization/infra-notifications/0788b29c dependency not ready
kustomization/infra-configs/0788b29c dependency not ready
kustomization/apps-namespaces/0788b29c dependency not ready
kustomization/apps/0788b29c dependency not ready
kustomization/apps-secrets/0788b29c dependency not ready
kustomization/infra-controllers/0788b29c health check failed
|
15 minutes ago | |
---|---|---|
apps | 15 minutes ago | |
clusters | 1 week ago | |
infrastructure | 19 hours ago | |
.editorconfig | 2 months ago | |
.gitignore | 3 months ago | |
.pre-commit-config.yaml | 2 months ago | |
.sops.yaml | 3 months ago | |
README.md | 2 months ago | |
decrypt.sh | 2 months ago | |
encrypt.sh | 2 months ago |
README.md
Cluster Configs for Midnightthoughts and Nordgedanken
These are the current running setups at Midnightthoughts and Nordegedanken.
This is based on https://github.com/fluxcd/flux2-kustomize-helm-example/tree/d54e250182ead1f4a00e9fd78b05dc9e0186246d
Prerequisites
You will need a Kubernetes cluster version 1.21 or newer. For a quick local test, you can use Kubernetes kind. Any other Kubernetes setup will work as well though.
Install the Flux CLI on MacOS or Linux using Homebrew:
brew install fluxcd/tap/flux
Or install the CLI by downloading precompiled binaries using a Bash script:
curl -s https://fluxcd.io/install.sh | sudo bash
Repository structure
The Git repository contains the following top directories:
- apps dir contains Helm releases with a custom configuration per cluster
- infrastructure dir contains common infra tools such as ingress-nginx and cert-manager
- clusters dir contains the Flux configuration per cluster (Note that staging isn't deployed anywhere at this time)
├── apps
│ ├── base
│ ├── production
│ └── staging
├── infrastructure
│ ├── configs
│ └── controllers
└── clusters
├── production
└── staging
Applications
The apps configuration is structured into:
- apps/base/ dir contains namespaces and Helm release definitions
- apps/production/ dir contains the production Helm release values
- apps/staging/ dir contains the staging values
./apps/
├── base
│ └── podinfo
│ ├── kustomization.yaml
│ ├── namespace.yaml
│ ├── release.yaml
│ └── repository.yaml
├── production
│ ├── kustomization.yaml
│ └── podinfo-patch.yaml
└── staging
├── kustomization.yaml
└── podinfo-patch.yaml
Infrastructure
The infrastructure is structured into:
- infrastructure/controllers/ dir contains namespaces and Helm release definitions for Kubernetes controllers
- infrastructure/configs/ dir contains Kubernetes custom resources such as cert issuers and networks policies
./infrastructure/
├── configs
│ ├── cluster-issuers.yaml
│ ├── network-policies.yaml
│ └── kustomization.yaml
└── controllers
├── cert-manager.yaml
├── ingress-nginx.yaml
├── weave-gitops.yaml
└── kustomization.yaml
In clusters/production/infrastructure.yaml we replace the Let's Encrypt server value to point to the production API:
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: infra-configs
namespace: flux-system
spec:
# ...omitted for brevity
dependsOn:
- name: infra-controllers
patches:
- patch: |
- op: replace
path: /spec/acme/server
value: https://acme-v02.api.letsencrypt.org/directory
target:
kind: ClusterIssuer
name: letsencrypt
Note that with dependsOn
we tell Flux to first install or upgrade the controllers and only then the configs.
This ensures that the Kubernetes CRDs are registered on the cluster, before Flux applies any custom resources.
Useful things
Watch for the Helm releases being installed:
$ watch flux get helmreleases --all-namespaces
NAMESPACE NAME REVISION SUSPENDED READY MESSAGE
flux-system weave-gitops 4.0.12 False True Release reconciliation succeeded
Watch kustomizations getting deployed:
$ flux get kustomizations -w
NAME REVISION SUSPENDED READY MESSAGE
flux-system main@sha1:21ebd912 False True Applied revision: main@sha1:21ebd912
infra-controllers main@sha1:21ebd912 False True Applied revision: main@sha1:21ebd912
TODOs
- Migrate old deployments here
- Gitea
- Woodpecker
- Docker repo
- Traefik
- Certmanager
- External DNS
- Matrix
- Media Repo
- Move DB to DB Cluster
- Synapse
- Prepare DB in DB Cluster
- Move DB to DB Cluster
- Sliding Proxy
- Prepare DB in DB Cluster
- Move DB to DB Cluster
- Mjolnir
- Bridges
- Prepare DBs in DB Cluster
- Move DBs to DB Cluster
- Media Repo
- Keycloak
- Prometheus/grafana
- Cosign
- Mailu (https://just-4.fun/blog/howto/oc-k8s-mailu/ with https://github.com/fastlorenzo/helm-charts-1/tree/master/mailu)
- Imapsync from old server to new server
- ...
- Port validate script
Setup CI for github and woodpecker(Fluxcd can pull it)- Verify sops is working as expected and then publish repo