Smart-Cash Project - GitOps with FluxCD
A personal project to implement K8 features doing a real hands-on
Published Jan 16, 2024
In a previous article I mentioned the idea behind this project that I named SmartCash. I began building the terraform code for the infrastructure in AWS and the pipeline to deploy it.
In this article, I will introduce FluxCD as a GitOps tool and demonstrate its usage.
A new release has been created in the smart-cash repository for the project. v1.1.0 version will be used, you can check the repository here.
Additionally, a new repository will be created to store the K8 manifest that will be synced with the EKS cluster using FluxCD, you can view the repo here.
GitOps is an operational model for cloud-native architectures, it relies on a Git repository as the single source of truth. New commits imply infrastructure and application updates.
OpenGitOps group has defined 5 principles, and while I won't delve into them, here, you can read more. If you take a look at those principles you will see that they are, in some sense related to some Kubernetes concepts.
A great book to gain a better understanding of GitOps history and concepts is The Path to GitOps.
In summary, GitOps is centered around using a Git repository for defining and managing both infrastructure and application configurations through a Git-based workflow.
FluxCD is an open-source GitOps operator for Kubernetes, you can declaratively define the desired state of your infrastructure and configurations in a Git repository. Flux monitors the repository and applies updates to the Kubernetes cluster when new changes arrive.
Flux started as a monolith but in v2 it was broken up into individual components called GitOps Toolkit, this refers collection of specialized tools, Flux Controllers, composable APIs, and reusable Go packages available under the fluxcd GitHub organization.
Core concepts and toolkit components are described here.
FluxCD installation can be done by Flux CLI, the most straightforward method can be done by the flux bootstrap command, this deploys the Flux controllers on the K8 cluster and configures them to synchronize the cluster to the Git repository, if the Git Repo doesn't exist, the bootstrap command will create it.
To incorporate FluxCD installation into the pipeline created for Terraform, you can add a new job to the workflow template and perform the following steps:
- Validate if flux is installed
- Install FluxCD CLI
- Get credentials for the EKS cluster
- Run flux bootstrap command.
Those steps have been added in a bash script that will be placed in the .github/jobs directory.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#!/bin/bash
aws eks update-kubeconfig --name $CLUSTER_NAME --region $AWS_REGION
flux_installed=$(kubectl api-resources | grep flux)
if [ -z "$flux_installed" ]; then
echo "flux is not installed"
curl -s https://fluxcd.io/install.sh | sudo bash
flux bootstrap github \
--owner=$GH_USER_NAME \
--repository=$FLUX_REPO_NAME \
--path="clusters/$ENVIRONMENT/$CLUSTER_NAME/bootstrap" \
--branch=main \
--personal
else
echo "flux is installed"
fi
The flux bootstrap github command deploys the Flux controllers on the K8 cluster and configures the controllers to synchronize the Git repo with the cluster. This is done by some K8 manifests that are created and pushed to the repo in the path passed in the command.
It's worth noting that some env variables like FLUX_REPO_NAME, and GH_USER_NAME are used by the bash script, these have been added as GH Workflow Input in the template.
The bash script will be executed in the GH workflow template created to deploy the infrastructure, the following job is added to the Workflow template.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
flux-bootstrap:
runs-on: ubuntu-latest
needs: terraform-apply
steps:
- name: checkout-repo
uses: actions/checkout@v4
- name: configure aws credentials
# Removed for brevity
- name: bootstrap flux
id: bootstrap-flux
run: ./bootstrap-flux.sh
working-directory: .github/jobs/
env:
GITHUB_TOKEN: ${{ secrets.GH_TOKEN_FLUX }}
Notice that the GITHUB_TOKEN variable is passed directly in the job, this is to avoid propagating the token in all the jobs, for now, it is just necessary for the Flux bootstrap. PAT is stored as a GitHub Secret in the repo.
Once the workflow is ready you can push it to the repo and see the logs in GH actions, which can look like this:
You can run flux check command locally to validate the status of the installation(you should have access to the cluster in your local env)
If you take a look at the above image you will see that the Source Controller is deployed, Source Controller enables seamless integration of various Git repositories with your Kubernetes cluster. Think of the Source Controller as an interface to connect with GitRepositories, OCIRepository, HelmRepository, and Bucket resources.
The bootstrap command will create a flux source and associate it to the repo passed in the command, to validate this you can list the git sources created and you will see the one, for now.
1
flux get sources git
There are different strategies to structure the GitOps repository, for this scenario, a mono-repo strategy is used and kustomize will be used to manage the K8 manifest for the application.
- ./clusters/smart-cash-develop: Yaml files that create new FluxCD resources like sources, Kustomizations, and Helm releases.
- common: Manifests that create common resources for the clusters.
- app: Manifests that create resources for an application or service.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
├── clusters
| └── smart-cash-develop
| |── flux-system
| |── helm-source.yaml
| |── common-kustomize.yaml
|── common
| └── ingress-namespace.yaml
└── app1
|── base
| |── deployment.yaml
| └── service.yaml
└── overlays.yaml
|── dev
| └── kustomization.yaml
└── prod
└── kustomization.yaml
Let's create a K8 namespace to be used for an nginx-ingress. The manifest for this can be placed in the common folder. A FluxCD Kustomization can be added to synchronize the contents of this folder with the K8 cluster.
The following is the Flux Kustomization that reconciles the Kubernetes manifests located at the path ./common in the Git repository .
- Note: This file can be added in clusters/smart-cash-develop folder, FluxCD will automatically create the Kustomization resource because this path was specified in the bootstrap command, and Flux created a Kustomization to synchronize it.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: smartcash-common
namespace: flux-system
spec:
interval: 5m
targetNamespace: default
sourceRef:
kind: GitRepository
name: flux-system
path: "./kustomize"
prune: true
- interval: The period at which the Kustomization is reconciled.
- sourceRef: refers to the Source object with the required Artifacts, in this case, our GitOps repository.
- prune:: When is true, if previously applied objects are missing from the current revision, these objects are deleted from the cluster
Once you push the Yaml file to the GitOps repo, Flux will create the resources in the cluster. You can validate running:
The previous steps have created the FluxCD Kustomization to sync the common folder with the cluster. Now, a Kustomize file needs to be added to specify which resource to create.
Don't confuse the FluxCD Kustomization file with the K8 configuration management Kustomize. FluxCD will look for the Kustomize file in the common folder.
Let's create and push the following files in the common folder.
1
2
3
4
5
6
7
8
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ns-nginx-ingress.yaml
1
2
3
4
5
6
7
8
apiVersion: v1
kind: Namespace
metadata:
name: nginx-ingress
You can wait for the flux reconciliation or force it using the following command:
1
2
flux reconcile kustomization smartcash-common
If the process was successful you should see the nginx-ingress namespace.
To validate the status of the reconciliation you can use the following command:
1
2
flux get kustomization smartcash-common
For instance, a mistake in the name of the YAML files caused this error, which was visible in the output of the flux command.
If you want more details you can check the K8 CDRs using:
1
kubectl describe kustomization smartcash-common -n flux-system The Flux Helm Controller will be used to install the ingress. The Helm Controller is a Kubernetes operator that enables the management of Helm chart releases.
A FluxCD source for Helm needs to be added. This can be accomplished by using the following manifest, which should be placed in clusters/smart-cash-develop.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: helm-repo-nginx-ingress
namespace: flux-system
spec:
interval: 5m0s
type: oci
url: oci://ghcr.io/nginxinc/charts
This source fetches the Helm OCI repository oci://ghcr.io/nginxinc/charts every 5 minutes, and the artifact is stored and updated each time new updates are done to the repository.
After creating the Helm source, you can proceed to create the Helm release. This release specifies the chart to install in the cluster, with the chart being fetched from the source already created. The following manifest can be used.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: nginx-ingress
namespace: nginx-ingress
spec:
interval: 10m0s
chart:
spec:
chart: nginx-ingress
version: 0.17.1
sourceRef:
kind: HelmRepository
name: helm-repo-nginx-ingress
namespace: flux-system
To delegate the creation of the HelmRelease task to flux, this file can be added to the common folder and in the Kustomize file as well.
1
2
3
4
5
6
7
8
9
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ns-nginx-ingress.yaml
- nginx-ingress-helm.yaml
After updating and pushing the files, you can validate the creation of the Helm Release and nginx-ingress resources.
1
flux get helmreleases -n nginx-ingress
Up to this point, we've covered the second phase of this project. In the upcoming articles, you'll delve into the implementation of various other tools and continue building the project.
If you have any feedback or suggestions, please feel free to reach out to me on LinkedIn.