Bazel GitOps Rules

Bazel GitOps Rules provides tooling to bridge the gap between Bazel (for hermetic, reproducible, container builds) and continuous, git-operation driven, deployments. Users author standard kubernetes manifests and kustomize overlays for their services. Bazel GitOps Rules handles image push and substitution, applies necessary kustomizations, and handles content addressed substitutions of all object references (configmaps, secrets, etc). Bazel targets are exposed for applying the rendered manifest directly to a Kubernetes cluster, or into version control facilitating deployment via Git operations.

Bazel GitOps Rules is an alternative to rules_k8s. The main differences are:

• Utilizes and integrates the full set of Kustomize capabilities for generating manifests.
• Implements GitOps target.
• Supports personal namespace deployments.
• Provides integration test setup utility.
• Speeds up deployments iterations:
  • The results manifests are rendered without pushing containers.
  • Pushes all the images in parallel.
• Provides an utility that creates GitOps pull requests.

Rules

• k8s_deploy
• k8s_test_setup

Guides

• Base Manifests and Overlays
• Generating Configmaps
• Injecting Docker Images
• Adding Dependencies
• GitOps and Deployment
• Integration Testing Support

Installation

From the release you wish to use: https://github.com/adobe/rules_gitops/releases copy the WORKSPACE snippet into your WORKSPACE file.

k8s_deploy

The k8s_deploy creates rules that produce the .apply and .gitops targets k8s_deploy is defined in k8s.bzl. k8s_deploy takes the files listed in the manifests, patches, and configmaps_srcs attributes and combines (renders) them into one YAML file. This happens when you bazel build or bazel run a target created by the k8s_deploy. The file is created at bazel-bin/path/to/package/name.yaml. When you run a .apply target, it runs kubectl apply on this file. When you run a .gitops target, it copies this file to the appropriate location in the same os separate repository.

For example, let's look at the example's k8s_deploy. We can peek at the file containing the rendered K8s manifests:

cd examples
bazel run //helloworld:mynamespace.show

When you run bazel run ///helloworld:mynamespace.apply, it applies this file into your personal ({BUILD_USER}) namespace. Viewing the rendered files with .show can be useful for debugging issues with invalid or misconfigured manifests.

Base Manifests and Overlays

The manifests listed in the manifests attribute are the base manifests used by the deployment. This is where the important manifests like Deployments, Services, etc. are listed.

The base manifests will be modified by most of the other k8s_deploy attributes like substitutions and images. Additionally, they can be modified to configure them different clusters/namespaces/etc. using overlays.

To demonstrate, let's go over hypothetical multi cluster deployment.

Here is the fragment of the k8s_deploy rule that is responsible for generating manifest variants per CLOUD, CLUSTER, and NAMESPACE :

k8s_deploy(
    ...
    manifests = glob([                 # (1)
      "manifests/*.yaml",
      "manifests/%s/*.yaml" % (CLOUD),
    ]),
    patches = glob([                   # (2)
      "overlays/*.yaml",
      "overlays/%s/*.yaml" % (CLOUD),
      "overlays/%s/%s/*.yaml" % (CLOUD, NAMESPACE),
      "overlays/%s/%s/%s/*.yaml" % (CLOUD, NAMESPACE, CLUSTER),
    ]),
    ...
)

The manifests list (1) combines common base manifests and CLOUD specific manifests.

manifests
├── aws
│   └── pvc.yaml
├── onprem
│   ├── pv.yaml
│   └── pvc.yaml
├── deployment.yaml
├── ingress.yaml
└── service.yaml

Here we see that aws and onprem clouds have different persistence configurations aws/pvc.yaml and onprem/pvc.yaml.

The patches list (2) requires more granular configuration that introduces 3 levels of customization: CLOUD, NAMESPACE, and CLUSTER. Each manifest fragment in the overlays subtree applied as strategic merge patch update operation.

overlays
├── aws
│   ├── deployment.yaml
│   ├── prod
│   │   ├── deployment.yaml
│   │   └── us-east-1
│   │       └── deployment.yaml
│   └── uat
│       └── deployment.yaml
└── onprem
    ├── prod
    │   ├── deployment.yaml
    │   └── us-east
    │       └── deployment.yaml
    └── uat
        └── deployment.yaml

That looks like a lot. But lets try to decode what is happening here:

1. aws/deployment.yaml adds persistent volume reference specific to all AWS deployments.
2. aws/prod/deployment.yaml modifies main container CPU and memory requirements in production configurations.
3. aws/prod/us-east-1/deployment.yaml adds monitoring sidecar.

Generating Configmaps

Configmaps are a special case of manifests. They can be rendered from a collection of files of any kind (.yaml, .properties, .xml, .sh, whatever). Let's use hypothetical Grafana deployment as an example:

[
    k8s_deploy(
        name = NAME,
        cluster = CLUSTER,
        configmaps_srcs = glob([                 # (1)
            "configmaps/%s/**/*" % CLUSTER
        ]),
        configmaps_renaming = 'hash',            # (2)

        ...
    )
    for NAME, CLUSTER, NAMESPACE in [
        ("mynamespace", "dev", "{BUILD_USER}"),  # (3)
        ("prod-grafana", "prod", "prod"),        # (4)
    ]
]

Here we generate two k8s_deploy targets, one for mynamespace (3), another for production deployment (4).

The directory structure of configmaps looks like this:

grafana
└── configmaps
    ├── dev
    │   └── grafana
    │       └── ldap.toml
    └── prod
        └── grafana
            └── ldap.toml

The configmaps_srcs parameter (1) will get resolved into the patterns configmaps/dev/**/* and configmaps/prod/**/*. The result of rendering the manifests bazel run //grafana:prod-grafana.show will have following manifest fragment:

apiVersion: v1
data:
  ldap.toml: |
    [[servers]]
    ...
kind: ConfigMap
metadata:
  name: grafana-k75h878g4f
  namespace: ops-prod

The name of directory on the first level of glob patten grafana become the configmap name. The ldap.toml file on the next level were embedded into the configmap.

In this example, the configmap renaming policy (2) is set to hash, so the configmap's name appears as grafana-k75h878g4f. (If the renaming policy was None, the configmap's name would remain as grafana.) All the references to the grafana configmap in other manifests are replaced with the generated name:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      volumes:
      ...
      - configMap:
          items:
          - key: ldap.toml
            path: ldap.toml
          name: grafana-k75h878g4f
        name: grafana-ldap

Injecting Docker Images

Third-party Docker images can be referenced directly in K8s manifests, but for most apps, we need to run our own images. The images are built in the Bazel build pipeline using rules_docker. For example, the java_image rule creates an image of a Java application from Java source code, dependencies, and configuration.

Here's a (very contrived) example of how this ties in with k8s_deploy. Here's the BUILD file located in the package //examples:

java_image(
    name = "helloworld_image",
    srcs = glob(["*.java"]),
    ...
)
k8s_deploy(
    name = "helloworld",
    manifests = ["helloworld.yaml"],
    images = {
        "helloworld_image": ":helloworld_image",  # (1)
    }
)

And here's helloworld.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: helloworld
spec:
  containers:
    - image: //examples:helloworld_image  # (2)

There images attribute dictionary (1) defines the images available for the substitution. The manifest file references the fully qualified image target path //examples:helloworld_image (2).

The image key value in the dictionary is used as an image push identifier. The best practice (as provided in the example) is to use image key that matches the label name of the image target.

When we bazel build the example, the rendered manifest will look something like this:

apiVersion: v1
kind: Pod
metadata:
  name: helloworld
spec:
  containers:
    - image: registry.example.com/examples/helloworld_image@sha256:c94d75d68f4c1b436f545729bbce82774fda07

The image substitution using an images key is supported, but not recommended (this functionality might be removed in the future). For example, helloworld.yaml can reference helloworld_image:

apiVersion: v1
kind: Pod
metadata:
  name: helloworld
spec:
  containers:
    - image: helloworld_image

Image substitutions for Custom Resource Definitions (CRD) resources could also use target references directly. Their digests are available through string substitution. For example,

apiVersion: v1
kind: MyCrd
metadata:
  name: my_crd
  labels:
    app_label_image_digest: "{{//examples:helloworld_image.digest}}"
    app_label_image_short_digest: "{{//examples:helloworld_image.short-digest}}"
spec:
  image: "{{//examples:helloworld_image}}"

would become

apiVersion: v1
kind: MyCrd
metadata:
  name: my_crd
  labels:
    app_label_image_digest: "e6d465223da74519ba3e2b38179d1268b71a72f"
    app_label_image_short_digest: "e6d465223d"
spec:
  image: registry.example.com/examples/helloworld_image@sha256:e6d465223da74519ba3e2b38179d1268b71a72f

An all examples above the image: URL points to the helloworld_image in the private Docker registry. The image is uploaded to the registry before any .apply or .gitops target is executed. See