Contribution Experience in Kong ingress controller

In recent days, I tried to contribute to the Kong ingress controller project.
During the development, I faced some challenges and learned a lot about the controller development in Kubernetes.
Here are some experience of how to build the environment for testing and debugging the Kong ingress controller.

Background

This issue is a task for migrating the integration test to the isolated test with kube-e2e test framework.

Environment

The first problem that I faced is how to create a Kubernetes cluster for local testing.

At first time I thought that I need to prepare a Kubernetes cluster by myself.
But after reading the GitHub Actions workflow of the Kong ingress controller project, I found that they have already integrated the bootstrap of the Kubernetes cluster in the unit test framework.

Here is related code

1
2
3
4
5
6
7
8
9
10
11
12
13
switch clusterType {
case string(kind.KindClusterType):
 cluster, err := kind.NewFromExisting(clusterName)
 helpers.ExitOnErr(ctx, err)
 builder.WithExistingCluster(cluster)
 builder.WithAddons(metallb.New())
case string(gke.GKEClusterType):
 cluster, err := gke.NewFromExistingWithEnv(ctx, clusterName)
 helpers.ExitOnErr(ctx, err)
 builder.WithExistingCluster(cluster)
default:
 helpers.ExitOnErrWithCode(ctx, fmt.Errorf("unknown cluster type: %s", clusterType), consts.ExitCodeCantUseExistingCluster)
}

And in the isolated package, they also have test suite for bootstrapping.

Dependencies

Make sure that you ahve read the contribution document and testing document before startup.

After I read the workflows’ configurations and makefile, I found that many information about development and testing are already available in these files.

Mise

The bootstrapping have some dependencies, we can find it in the makefile.

Remember that each workflow which need to be executed in the CI/CD has a workflow file and generally you will find that it invoked the makefile.

1
2
3
4
MISE := $(shell which mise)
.PHONY: mise
mise:
 @mise -V >/dev/null || (echo "mise - https://github.com/jdx/mise - not found. Please install it." && exit 1

They use a tool called “mise” which is a tool for managing environments.

So you should follow the instructions to install mise first.

ja and yq

They dependes on jq and yq to parse yaml and json files.

So, install the latest version of jq and yq by following the instructions in the official website.

Docker Engine

They use the kind to create a “Kubernetes in Docker” cluster for testing.

So, you don’t have to install a cluster ahead of time, just install the kind and you will be able to run the tests.

You should install depenecies list in the contribution guidances.

Conclusion

These are engough for local development during my experience.

Notice:

I faced some problem with WSL and use my rocky linux server instead.

Maybe a vm on hyperv/virtualbox is better than WSL.

Before run the tests, cleanup current kind clusters will be better.

Testing

I modified the makefile for local testing, but after reading the testing document I find that the makefile has provide the ability to run specific tests.

Firstly, you should find your testing command line in makefile or workflow file.

Running all testing cases will take a long time, so I suggest that you run the specific test case that you want to test.

Here is an sample.

1
make test.integration.dbless GOTESTFLAGS="-count 1 -run TestUDPRouteEssentials"

Just pass the test to the make command with GOTESTFLAGS variable.

Tips for refactoring tests

If you want to migrate the integration tests to isolated package, remember the way of assert package.

It increments the failures rather than exit on failure.
So you should make sure the previous failure won’t effect next process.

Debugging

The robust way of debugging is log.

Leverage the t.Log to generate debug logs and remove them before committing to upstream.

Keep monitoring the Kubernetes status and use related commands to get status and logs of different components such as the kong ingress deployment.

I love the k9s because it provides interactive and live status of different components.
Native kubectl or any tools you like is ok. 😊

About break points if you really wnat to debug the process

If you really really want to debug the process in an interactive IDE way, after you run the makefile, you will find the output like below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
KONG_CLUSTER_VERSION="v1.28.0" \
        TEST_KONG_HELM_CHART_VERSION="2.40.0" \
        TEST_DATABASE_MODE="off" \
        GOFLAGS="-tags=integration_tests" \
        KONG_CONTROLLER_FEATURE_GATES="GatewayAlpha=true" \
        go test  \
        -v \
        -timeout "45m" \
        -parallel 12 \
        -race \
        -ldflags="" \
        -covermode=atomic \
        -coverpkg=./pkg/...,./internal/... \
        -coverprofile=coverage.dbless.out \
        ./test/integration/isolated -run 'TestTCPRouteReferenceGrant' -args --parallel

Use the configuration of this command line to configure your IDE and you will be able to attach to the running process.

Summary

Here is a brief introduction of my contribution experience with the Kong ingress controller project.

I hope this article can help you to get started with the Kong ingress controller project.

Plus, during my development, I find the way Kong using is efficient and I have the plan to apply the e2e testing framework to my project in CI.

References

[1] https://github.com/Kong/kubernetes-ingress-controller/blob/main/CONTRIBUTING.md

[2] https://github.com/Kong/kubernetes-ingress-controller/blob/main/TESTING.md

[3] https://github.com/kubernetes-sigs/e2e-framework

[4] https://github.com/Kong/kubernetes-ingress-controller/pull/6450