From e49283e894f9fab7c1788182ca70865b8b78a51f Mon Sep 17 00:00:00 2001 From: Toni Finger Date: Wed, 7 Aug 2024 12:52:22 +0200 Subject: [PATCH 1/6] Add guidelines to write kaas conformace tests using sonobuoy Signed-off-by: Toni Finger --- .../tests/conformance-tests-sonobuoy.md | 221 ++++++++++++++++++ 1 file changed, 221 insertions(+) create mode 100644 contributor-docs/development/tests/conformance-tests-sonobuoy.md diff --git a/contributor-docs/development/tests/conformance-tests-sonobuoy.md b/contributor-docs/development/tests/conformance-tests-sonobuoy.md new file mode 100644 index 0000000000..be6e50cca6 --- /dev/null +++ b/contributor-docs/development/tests/conformance-tests-sonobuoy.md @@ -0,0 +1,221 @@ +--- +title: SCS Conformance Test KaaS Sonobuoy +type: +status: Draft +track: Global +--- + +SovereignCloudStack (SCS) makes use of [Sonobuoy][sonobuoy]] as a test framework to run its tests on Kubernetes clusters, which are to be audited for compliance. +The aim of using this framework is to make the execution of tests on a KaaS infrastructure as simple as possible. +Hence this reduces the effort required to establish SCS conformity to a minimum. + +In short this is achieved by storing all tests in a container image, which can then be called and launched on the clusters managed by sonoobuoy. + +> A more detailed description of why the SCS has decided to use sonobuoys can be found on the corresponding [Decsision Record][sonbouy-decision-record] +> In addition, sonobuoy is also used as the toolset for executing Kubernetes very own [conformance tests][k8s-conformance]. + + +This document is intended to assist conformance test authors to integrate their tests into the sonobuoy framework. +This requires a user to write the conformance tests in Golang, as this is the language provided by the framework itself. + + +# Step-by-step instructions for the development of sonobuoy tests using `docker` and `kind`: + +> This guide refers to the brief instructions provided in the [standards repository][scs-sonobuoy-example-guide] + + +## Prerequisite + +* [docker][docker-installation] +* [kind][kind-installation] +* [sonobuoy][sonobuoy-installation] + +``` +go install github.com/vmware-tanzu/sonobuoy@latest +``` + +## Set up development environment: + + +1. Clone the standard repository containing the conformance tests and navigate to the Sonobuoy example located at `standards/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework` + + +``` +git clone https://github.com/SovereignCloudStack/standards +cd standards/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework/ +``` + + +2. Check the prerequisites + +Within this directory you will find a `Makefile`, which is also used to create the test images. +To begin with, this can be used to check whether all the requirements for the development environment are met. + +``` +make dev-prerequests +``` + + +3. Create a kind cluster + +Once all the prerequisite software is installed, you can proceed by starting an kind cluster using the `Makefile` as follows: + +``` +make dev-setup +``` + + +4. Setting environment variables for the image build process + +``` +kubectl config view +``` + +``` +export IMAGE_VERSION_TAG="dev" +export K8S_HOST= +export K8S_PORT= +``` + +## Create a test: + + + + +In general, SCS tests for KaaS are derived from standards that define certain expected behaviors and functionalities of Kubernetes. +As an example for this step-by-step guide, let's assume a scenario in which there is a fictional standard called "scs-0299-v1-example-standard.md". +Pretend that the fictitious standard here stipulates that at least one pod MUST run in the namespace "namespace-test-a". + +> The functions and behaviors to be tested MUST be precisely defined in a standard. +> If you as a developer want to test something that you think is best tested but is not yet part of any standard, you MUST update the standard accordingly. + + + + + +1. Create example test + +The `scs_k8s_tests` directory contains the Golang files that define the tests. + +``` +cd standards/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework/scs_k8s_tests +``` + +First create a test file according to your standard and adhere to the naming convention accordingly. + +* The prefix MUST contain the name of the standard in lower case letters. +* As a suffix, the file must end with "_test.go". + +> The suffix requirement comes from the go test framework itself. All test files must end with `_test.go`. +> Otherwise they will not be selected by the test environment. + +As an example, we will create a file for the fictitious standard "scs-0299-v1-example-standard.md" as follows: + +``` +touch scs_0299_v1_example_standard_test.go +``` + +In this file, we can define the behavior we want to test for. +As an example, we test here whether there are more than zero pods in the namespace "namespace-test-a". +The execution of this test should fail by default as there should be no pods in the namespace and the namespace itself should not exist. +The aim is to display the results of a failed test so that we can show their interpretation in a later step. + +> Attention!!!: in order for the framework to select the functions for testing, their names must begin with "TEST_" in accordance with the naming convention of the golang test framework. +> TODO:!!! link to golang test framework docs + +Copy the following text into the file `scs_0299_v1_example_standard_test.go`: + +``` +package scs_k8s_tests + +import ( + "context" + "testing" + "time" + "fmt" + plugin_helper "github.com/vmware-tanzu/sonobuoy-plugins/plugin-helper" + corev1 "k8s.io/api/core/v1" + "sigs.k8s.io/e2e-framework/pkg/envconf" + "sigs.k8s.io/e2e-framework/pkg/features" +) + + +// This function checks if there are any pods in 'namespace-test-a'. +// The check should fail as there should be no pods present in this namespace. +// The purpose of this function is to display the handling of failed tests. +func Test_scs_0299_TestListPodsFailing(t *testing.T) { + f := features.New("pod list").Assess( + "pods from namespace 'namespace-test-a'", + func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context { + var pods corev1.PodList + err := cfg.Client().Resources("namespace-test-a").List(context.TODO(), &pods) + if err != nil { + t.Fatal(err) + } + t.Logf("found %d pods", len(pods.Items)) + if len(pods.Items) == 0 { + t.Fatal("no pods in namespace 'namespace-test-a'") + } + return ctx + }) + + testenv.Test(t, f.Feature()) +} + +``` + +3. Build the test image, upload it to the art cluster and run it + + +To create the image, execute the following: +``` +make dev-build +``` +This creates the Sonobuoy image and automatically uploads it to the image register of the kind cluster. + + +You can then run the tests by: +``` +make dev-run +``` + +Depending on how extensive the tests are, this may take some time. +To check the current test status manually, you can use the following command: + +``` +sonobuoy status +``` + + +4. Retrieving the results + +Once all tests have been executed successfully, you can read the results and receive feedback. +You can call up the results as follows: + +``` +TODO:!!! be described in more detail +``` + + + +## Clean up after: + +``` +make dev-clean +make dev-purge +``` + + + +[sonobuoy]: https://sonobuoy.io/ +[sonbouy-decision-record]: https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0200-v1-using-sonobuoy-for-kaas-conformance-tests.md +[k8s-conformance]: https://github.com/cncf/k8s-conformance/blob/master/instructions.md + +[kaas-sonobuoy-example] https://github.com/SovereignCloudStack/standards/tree/main/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework#sonobuoy-usage-for-development-of-tests + + +[docker-installation] https://docs.docker.com/engine/install/ +[sonobuoy-installation] https://sonobuoy.io/docs/v0.57.1/#installation +[kind-installation]https://kind.sigs.k8s.io/docs/user/quick-start/#installation + +[scs-sonobuoy-example-guide] https://github.com/SovereignCloudStack/standards/tree/main/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework#sonobuoy-usage-for-development-of-tests From 634e92daf73182fdd229399b5c3cf06128ea882a Mon Sep 17 00:00:00 2001 From: Toni Finger Date: Wed, 14 Aug 2024 10:17:48 +0200 Subject: [PATCH 2/6] Update guideline instructions Signed-off-by: Toni Finger --- .../development/tests/conformance-tests-sonobuoy.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/contributor-docs/development/tests/conformance-tests-sonobuoy.md b/contributor-docs/development/tests/conformance-tests-sonobuoy.md index be6e50cca6..06fd90839a 100644 --- a/contributor-docs/development/tests/conformance-tests-sonobuoy.md +++ b/contributor-docs/development/tests/conformance-tests-sonobuoy.md @@ -5,13 +5,13 @@ status: Draft track: Global --- -SovereignCloudStack (SCS) makes use of [Sonobuoy][sonobuoy]] as a test framework to run its tests on Kubernetes clusters, which are to be audited for compliance. +SovereignCloudStack (SCS) makes use of [Sonobuoy][sonobuoy] as a test framework to run its tests on Kubernetes clusters, which are to be audited for compliance. The aim of using this framework is to make the execution of tests on a KaaS infrastructure as simple as possible. Hence this reduces the effort required to establish SCS conformity to a minimum. In short this is achieved by storing all tests in a container image, which can then be called and launched on the clusters managed by sonoobuoy. -> A more detailed description of why the SCS has decided to use sonobuoys can be found on the corresponding [Decsision Record][sonbouy-decision-record] +> A more detailed description of why the SCS has decided to use sonobuoys can be found on the corresponding [Decsision Record][sonbouy-decision-record]. > In addition, sonobuoy is also used as the toolset for executing Kubernetes very own [conformance tests][k8s-conformance]. @@ -192,28 +192,27 @@ sonobuoy status Once all tests have been executed successfully, you can read the results and receive feedback. You can call up the results as follows: -``` -TODO:!!! be described in more detail -``` +> TODO:!!! to be described in more detail + ## Clean up after: +To clean up the resourcec used for development, you can use the following commands: + ``` make dev-clean make dev-purge ``` - [sonobuoy]: https://sonobuoy.io/ [sonbouy-decision-record]: https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0200-v1-using-sonobuoy-for-kaas-conformance-tests.md [k8s-conformance]: https://github.com/cncf/k8s-conformance/blob/master/instructions.md [kaas-sonobuoy-example] https://github.com/SovereignCloudStack/standards/tree/main/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework#sonobuoy-usage-for-development-of-tests - [docker-installation] https://docs.docker.com/engine/install/ [sonobuoy-installation] https://sonobuoy.io/docs/v0.57.1/#installation [kind-installation]https://kind.sigs.k8s.io/docs/user/quick-start/#installation From 2985fbac76458303c5a92e5c1d27b98b20ae1e5e Mon Sep 17 00:00:00 2001 From: Toni Finger Date: Wed, 21 Aug 2024 14:19:02 +0200 Subject: [PATCH 3/6] fixup markdownlint findings Signed-off-by: Toni Finger --- .../tests/conformance-tests-sonobuoy.md | 154 ++++++++---------- 1 file changed, 66 insertions(+), 88 deletions(-) diff --git a/contributor-docs/development/tests/conformance-tests-sonobuoy.md b/contributor-docs/development/tests/conformance-tests-sonobuoy.md index 06fd90839a..b6976822c6 100644 --- a/contributor-docs/development/tests/conformance-tests-sonobuoy.md +++ b/contributor-docs/development/tests/conformance-tests-sonobuoy.md @@ -1,9 +1,4 @@ ---- -title: SCS Conformance Test KaaS Sonobuoy -type: -status: Draft -track: Global ---- +# SCS Conformance Test KaaS Sonobuoy SovereignCloudStack (SCS) makes use of [Sonobuoy][sonobuoy] as a test framework to run its tests on Kubernetes clusters, which are to be audited for compliance. The aim of using this framework is to make the execution of tests on a KaaS infrastructure as simple as possible. @@ -14,90 +9,75 @@ In short this is achieved by storing all tests in a container image, which can t > A more detailed description of why the SCS has decided to use sonobuoys can be found on the corresponding [Decsision Record][sonbouy-decision-record]. > In addition, sonobuoy is also used as the toolset for executing Kubernetes very own [conformance tests][k8s-conformance]. - This document is intended to assist conformance test authors to integrate their tests into the sonobuoy framework. This requires a user to write the conformance tests in Golang, as this is the language provided by the framework itself. - -# Step-by-step instructions for the development of sonobuoy tests using `docker` and `kind`: +## Step-by-step instructions for the development of sonobuoy tests using `docker` and `kind` > This guide refers to the brief instructions provided in the [standards repository][scs-sonobuoy-example-guide] - -## Prerequisite +### Prerequisite * [docker][docker-installation] * [kind][kind-installation] -* [sonobuoy][sonobuoy-installation] +* [sonobuoy][sonobuoy-installation] -``` +```bash go install github.com/vmware-tanzu/sonobuoy@latest ``` -## Set up development environment: - - -1. Clone the standard repository containing the conformance tests and navigate to the Sonobuoy example located at `standards/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework` +### Set up development environment +#### 1. Clone the standard repository containing the conformance tests and navigate to the Sonobuoy example located at `standards/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework` -``` +```bash git clone https://github.com/SovereignCloudStack/standards cd standards/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework/ ``` - -2. Check the prerequisites +#### 2. Check the prerequisites Within this directory you will find a `Makefile`, which is also used to create the test images. To begin with, this can be used to check whether all the requirements for the development environment are met. -``` +```bash make dev-prerequests ``` - -3. Create a kind cluster +#### 3. Create a kind cluster Once all the prerequisite software is installed, you can proceed by starting an kind cluster using the `Makefile` as follows: -``` +```bash make dev-setup ``` +#### 4. Setting environment variables for the image build process -4. Setting environment variables for the image build process - -``` +```bash kubectl config view ``` -``` +```bash export IMAGE_VERSION_TAG="dev" export K8S_HOST= export K8S_PORT= ``` -## Create a test: - - - +### Create a test In general, SCS tests for KaaS are derived from standards that define certain expected behaviors and functionalities of Kubernetes. As an example for this step-by-step guide, let's assume a scenario in which there is a fictional standard called "scs-0299-v1-example-standard.md". Pretend that the fictitious standard here stipulates that at least one pod MUST run in the namespace "namespace-test-a". -> The functions and behaviors to be tested MUST be precisely defined in a standard. +> The functions and behaviors to be tested MUST be precisely defined in a standard. > If you as a developer want to test something that you think is best tested but is not yet part of any standard, you MUST update the standard accordingly. - - - - -1. Create example test +#### 1. Create example test The `scs_k8s_tests` directory contains the Golang files that define the tests. -``` +```bash cd standards/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework/scs_k8s_tests ``` @@ -111,11 +91,11 @@ First create a test file according to your standard and adhere to the naming con As an example, we will create a file for the fictitious standard "scs-0299-v1-example-standard.md" as follows: -``` +```bash touch scs_0299_v1_example_standard_test.go ``` -In this file, we can define the behavior we want to test for. +In this file, we can define the behavior we want to test for. As an example, we test here whether there are more than zero pods in the namespace "namespace-test-a". The execution of this test should fail by default as there should be no pods in the namespace and the namespace itself should not exist. The aim is to display the results of a failed test so that we can show their interpretation in a later step. @@ -125,18 +105,18 @@ The aim is to display the results of a failed test so that we can show their int Copy the following text into the file `scs_0299_v1_example_standard_test.go`: -``` +```go package scs_k8s_tests import ( - "context" - "testing" - "time" - "fmt" - plugin_helper "github.com/vmware-tanzu/sonobuoy-plugins/plugin-helper" - corev1 "k8s.io/api/core/v1" - "sigs.k8s.io/e2e-framework/pkg/envconf" - "sigs.k8s.io/e2e-framework/pkg/features" + "context" + "testing" + //~ "time" + //~ "fmt" + //~ plugin_helper "github.com/vmware-tanzu/sonobuoy-plugins/plugin-helper" + corev1 "k8s.io/api/core/v1" + "sigs.k8s.io/e2e-framework/pkg/envconf" + "sigs.k8s.io/e2e-framework/pkg/features" ) @@ -144,77 +124,75 @@ import ( // The check should fail as there should be no pods present in this namespace. // The purpose of this function is to display the handling of failed tests. func Test_scs_0299_TestListPodsFailing(t *testing.T) { - f := features.New("pod list").Assess( - "pods from namespace 'namespace-test-a'", - func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context { - var pods corev1.PodList - err := cfg.Client().Resources("namespace-test-a").List(context.TODO(), &pods) - if err != nil { - t.Fatal(err) - } - t.Logf("found %d pods", len(pods.Items)) - if len(pods.Items) == 0 { - t.Fatal("no pods in namespace 'namespace-test-a'") - } - return ctx - }) - - testenv.Test(t, f.Feature()) + f := features.New("pod list").Assess( + "pods from namespace 'namespace-test-a'", + func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context { + var pods corev1.PodList + err := cfg.Client().Resources("namespace-test-a").List(context.TODO(), &pods) + if err != nil { + t.Fatal(err) + } + t.Logf("found %d pods", len(pods.Items)) + if len(pods.Items) == 0 { + t.Fatal("no pods in namespace 'namespace-test-a'") + } + return ctx + }) + + testenv.Test(t, f.Feature()) } ``` -3. Build the test image, upload it to the art cluster and run it - +#### 3. Build the test image, upload it to the art cluster and run it To create the image, execute the following: -``` + +```bash make dev-build ``` -This creates the Sonobuoy image and automatically uploads it to the image register of the kind cluster. +This creates the Sonobuoy image and automatically uploads it to the image register of the kind cluster. You can then run the tests by: -``` + +```bash make dev-run ``` -Depending on how extensive the tests are, this may take some time. +Depending on how extensive the tests are, this may take some time. To check the current test status manually, you can use the following command: -``` +```bash sonobuoy status ``` +#### 4. Retrieving the results -4. Retrieving the results +```bash +make dev-result +cat results/plugins/scsconformance/results/global/out.json +cat results/plugins/scsconformance/sonobuoy_results.yaml +``` Once all tests have been executed successfully, you can read the results and receive feedback. You can call up the results as follows: > TODO:!!! to be described in more detail - - - -## Clean up after: +### Clean up after To clean up the resourcec used for development, you can use the following commands: -``` +```bash make dev-clean make dev-purge ``` - [sonobuoy]: https://sonobuoy.io/ [sonbouy-decision-record]: https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0200-v1-using-sonobuoy-for-kaas-conformance-tests.md [k8s-conformance]: https://github.com/cncf/k8s-conformance/blob/master/instructions.md - -[kaas-sonobuoy-example] https://github.com/SovereignCloudStack/standards/tree/main/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework#sonobuoy-usage-for-development-of-tests - -[docker-installation] https://docs.docker.com/engine/install/ -[sonobuoy-installation] https://sonobuoy.io/docs/v0.57.1/#installation -[kind-installation]https://kind.sigs.k8s.io/docs/user/quick-start/#installation - -[scs-sonobuoy-example-guide] https://github.com/SovereignCloudStack/standards/tree/main/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework#sonobuoy-usage-for-development-of-tests +[docker-installation]: https://docs.docker.com/engine/install/ +[sonobuoy-installation]: https://sonobuoy.io/docs/v0.57.1/#installation +[kind-installation]: https://kind.sigs.k8s.io/docs/user/quick-start/#installation +[scs-sonobuoy-example-guide]: https://github.com/SovereignCloudStack/standards/tree/main/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework#sonobuoy-usage-for-development-of-tests From edd743c0e0004b915d50c2a71fe5ab9c3fb9abdd Mon Sep 17 00:00:00 2001 From: Toni Finger Date: Wed, 21 Aug 2024 15:34:27 +0200 Subject: [PATCH 4/6] fixup change quotes to notes Signed-off-by: Toni Finger --- .../development/tests/conformance-tests-sonobuoy.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/contributor-docs/development/tests/conformance-tests-sonobuoy.md b/contributor-docs/development/tests/conformance-tests-sonobuoy.md index b6976822c6..c9ee1d29af 100644 --- a/contributor-docs/development/tests/conformance-tests-sonobuoy.md +++ b/contributor-docs/development/tests/conformance-tests-sonobuoy.md @@ -6,6 +6,7 @@ Hence this reduces the effort required to establish SCS conformity to a minimum. In short this is achieved by storing all tests in a container image, which can then be called and launched on the clusters managed by sonoobuoy. +> [!NOTE] > A more detailed description of why the SCS has decided to use sonobuoys can be found on the corresponding [Decsision Record][sonbouy-decision-record]. > In addition, sonobuoy is also used as the toolset for executing Kubernetes very own [conformance tests][k8s-conformance]. @@ -14,6 +15,7 @@ This requires a user to write the conformance tests in Golang, as this is the la ## Step-by-step instructions for the development of sonobuoy tests using `docker` and `kind` +> [!NOTE] > This guide refers to the brief instructions provided in the [standards repository][scs-sonobuoy-example-guide] ### Prerequisite @@ -70,6 +72,7 @@ In general, SCS tests for KaaS are derived from standards that define certain ex As an example for this step-by-step guide, let's assume a scenario in which there is a fictional standard called "scs-0299-v1-example-standard.md". Pretend that the fictitious standard here stipulates that at least one pod MUST run in the namespace "namespace-test-a". +> [!NOTE] > The functions and behaviors to be tested MUST be precisely defined in a standard. > If you as a developer want to test something that you think is best tested but is not yet part of any standard, you MUST update the standard accordingly. @@ -178,6 +181,7 @@ cat results/plugins/scsconformance/sonobuoy_results.yaml Once all tests have been executed successfully, you can read the results and receive feedback. You can call up the results as follows: +> [!NOTE] > TODO:!!! to be described in more detail ### Clean up after From a41231abec1cfba4e1f59b4c3695b7952cb37c95 Mon Sep 17 00:00:00 2001 From: Toni Finger Date: Tue, 1 Oct 2024 09:18:51 +0200 Subject: [PATCH 5/6] Update conformance-tests-sonobuoy.md Signed-off-by: Toni Finger --- .../tests/conformance-tests-sonobuoy.md | 56 ++++++++++++++----- 1 file changed, 42 insertions(+), 14 deletions(-) diff --git a/contributor-docs/development/tests/conformance-tests-sonobuoy.md b/contributor-docs/development/tests/conformance-tests-sonobuoy.md index c9ee1d29af..9fd1a74dd2 100644 --- a/contributor-docs/development/tests/conformance-tests-sonobuoy.md +++ b/contributor-docs/development/tests/conformance-tests-sonobuoy.md @@ -23,6 +23,8 @@ This requires a user to write the conformance tests in Golang, as this is the la * [docker][docker-installation] * [kind][kind-installation] * [sonobuoy][sonobuoy-installation] +* [yq][yq-installation] +* [jq][jq-installation] ```bash go install github.com/vmware-tanzu/sonobuoy@latest @@ -34,7 +36,7 @@ go install github.com/vmware-tanzu/sonobuoy@latest ```bash git clone https://github.com/SovereignCloudStack/standards -cd standards/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework/ +cd standards/Tests/kaas/kaas-sonobuoy-tests/ ``` #### 2. Check the prerequisites @@ -54,16 +56,12 @@ Once all the prerequisite software is installed, you can proceed by starting an make dev-setup ``` -#### 4. Setting environment variables for the image build process +#### 4. Setting environment variables for the development process -```bash -kubectl config view -``` +Set the number code of the standard you are currently working on. ```bash -export IMAGE_VERSION_TAG="dev" -export K8S_HOST= -export K8S_PORT= +export TESTFUNCTION_CODE= ``` ### Create a test @@ -81,7 +79,7 @@ Pretend that the fictitious standard here stipulates that at least one pod MUST The `scs_k8s_tests` directory contains the Golang files that define the tests. ```bash -cd standards/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework/scs_k8s_tests +cd standards/Tests/kaas/kaas-sonobuoy-tests/scs_k8s_tests ``` First create a test file according to your standard and adhere to the naming convention accordingly. @@ -89,6 +87,7 @@ First create a test file according to your standard and adhere to the naming con * The prefix MUST contain the name of the standard in lower case letters. * As a suffix, the file must end with "_test.go". +> [!NOTE] > The suffix requirement comes from the go test framework itself. All test files must end with `_test.go`. > Otherwise they will not be selected by the test environment. @@ -103,10 +102,13 @@ As an example, we test here whether there are more than zero pods in the namespa The execution of this test should fail by default as there should be no pods in the namespace and the namespace itself should not exist. The aim is to display the results of a failed test so that we can show their interpretation in a later step. +> [!NOTE] > Attention!!!: in order for the framework to select the functions for testing, their names must begin with "TEST_" in accordance with the naming convention of the golang test framework. -> TODO:!!! link to golang test framework docs +> The framework in use is [kubernetes-sigs/e2e-framework][e2e-framework]. +> They also provide examples. Before you start implementing your own tests, you should check whether you can use one of the examples as a starting point for your own implementation. +> Have a look at: [kubernetes-sigs/e2e-framework/examples][e2e-framework-examples] -Copy the following text into the file `scs_0299_v1_example_standard_test.go`: +As an example of this step-by-step guide, copy the following text into the file `scs_0299_v1_example_standard_test.go`: ```go package scs_k8s_tests @@ -147,7 +149,7 @@ func Test_scs_0299_TestListPodsFailing(t *testing.T) { ``` -#### 3. Build the test image, upload it to the art cluster and run it +#### 3. Build the test image, upload it to the kind cluster and run it To create the image, execute the following: @@ -174,8 +176,8 @@ sonobuoy status ```bash make dev-result -cat results/plugins/scsconformance/results/global/out.json -cat results/plugins/scsconformance/sonobuoy_results.yaml +cat results/plugins/scsconformance/results/global/out.json | jq +cat results/plugins/scsconformance/sonobuoy_results.yaml | yq ``` Once all tests have been executed successfully, you can read the results and receive feedback. @@ -193,6 +195,27 @@ make dev-clean make dev-purge ``` +### (optional) Run all at once: + +In addition, you can trigger all processes from above with one command. There is a single target in the Makefile for this: + +```bash +make dev-rerun +``` + +### (optional) Run only your testfunctions + +The above process always executes all tests. This can take some time. +Therefore, it is also possible to run only your standard test functions outside the scope of sonobuoy: + +```bash +export TESTFUNCTION_CODE="0299" +make test-function +``` + +> [!NOTE] +> [e2e-framework-examples] + [sonobuoy]: https://sonobuoy.io/ [sonbouy-decision-record]: https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0200-v1-using-sonobuoy-for-kaas-conformance-tests.md [k8s-conformance]: https://github.com/cncf/k8s-conformance/blob/master/instructions.md @@ -200,3 +223,8 @@ make dev-purge [sonobuoy-installation]: https://sonobuoy.io/docs/v0.57.1/#installation [kind-installation]: https://kind.sigs.k8s.io/docs/user/quick-start/#installation [scs-sonobuoy-example-guide]: https://github.com/SovereignCloudStack/standards/tree/main/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework#sonobuoy-usage-for-development-of-tests + +[yq-installation]:https://github.com/mikefarah/yq/?tab=readme-ov-file#install +[jq-installation]:https://jqlang.github.io/jq/download/ +[e2e-framework]:https://github.com/kubernetes-sigs/e2e-framework +[e2e-framework-examples]:https://github.com/kubernetes-sigs/e2e-framework/tree/main/examples From 9ac54d1872432857c61e6bf4e5254fdf1130a531 Mon Sep 17 00:00:00 2001 From: Toni Finger Date: Tue, 1 Oct 2024 10:02:46 +0200 Subject: [PATCH 6/6] Update conformance-tests-sonobuoy.md Signed-off-by: Toni Finger --- .../development/tests/conformance-tests-sonobuoy.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/contributor-docs/development/tests/conformance-tests-sonobuoy.md b/contributor-docs/development/tests/conformance-tests-sonobuoy.md index 9fd1a74dd2..de558a59e0 100644 --- a/contributor-docs/development/tests/conformance-tests-sonobuoy.md +++ b/contributor-docs/development/tests/conformance-tests-sonobuoy.md @@ -6,7 +6,7 @@ Hence this reduces the effort required to establish SCS conformity to a minimum. In short this is achieved by storing all tests in a container image, which can then be called and launched on the clusters managed by sonoobuoy. -> [!NOTE] +> [!NOTE] > A more detailed description of why the SCS has decided to use sonobuoys can be found on the corresponding [Decsision Record][sonbouy-decision-record]. > In addition, sonobuoy is also used as the toolset for executing Kubernetes very own [conformance tests][k8s-conformance]. @@ -15,7 +15,7 @@ This requires a user to write the conformance tests in Golang, as this is the la ## Step-by-step instructions for the development of sonobuoy tests using `docker` and `kind` -> [!NOTE] +> [!NOTE] > This guide refers to the brief instructions provided in the [standards repository][scs-sonobuoy-example-guide] ### Prerequisite @@ -70,7 +70,7 @@ In general, SCS tests for KaaS are derived from standards that define certain ex As an example for this step-by-step guide, let's assume a scenario in which there is a fictional standard called "scs-0299-v1-example-standard.md". Pretend that the fictitious standard here stipulates that at least one pod MUST run in the namespace "namespace-test-a". -> [!NOTE] +> [!NOTE] > The functions and behaviors to be tested MUST be precisely defined in a standard. > If you as a developer want to test something that you think is best tested but is not yet part of any standard, you MUST update the standard accordingly. @@ -183,7 +183,7 @@ cat results/plugins/scsconformance/sonobuoy_results.yaml | yq Once all tests have been executed successfully, you can read the results and receive feedback. You can call up the results as follows: -> [!NOTE] +> [!NOTE] > TODO:!!! to be described in more detail ### Clean up after