[Feature Request] enable compliance tests to use plugins for cluster provisioning

[mbuechse]

The entry point for executing the conformance tests is a provided kubeconfig, which can then be used by the test framework sonobuoy.
Hence we would therefore have to place a prior process which ensures that this kubeconfig file is available for the execution of the test.
This process should be handled through the creation of plugins.
For each kubernetes provider, specific plugins should be added to the CI/CD pipeline, which will allow the required cluster to be created.

---

DRAFT:

In the context of this topic, we need to apply the kaas test in the zuul pipeline:

• (optional) for the development of the KaaS tests itself?
• for testing the k8s environment of the different CSP
  • https://github.com/SovereignCloudStack/zuul-config/tree/main/Tests
  • https://github.com/SovereignCloudStack/standards/tree/main/.github/workflows
  • https://github.com/SovereignCloudStack/standards/blob/main/.zuul.d/config.yaml

The plugin mechanism should cover the following scenarios:

1. plugin could just use a static cluster (with some provided Kubeconfig)
2. plugin could use the Cluster API to provision and tear down a cluster
3. plugin could use the "central API" to provision and tear down a cluster

Zuul pipeline integration

The zuul jobs for carrying out the conformance test could be designed as follows:

flowchart TB
  classDef PluginSpecificAdjustment fill:#bbf,stroke-width:2px,color:#fff,stroke-dasharray: 5 5
  classDef SonobuoySpecificAdjustment fill:#faa,stroke-width:2px,color:#fff,stroke-dasharray: 5 5
  subgraph TOP[zuul job execution]
    direction TB
    subgraph start[prepare tests]
        direction LR
        do[install requirements];
    end

    subgraph iaas[scs-test-runner.py:iaas]
        direction LR
        start_iaas[access OS]-->end_b1[run tests];

    end

    subgraph kaas[scs-test-runner.py:kaas]
        direction TB
        start_kaas[plugin: create clusters]:::PluginSpecificAdjustment-->scs-compliance;
        scs-compliance-->end_kaas[plugin: delete clusters]:::PluginSpecificAdjustment;
    end

    subgraph scs-compliance[scs-compliance-check.py]
        sonobuoy_executor_a[sonobuoy_executor:run cncf]:::SonobuoySpecificAdjustment;
        sonobuoy_executor_a-->run_python_tests_a[k8s_version_policy.py];
       run_python_tests_a-->run_python_tests_b[k8s_node_distribution_check.py]
       run_python_tests_b-->run_python_tests[***.py]
    end

    subgraph final[finalize]
        direction LR
        done[evaluate results];

    end
  end
  start --> iaas
  iaas --> kaas
  kaas --> final

Loading

We should therefore use the existing infrastructure of the following scripts

• https://github.com/SovereignCloudStack/standards/blob/main/Tests/scs-compliance-check.py
• https://github.com/SovereignCloudStack/standards/blob/main/Tests/scs-test-runner.py

which make use of these spec files:

• https://github.com/SovereignCloudStack/standards/blob/main/Tests/scs-compatible-kaas.yaml (relevant for KaaS)
• https://github.com/SovereignCloudStack/standards/blob/main/Tests/scs-compatible-iaas.yaml

Hence provide specific plugin implementations have to provide methods for

• installing requirements
• creating a cluster
• deleting a cluster