RFE: External tmt plugin design and procedures

[LecrisUT]

This is an overall summary of the current state of tmt plugins and what can be done to enable external projects to become tmt plugins. There is a companion tmt_plugins.html in the matrix room which github does not allow me to share.

Current state

Currently we are able to design and develop a tmt external plugin that would work with the tmt.plugin entry-point. The issue is that there is currently no mechanism to import the external plugin within the testing-farm framework. If we run tmt in other environments like in a GH Actions, we are able to pip install any additional plugins we need in between the action to call tmt, but because of security concerns testing-farm cannot install these plugins on the tmt-controller which would decipher the fmf files that it needs to execute.

The primary issue to resolve here is formulating a procedure for getting from a github repo to testing-farm so that test files in src.fedoraproject.org are recognized. Basically the package manager should be able to use a fmf plan like

prepare:
  - how: cmake
    source: /path/to/cmake/source
execute:
  how: tmt

Where the tmt-cmake plugin would be developed and maintained in a different github repo.

Benefits of external plugins

• Enable async development of testing frameworks, plugins etc.
• Reduce dependencies in tmt package
• Reduce testing overhead in tmt repo
• Easier development outside of testing-farm environment, e.g. GH actions, AWS/openstack providers, etc.

Issues to resolve

• Security concerns, e.g. don't mine crypto in the plugin
  Currently the proposed approach is to have testing-farm review the initial tmt plugin and its updates for suspicious behaviors and give the green-light for being included in the tmt-controller image
• Quality concerns, e.g. make sure packagers do not have a regression
  This is slightly addressed in my proposal for inclusion of a .fmf/plugins (see RFE: fmf plugin/dialect fmf#236 for more details), because than the test designer would know where to raise issues and it's less of a burden on keeping a monolith project 100% functional
• Formulate a procedure for testing-farm to review a plugin for inclusion in the tmt-controller image
• How often should tmt-controller image be rebuilt
• Recommended structure of a tmt-plugin repo
  I have a somewhat working structure of how a tmt-plugin repo can work from writing tmt-cmake, but I still want to develop it to completion before I can gather all of the issues that a developer is facing. A glimpse of some of the issues so far: extending the json-schema validator, providing a tmt.testing package for testing inside pytest, providing a nightly copr project to target the CI.
• Enable a way for plugin developers to request a test image to be used in testing-farm-cli. Is that --worker-image option in the current cli? In which case how could some developers gain some access to it to more easily work on functional testing some realistic projects?