Introduce testing standard/approach

[danielbate]

As we approach mainnet, it is important we start to implement some stricter testing standards that ensures parity with the protocol/FuelVM and gives us the greatest level of confidence in the changes we are making as uptake in the SDK increases. Additionally, we can consolidate and structure our testing to give greater clarity as to what behaviours we are currently supporting. I would like to propose some MoSCoW testing standards as well as a roadmap of changes to bring the current SDK in line with the standards.

Where we are now

• Most functions, classes and modules have a related test file that contains a mix of unit and feature tests
• Each package approaches testing slightly differently
• Each test case is using different language and syntax
• Some packages are separating unit and feature tests into separate directories
• Heavy reliance on fuel-gauge for feature and integration level testing
• Some integration testing is done within the documentation directories
• Ad-hoc creation of forc projects for tests

Proposed standards

• Functions, classes and modules must have a supporting unit test file
• Unit tests must verify all behaviours and correctness of functions, classes and modules
• Unit tests must mock any applicable functions, classes or modules that are not the service under test
• Unit tests must have 100% coverage of the service under test
• Packages should have a test directory
• Package test directories should contain fixtures and test data to be shared across tests within the package
• Package test directories could contain mocks and stubs of functions, classes and modules
• Package test directories should contain appropriate feature tests
• Feature tests must simulate real word interactions to validate the behaviour of a packages features from the perspective of an SDK consumer
• Integration tests should be used to demonstrate the interoperability of packages to deliver key behaviours to the SDK consumer
• Test Driven Development (TDD) could be used to identify, isolate and resolve erroneous behaviours and to implement bug fixes
• Shared test helpers and utilities should live within the utils packages
• Test cases should follow the gherkin syntax (e.g. Given X, When Y, Then Z - for unit tests only when and then are usually required as the given is our service under test. We can then utilise the it() syntax for unit tests and test() syntax for feature and integration tests)
• CI pipeline should enforce a coverage prerequisite to push to master (coverage level TBD)

Tasks (TBD)

• Document the test standards and approach within the SDK
• Refactor each package to contain a feature test directory
• Create tests for browser compatibility - Create test to browser compatibility #284
• Add test coverage check to CI pipeline

This should bring greater clarity to the behaviours supported by the SDK, give greater confidence in the quality of our tests and increase overall coverage as discussed in #753 . All standards and tasks are up for discussion and I would appreciate any insights 😃

[arboleya]

Good stuff @danielbate! 🚀

Somewhat related, we also have this:

• Create test to browser compatibility #284

We can resume the work on it now that #984 was merged.

Here's the spec from the last attempt:

• test: add testing groups and browser testing environment #728

[danielbate]

Good point @arboleya! Will add a task for just that.

[arboleya]

Do you mean other than the #284?

[danielbate]

Apologies, I will not be writing any more issues but I will be using the above task list as a reference to track changes made to bring us closer to adhering to the above standards. So I just meant that I will add another task to the above list, that references #284.

[danielbate]

The above requires updating due to further developments, however this is a good classification from @arboleya that will insight the specification:

Unit = Worries about units, usually mocking external stuff
Feature = Worries about features, not mocking or mocking minimally
Integration = Tests that target fuel-core locally
E2E - Tests that target fuel-core remotely

[arboleya]

Here's how these tests could be spread in directories:

<package-name>
├── src
│   ├── index.test.ts # This would be a unit
│   └── index.ts
└── test
    ├── e2e
    ├── feature
    ├── fixtures
    ├── integration
    └── utils

I ended up with test/{fixtures,utils,feature} in:

• feat: fuels CLI #1291

It worked great, and I think all packages should segregate their tests.

This goes back to maybe re-distributing all tests from inside fuel-gauge to other packages, and although something like this might not work (because these tests must import from fuels):

• packages/contract/test/integration/contract.test.ts

One thing that could work is:

• packages/fuels/test/integration/contract.test.ts

[petertonysmith94]

• Are we at a consensus with this discussion?
• Is it worth putting this in the CONTRIBUTING.md?
• Can we close this discussion?

cc @danielbate @arboleya

[arboleya]

I suspect we could close it for now, and we'll have a chance of resurfacing this after #2493 and #2490. The former could simplify solving the structural part, while the latter could narrow our test efforts toward public API.

@danielbate Thoughts?

[danielbate]

Yeah I'm okay with closing this, I don't think such strict standards are best suited for us right now. Our test suite is good shape and covers lots of functionality. We can revisit standards and structure after the restructuring.

[petertonysmith94]

Closing discussion as we've concluded discussions