unit tests

[JoFrhwld]

We need some kind of unit tests to try out new code.

[chrisbrickhouse]

This was partially implemented in #55 and is improved in #60, and should be a top priority behind improving user documentation.

Benefits of tests

Tests are beneficial for new features, but also ensuring that existing code is maintainable.

• When fixing bugs, we want to be sure that our fix did not cause problems elsewhere in the system, and the likelihood of a "butterfly effect" increases as the code complexity increases. Tests mitigate this problem because they will show whether distant parts of the code still work as expected.
• Tests speed up the development cycle. When a bug is identified, an ideal cycle would be to write a test for the expected behavior and fails (i.e., replicates the bug) and then develop until the test passes. This helps uncover other bugs in the code (e.g., what happened in Bug fixes for custom dictionaries #60), ensures that future development never introduces the same bug, and organically increases code coverage without needing to write tests for everything all at once.
• Code refactoring is safer and triaged by testing. Certain aspects of the code are redundant or complex, and should be refactored to their own functions or reduce complexity. Tests prioritize this by identifying functional units---if a function is hard to write a test for, it should probably be broken up. They make this refactoring safer because the tests should still pass after the refactor as the input-output should be the same regardless of what we do to the code inside.

Proposed contributor policies

With the stated benefits in mind, we should identify some policies which will help ensure adequate testing going forward (see RFC2119 for definitions of capitalized terms).

1. Pull requests to the development branch (FAVE/dev) SHOULD include tests which demonstrate the expected behavior.
2. Pull requests to the stable branch (FAVE/master) MUST include tests which demonstrate the expected behavior.
3. Pull requests MUST NOT be merged if they have failing tests.

With 1 we still allow contributions from developers who may not be proficient in test writing which keeps the barrier to contribution low. To comply with 2, maintainers can either write tests themselves before merging to stable or tutor developers on test writing. Policy 3 ensures that we avoid regressions at all times and IIRC GitHub has a built-in option to enforce this on "protected" branches.

Ideally we would have some minimal documentation on how to write tests, but IMO that's low priority compared with end user documentation. These and any future policies can be documented in a CONTRIBUTING.md file or in a relevant section of the documentation like README.md or the sphinx docs.