Report on category 2 type tests

Epubcheck based test reports for EPUB 3.3

This is just a write-up on a series of long discussions among Romain, Matt, and I, as well as some coding I did the past few days. It summarizes the process and the resulting report files that would be used to report the testing of the “category 2” type statement in the core spec. As agreed, the tests themselves will reuse the relevant tests of epubcheck; the question to be solved is how to report back to the Director the fact that all normative statements in the spec do have a proper tests and epubcheck is then able to test any EPUB 3.3 files for conformance.

Preamble

The structure of the epubcheck tests are very different from our own testing; they are not grouped as a series of EPUB document instances. Instead, epubcheck has a number of “feature” files (here is an EPUB 3.2 example). Each file contains, in a higher level language definition, a series of test cases by describing a “scenario” for each, and that is then dynamically compiled into a format that can be used via epubcheck. The details are not necessarily interesting for us; suffices to say that these “scenarios” are annotated, referring back to a section, and providing one or more tests for each normative statement in that section.

What would we aim to have?

The high level idea is to have:

• An EPUB 3.3 report file for the Director, much like our current test suite description, containing
  • a table per section, corresponding to the sections in the core spec
  • each row in that table would contain:
    • ID of a test, expressed through the name of a feature file and a line number in that file. This cell with the ID would also be a link to the feature file itself, with a fragment ID set to the line number (GitHub properly dereferences such URLs for source files)
    • a short description of the test.
• Use the same, respec-based technique to annotate the EPUB 3.3 core spec to add references to the relevant table row in the report file.

Note the major difference whereby, instead of having pointers from each individual MUST statement in the specification to the related tests, as we have now for the processing tests, we would have those pointers on the section level, with the assumption that the collection of the tests in a section would cover all the MUST statements in the section. The reason for this discrepancy is simply a measure of managing our own time: while it would be possible to get to the same granularity as we have for the processing tests, it would require a huge work to do that for all the category 2 statements in view of the very large number of tests in epubcheck (note that there are no anchors in the core spec source for the MUST statements). This would be tedious, error-prone, and we'd propose not to do it (at least for now).

How do we get there

The road leading to this goal contains the following steps:

1. Have a comprehensive epubcheck test suite available for EPUB 3.3. This is close, the epubcheck team is working on it. Our goal is to avoid any requirements on the test suite that would slow the work on epubcheck down.
2. Annotate the epubcheck feature files (manually?) to include, for each section reference:
   1. The exact title, with section numbers, of the relevant section
   2. The @id value in the core spec corresponding to the section (this is the basis for the various cross-references)
3. Have a script extracting the values from all feature files into, say, a simple JSON (or CSV) file that lists all the tests with a reference to the section whose statement they test, and also include the scenarios that describe what the test do
4. Have a script that, using the aforementioned JSON file
   1. Produces the report file
   2. Modifies the EPUB core spec by adding the respec-style test references to relevant sections (note that absolute URL-s to the tests must be used to avoid interfering with the processing test references that also occur in the core spec)

Matt has already done a preliminary work (using EPUB 3.2 tests) for (2) and (3) above. A combination of some manual work and scripts can be used to do a final version for those steps.

I have just implemented the script in (4) using some fake data. (Note that, for technical reasons related to the way respec does its job, some additional script had to be added to the specification source to provide a proper display of those test references.) The scripts are currently on my machine, it will have to be added to the repository once the tests are complete.

Note that, in contrast to the processing steps, I expect the regeneration process to be run only a few times; we will have the bulk of the tests available. I.e., I do not think it will be necessary to combine this with some GitHub actions with an automatic processing; much like the current acknowledgement section generation in the specs, we would have to run these only a few times.