Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testutils.github: Only first result of parametrized test is reported #215

Open
miri64 opened this issue Apr 23, 2021 · 0 comments
Open

testutils.github: Only first result of parametrized test is reported #215

miri64 opened this issue Apr 23, 2021 · 0 comments
Assignees
Labels

Comments

@miri64
Copy link
Member

miri64 commented Apr 23, 2021

Task 1.3 is parametrized to run the unittests seperately and give a nicely parsable result:

@pytest.mark.parametrize('nodes, test_suite',
[pytest.param(['native'], t) for t in TESTS_SUITES],
indirect=['nodes'])
def test_task03(nodes, log_nodes, riotbase, test_suite):
run_unittests(test_suite, nodes, log_nodes, os.path.join(riotbase, APP))

However, this has the disadvantage, that with the current implementation of the testutils.github modules, only the first test is reported: For each test report, the bot checks the list of tasks in OP and only makes a comment if the test is not marked as done (i.e. if the tick box is empty):

try:
task_line, task = find_task_text(issue.body, tested_task)
except GithubException as e:
logger.error("Unable to get issue text of {}: {}"
.format(issue, e))
return
if not task_line or not task:
logger.warning("Unable to find task {spec}.{task} in the "
"tracking issue".format(**tested_task))
elif not task["done"]:
comment = make_comment(pytest_report, issue, task)

However, it then marks the tasks as done, when the test passed:

comment = make_comment(pytest_report, issue, task)
if comment:
comment_url = comment.html_url
else:
comment_url = None
if pytest_report.outcome == "passed":
mark_task_done(github.get_user().login, comment_url, issue,
task_line, tested_task)

This could lead to a misleading report, as all subsequent runs of the test (with different parameters) are not shown.

@miri64 miri64 added the bug label Apr 23, 2021
@miri64 miri64 self-assigned this Apr 23, 2021
miri64 added a commit to miri64/Release-Specs that referenced this issue Apr 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant