Implement Cobertura coverage format #2298

joeskeen · 2023-02-10T18:14:24Z

PR Summary

Cobertura is a commonly used code coverage format, and has been requested as a feature: #2203

To implement this feature, I copy-pasted the JaCoCo code and modified it to produce the required XML structure for Cobertura.

I am not an expert on the Cobertura format, but I did reference their DTD and the Cobertura output from my TypeScript project that uses jest-junit. If anyone is able to review it for correctness, I would greatly appreciate it. A good place to analyze the output is the unit test I added, where you can compare the JaCoCo output directly above it to the new Cobertura output (I used the same inputs for both reports).

PR Checklist

PR has meaningful title
Summary describes changes
PR is ready to be merged
- If not, use the arrow next to Create Pull Request to mark it as a draft. PR can be marked Ready for review when it's ready.
Tests are added/update (if required)
Documentation is updated/added (if required)

joeskeen · 2023-02-10T18:17:08Z

@eizedev Since you were the OP for issue #2298, would you please review?

joeskeen · 2023-02-10T18:33:07Z

Also, it seems that the XML attributes don't always have consistent ordering when the XML is serialized. This is currently failing the tests. Does anyone know how we could get past this?

https://nohwnd.visualstudio.com/Pester/_build/results?buildId=2226&view=logs&j=83ea85d6-f9df-5810-618d-ec9b0f05919b&t=863f8651-66b5-5008-0935-1e501bd49882&l=4971

##[error]  Expected: '...eforge.net/xml/coverage-04.dtd"[]><coverage branch-rate=...'
##[error]  But was:  '...eforge.net/xml/coverage-04.dtd"[]><coverage lines-covere...'

src/functions/Coverage.ps1

fflaten · 2023-02-10T18:49:36Z

Thanks for looking into this so fast! :)
This is more @nohwnd's territory but I'll give a review later. Left a quick comment for now.

eizedev · 2023-02-10T19:00:06Z

@eizedev Since you were the OP for issue #2298, would you please review?

I can not make it today to look at it, but try to look at it the next few days and test! Many, many thanks

joeskeen · 2023-02-10T21:07:37Z

After fixing all the build failures (related to multiple PS versions and OSs), I re-ran this branch to produce a Cobertura report against my other project and compared it to the HTML report I previously generated from the JaCoCo option. Everything seems to match up (whew!).

eizedev · 2023-02-11T14:09:26Z

@joeskeen just checked it, looks good to me! Again, thanks for your help and work.
I need to (and will) test it in our gitlab environment at work in the next 2 weeks to verify it, if gitlab can handle the cobertura coverage format.
if anyone else here could also test it, I would appreciate it.

fflaten

Please see comments below. As mentioned, I'm not an expert on coverage and haven't tested this format, so I might be wrong.

Still, it feels like line-elements are unnecessarily duplicated in the report and should only be reported once per method or class. Though it might work fine, it would cause large files

src/Main.ps1

src/functions/Coverage.ps1

tst/functions/Coverage.Tests.ps1

src/functions/Coverage.ps1

joeskeen · 2023-02-14T15:10:02Z

@fflaten thanks for the thorough review. I did some more testing yesterday and found that there were a lot of problems with my initial implementation. So yesterday I started from scratch and I'm liking it much better. I'll push the changes shortly after I address a couple linting issues.

joeskeen · 2023-02-14T16:22:11Z

@fflaten (or others) - my last remaining linting error is Where-Object usage:

The built-in *-Object-cmdlets are slow compared to alternatives in .NET. To fix a violation of this rule, consider using an alterantive like foreach-keyword etc.

What is the suggested alternative for Where-Object to comply with the linting and coding standards in this repository?

Edit: maybe this isn't an issue, I just found that Get-CoverageMissedCommands in the same file uses & $SafeCommands['Where-Object'] { $_.Breakpoint.HitCount -eq 0 } which produces the same warning, but that seems to be acceptable?

joeskeen · 2023-02-14T16:44:45Z

Looks like using System.Xml.Linq isn't OK for compatibility reasons; I'll rewrite those parts to use System.Xml instead.

src/functions/Coverage.ps1

fflaten · 2023-02-14T16:58:22Z

@fflaten (or others) - my last remaining linting error is Where-Object usage:

The built-in *-Object-cmdlets are slow compared to alternatives in .NET. To fix a violation of this rule, consider using an alterantive like foreach-keyword etc.

What is the suggested alternative for Where-Object to comply with the linting and coding standards in this repository?

Edit: maybe this isn't an issue, I just found that Get-CoverageMissedCommands in the same file uses & $SafeCommands['Where-Object'] { $_.Breakpoint.HitCount -eq 0 } which produces the same warning, but that seems to be acceptable?

A replacement depends on each scenario, but often foreach/for with if-condition.

If you can avoid them, that's great, but it's not critical in this code as it's only executed once in post-process. The rule is there to highlight potential performance issues in the runtime for code executed 100s or 1000s of times. 🙂

Please let me know when it's ready for review again.

src/functions/Coverage.ps1

azure-pipelines · 2024-07-10T15:19:25Z

Azure Pipelines successfully started running 1 pipeline(s).

fflaten · 2024-07-10T15:45:30Z

/azp run

azure-pipelines · 2024-07-10T15:45:39Z

Azure Pipelines successfully started running 1 pipeline(s).

nohwnd

Looks great, few nitpicks.

src/functions/Coverage.ps1

nohwnd · 2024-07-10T19:00:14Z

I was going to fix this PR but you beat me to it @fflaten .

Makes sense to backport to v5?

fflaten · 2024-07-10T19:11:41Z

I was going to fix this PR but you beat me to it @fflaten .

Felt this was a "you break it, you buy it" scenario 😄

Started testing it with ReportGenerator and got same HTML report with JaCoCo and Cobertura which is good. Haven't tested multiple source paths yet as I got distracted by weird codecov results like no coverage for BeLessThan (tested tst/functions against src/functions) in both formats.

Not too familiar with CC so please help testdrive it.

Makes sense to backport to v5?

Yes

nohwnd · 2024-07-10T19:55:20Z

I will try to test drive it, but not today. :)

And sorry @joeskeen for taking so long to merge this.

Co-authored-by: Jakub Jareš <[email protected]>

fflaten · 2024-07-12T22:03:42Z

src/functions/Coverage.ps1

+    # Report uses unix epoch time format (milliseconds since midnight 1/1/1970 UTC)
+    [long] $endTime = [System.DateTimeOffset]::UtcNow.ToUnixTimeMilliseconds()
+    [long] $startTime = [math]::Floor($endTime - $TotalMilliseconds)


Source for epoch time format: https://github.com/cobertura/cobertura/blob/0ff963284cecaace30f409a977dccb07c41a5a8f/cobertura/src/main/java/net/sourceforge/cobertura/reporting/xml/XMLReport.java#L89

Backport version will need to use:

$nineteenSeventy = & $SafeCommands['New-Object'] 'System.DateTime' -ArgumentList @(1970, 1, 1, 0, 0, 0, [System.DateTimeKind]::Utc) $now = [DateTime]::Now.ToUniversalTime() [long] $endTime = [math]::Floor(($now - $nineteenSeventy).TotalMilliseconds) [long] $startTime = [math]::Floor($endTime - $TotalMilliseconds)

fflaten · 2024-07-12T22:07:45Z

src/functions/Coverage.ps1

+    # Report uses unix epoch time format (milliseconds since midnight 1/1/1970 UTC)
+    [long] $endTime = [System.DateTimeOffset]::UtcNow.ToUnixTimeMilliseconds()


@nohwnd I included this bugfix for JaCoCo as well. [DateTime]"01/01/1970" is not proper epoch origin (missing UTC kind), so off by one hour in my timezone.

Source for epoch time format:
https://github.com/jacoco/jacoco/blob/77d2af5ec31faf04419d945e1a0c34da49f8a702/org.jacoco.core/src/org/jacoco/core/data/SessionInfo.java#L33-L38

Backport version will need to use:

$nineteenSeventy = & $SafeCommands['New-Object'] 'System.DateTime' -ArgumentList @(1970, 1, 1, 0, 0, 0, [System.DateTimeKind]::Utc) $now = [DateTime]::Now.ToUniversalTime() [long] $endTime = [math]::Floor(($now - $nineteenSeventy).TotalMilliseconds) [long] $startTime = [math]::Floor($endTime - $TotalMilliseconds)

fflaten · 2024-08-02T23:44:59Z

TODO: Need to update the hardcoded report in Coverage.Tests.ps1 when #2553 gets merged. Conflict.

nohwnd · 2024-10-03T18:31:45Z

/azp run

azure-pipelines · 2024-10-03T18:31:55Z

Azure Pipelines successfully started running 1 pipeline(s).

nohwnd · 2024-10-03T20:20:09Z

TODO: Need to update the hardcoded report in Coverage.Tests.ps1 when #2553 gets merged. Conflict.

You were right, it is all broken now 😅

fflaten · 2024-10-15T19:21:08Z

Hard-coded reports = The gift that keeps on giving 🙏 Fixed and ready for test & review.

* wip: Implement Cobertura coverage format * fix unit test * revert editor settings change * remove ?? operator * fix attribute ordering * make coverage report test work on all platforms * Fix windows paths * fix unit test for Windows paths * kick the build * re-implement Cobertura coverage report generation * fix compatibility issues * fix tests * removing Cobertura from v4 parameter options * fix compatibility with ReportGenerator * Update src/functions/Coverage.ps1 Co-authored-by: Frode Flaten <[email protected]> * fix whitespace * fix output * fix windows paths * order packages,classes,methods by name * change Cobertura DTD to loose * Tune coverage report for performance * Remove outdated condition * Add Cobertura DTD file * Apply suggestions from code review Co-authored-by: Jakub Jareš <[email protected]> * Fix typo and update JaCoCo starttime * Fix tests * Use epoch time for Cobertura and JaCoCo * Update test --------- Co-authored-by: Frode Flaten <[email protected]> Co-authored-by: Jakub Jareš <[email protected]>

* Implement Cobertura coverage format (#2298) * wip: Implement Cobertura coverage format * fix unit test * revert editor settings change * remove ?? operator * fix attribute ordering * make coverage report test work on all platforms * Fix windows paths * fix unit test for Windows paths * kick the build * re-implement Cobertura coverage report generation * fix compatibility issues * fix tests * removing Cobertura from v4 parameter options * fix compatibility with ReportGenerator * Update src/functions/Coverage.ps1 Co-authored-by: Frode Flaten <[email protected]> * fix whitespace * fix output * fix windows paths * order packages,classes,methods by name * change Cobertura DTD to loose * Tune coverage report for performance * Remove outdated condition * Add Cobertura DTD file * Apply suggestions from code review Co-authored-by: Jakub Jareš <[email protected]> * Fix typo and update JaCoCo starttime * Fix tests * Use epoch time for Cobertura and JaCoCo * Update test --------- Co-authored-by: Frode Flaten <[email protected]> Co-authored-by: Jakub Jareš <[email protected]> * Ignore call base class contructor in code coverage (#2553) * Ignore calls to base class constructor in code coverage * Update tests * Unix timestamps with supported apis * Add info about mode * output bps * output bps * output bps * What if we don't call it? * Revert "What if we don't call it?" This reverts commit 1c03a6b. * bsclass --------- Co-authored-by: Joe Skeen <[email protected]> Co-authored-by: Frode Flaten <[email protected]>

joeskeen added 2 commits February 10, 2023 10:51

wip: Implement Cobertura coverage format

932d9b4

fix unit test

36af738

joeskeen mentioned this pull request Feb 10, 2023

Would anyone from the Cobertura community be available for a quick code review? cobertura/cobertura#431

Open

joeskeen added 2 commits February 10, 2023 11:21

revert editor settings change

14e36af

remove ?? operator

658b358

fflaten reviewed Feb 10, 2023

View reviewed changes

src/functions/Coverage.ps1 Outdated Show resolved Hide resolved

github-advanced-security bot found potential problems Feb 10, 2023

View reviewed changes

src/functions/Coverage.ps1 Fixed Show fixed Hide fixed

joeskeen added 4 commits February 10, 2023 12:07

fix attribute ordering

1f6525b

make coverage report test work on all platforms

a2e5eb0

Fix windows paths

61eefc1

fix unit test for Windows paths

6c3db56

kick the build

3b6e06b

fflaten requested changes Feb 13, 2023

View reviewed changes

re-implement Cobertura coverage report generation

14e8ef7

github-advanced-security bot found potential problems Feb 14, 2023

View reviewed changes

src/functions/Coverage.ps1 Fixed Show fixed Hide fixed

src/functions/Coverage.ps1 Fixed Show fixed Hide fixed

src/functions/Coverage.ps1 Fixed Show fixed Hide fixed

src/functions/Coverage.ps1 Fixed Show fixed Hide fixed

fix compatibility issues

ab0fa57

fflaten reviewed Feb 14, 2023

View reviewed changes

src/functions/Coverage.ps1 Outdated Show resolved Hide resolved

joeskeen added 3 commits February 14, 2023 10:36

fix tests

70ed47a

removing Cobertura from v4 parameter options

386cd3c

fix compatibility with ReportGenerator

f0ac943

github-advanced-security bot found potential problems Jul 10, 2024

View reviewed changes

src/functions/Coverage.ps1 Outdated Show resolved Hide resolved

Remove outdated condition

13c13a7

nohwnd reviewed Jul 10, 2024

View reviewed changes

fflaten and others added 6 commits July 12, 2024 20:37

Merge remote-tracking branch 'upstream/main' into pr/joeskeen/2298

505afef

Add Cobertura DTD file

a12b1c7

Apply suggestions from code review

13f0b75

Co-authored-by: Jakub Jareš <[email protected]>

Fix typo and update JaCoCo starttime

b1f7f2e

Fix tests

c643f5e

Use epoch time for Cobertura and JaCoCo

80f2120

fflaten reviewed Jul 12, 2024

View reviewed changes

fflaten approved these changes Jul 12, 2024

View reviewed changes

fflaten mentioned this pull request Aug 3, 2024

Include more statements in code coverage #2554

Open

5 tasks

nohwnd self-assigned this Oct 1, 2024

fflaten added 2 commits October 15, 2024 19:08

Merge remote-tracking branch 'upstream/main' into pr/joeskeen/2298

58590c9

Update test

98b7881

nohwnd merged commit e7a5259 into pester:main Oct 30, 2024
11 checks passed

nohwnd mentioned this pull request Oct 30, 2024

[rel/5.x.x] Implement Cobertura coverage format #2572

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Cobertura coverage format #2298

Implement Cobertura coverage format #2298

joeskeen commented Feb 10, 2023 •

edited

Loading

joeskeen commented Feb 10, 2023

joeskeen commented Feb 10, 2023

fflaten commented Feb 10, 2023 •

edited

Loading

eizedev commented Feb 10, 2023

joeskeen commented Feb 10, 2023 •

edited

Loading

eizedev commented Feb 11, 2023 •

edited

Loading

fflaten left a comment •

edited

Loading

joeskeen commented Feb 14, 2023

joeskeen commented Feb 14, 2023 •

edited

Loading

joeskeen commented Feb 14, 2023

fflaten commented Feb 14, 2023 •

edited

Loading

azure-pipelines bot commented Jul 10, 2024

fflaten commented Jul 10, 2024

azure-pipelines bot commented Jul 10, 2024

nohwnd left a comment

nohwnd commented Jul 10, 2024

fflaten commented Jul 10, 2024 •

edited

Loading

nohwnd commented Jul 10, 2024

fflaten Jul 12, 2024 •

edited

Loading

fflaten Jul 12, 2024 •

edited

Loading

fflaten commented Aug 2, 2024 •

edited

Loading

nohwnd commented Oct 3, 2024

azure-pipelines bot commented Oct 3, 2024

nohwnd commented Oct 3, 2024

fflaten commented Oct 15, 2024 •

edited

Loading

		# Report uses unix epoch time format (milliseconds since midnight 1/1/1970 UTC)
		[long] $endTime = [System.DateTimeOffset]::UtcNow.ToUnixTimeMilliseconds()

Implement Cobertura coverage format #2298

Implement Cobertura coverage format #2298

Conversation

joeskeen commented Feb 10, 2023 • edited Loading

PR Summary

PR Checklist

joeskeen commented Feb 10, 2023

joeskeen commented Feb 10, 2023

fflaten commented Feb 10, 2023 • edited Loading

eizedev commented Feb 10, 2023

joeskeen commented Feb 10, 2023 • edited Loading

eizedev commented Feb 11, 2023 • edited Loading

fflaten left a comment • edited Loading

Choose a reason for hiding this comment

joeskeen commented Feb 14, 2023

joeskeen commented Feb 14, 2023 • edited Loading

joeskeen commented Feb 14, 2023

fflaten commented Feb 14, 2023 • edited Loading

azure-pipelines bot commented Jul 10, 2024

fflaten commented Jul 10, 2024

azure-pipelines bot commented Jul 10, 2024

nohwnd left a comment

Choose a reason for hiding this comment

nohwnd commented Jul 10, 2024

fflaten commented Jul 10, 2024 • edited Loading

nohwnd commented Jul 10, 2024

fflaten Jul 12, 2024 • edited Loading

Choose a reason for hiding this comment

fflaten Jul 12, 2024 • edited Loading

Choose a reason for hiding this comment

fflaten commented Aug 2, 2024 • edited Loading

nohwnd commented Oct 3, 2024

azure-pipelines bot commented Oct 3, 2024

nohwnd commented Oct 3, 2024

fflaten commented Oct 15, 2024 • edited Loading

joeskeen commented Feb 10, 2023 •

edited

Loading

fflaten commented Feb 10, 2023 •

edited

Loading

joeskeen commented Feb 10, 2023 •

edited

Loading

eizedev commented Feb 11, 2023 •

edited

Loading

fflaten left a comment •

edited

Loading

joeskeen commented Feb 14, 2023 •

edited

Loading

fflaten commented Feb 14, 2023 •

edited

Loading

fflaten commented Jul 10, 2024 •

edited

Loading

fflaten Jul 12, 2024 •

edited

Loading

fflaten Jul 12, 2024 •

edited

Loading

fflaten commented Aug 2, 2024 •

edited

Loading

fflaten commented Oct 15, 2024 •

edited

Loading