Fix for issue 47: When Pester tests fail, it should fail the build.

[splatteredbits]

No description provided.

[vexx32]

Typically setting $host.SetShouldExit() to anything other than 0 should fail the build. Note that a Write-Error -ErrorAction Stop or a throw will only cause the external error code to be set to 1.

Have you tested and confirmed this change allows chef verify and others to properly register the failure?

It may be worth testing to see whether exit $result.FailedCount may work more effectively, as well.

[vexx32]

I would be inclined to suggest using throw rather than Write-Error -ErrorAction Stop 🙂

Suggested change

	Write-Error -Message ('Pester finished with {0} failing tests.' -f $result.FailedCount) -ErrorAction Stop
	throw 'Pester finished with {0} failing tests.' -f $result.FailedCount

[vexx32]

Also, /cc @smurawski for the build failure, as it appears to be related to some changes he made (it's throwing up on the Kitchen::VERSION constant being missing; perhaps there's a place we should be setting that, but not sure exactly where that should be being set / pulled from).

[splatteredbits]

Typically setting $host.SetShouldExit() to anything other than 0 should fail the build. Note that a Write-Error -ErrorAction Stop or a throw will only cause the external error code to be set to 1.

It typically would, but Chef/Test Kitchen/something wraps the Pester verifiers PowerShell script and adds its own $host.SetShouldExit(). The script actually being run does this:

          Import-Module Pester -Force
          $TestPath = "#{config[:root_path]}"
          $OutputFilePath = $TestPath | Join-Path -ChildPath 'PesterTestResults.xml'
          $result = Invoke-Pester -OutputFile $OutputFilePath -OutputFormat NUnitXml -Path $TestPath -Passthru
          $result | Export-CliXml -Path (Join-Path -Path $TestPath -ChildPath 'result.xml')
          $host.SetShouldExit($result.FailedCount)
          if( $result.FailedCount )
          {
            Write-Error -Message ('Pester finished with {0} failing tests.' -f $result.FailedCount) -ErrorAction Stop
          }


$host.SetShouldExit($LASTEXITCODE)

Which doesn't work since Pester isn't a console app.

So, we can:

1. Write-Error -ErrorAction Stop, which prevents the forced $host.SetShouldExit($LASTEXITCODE) from being reached.
2. Explicitly set $LASTEXITCODE: $LASTEXITCODE = $result.FailedCount (I would need to test but pretty sure this would work).
3. exit $result.FailedCount (again, I'd have to test)

I think I prefer #2, but I'll let you make that call.

Have you tested and confirmed this change allows chef verify and others to properly register the failure?

Yes. I'd like to add an automated way of testing, but don't know how to tell Test Kitchen "I expect these tests to fail".

[smurawski]

@vexx32 I think pester.rb is missing a require "kitchen/version"

As to throw vs write-error -erroraction stop, I'm 🤷‍♂

[splatteredbits]

I confirmed that exit $result.FailedCount and $LASTEXITCODE = $result.FailedCount both cause the build to fail if there are any failed tests. I'm going to defensively use both, just in case future versions of Test Kitchen change how it wraps PowerShell code.

[vexx32]

I don't think setting $LASTEXITCODE does anything in particular, but there's nothing wrong with it other than that it's an automatic variable (which can sometimes be a bit iffy to try to overwrite).

Other than that minor point, this looks good. Thanks for fixing the build too! 💖 😊

[splatteredbits]

@smurawski @vexx32 Any idea when this change will get released?

[smurawski]

@splatteredbits Sorry for the delay - I was mostly out last week. I'll get it released today.

[smurawski]

Released!