You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We added basic support for XCTest's performance testing APIs in https://bugs.swift.org/browse/SR-1355, however one crucial component of that is still missing, namely recording baseline results of performance tests and triggering test failures when significant regressions are detected relative to those baselines.
I see a few pieces to this:
Persisting baseline data to disk. Xcode uses a Plist format for this. It would probably make sense to use a compatible format here.
Comparing test results to a previously recorded baseline and triggering a failure if the performance has regressed. Xcode's supposedly triggers a failure if Average - BaselineAverage is greater than 0.1 seconds and is more than 10% higher than BaselineAverage itself.
Providing a mechanism (probably command-line flags?) instructing the runner where to find and create baseline data, and for causing the baseline data to be updated.
The text was updated successfully, but these errors were encountered:
Additional Detail from JIRA
md5: 89bb688c4da749aeb43653263685237d
Issue Description:
We added basic support for XCTest's performance testing APIs in https://bugs.swift.org/browse/SR-1355, however one crucial component of that is still missing, namely recording baseline results of performance tests and triggering test failures when significant regressions are detected relative to those baselines.
I see a few pieces to this:
Persisting baseline data to disk. Xcode uses a Plist format for this. It would probably make sense to use a compatible format here.
Comparing test results to a previously recorded baseline and triggering a failure if the performance has regressed. Xcode's supposedly triggers a failure if
Average - BaselineAverage
is greater than 0.1 seconds and is more than 10% higher thanBaselineAverage
itself.Providing a mechanism (probably command-line flags?) instructing the runner where to find and create baseline data, and for causing the baseline data to be updated.
The text was updated successfully, but these errors were encountered: