[SR-9222] XCTest performance tests are always much slower on the first measurement

[swift-ci]

Previous ID	SR-9222
Radar	None
Original Reporter	bnut (JIRA User)
Type	Bug

Attachment: Download

Environment

Xcode 10.1 (10B61)
MacBook Pro (15-inch, 2016)
2.9 GHz Intel Core i7
16 GB 2133 MHz LPDDR3

Additional Detail from JIRA

Votes	0
Component/s	XCTest
Labels	Bug
Assignee	None
Priority	Medium

md5: d535dadca1b8fdfb15ae346fea01370a

Issue Description:

Using the measure block in XCTest always results in a significant overhead on the first measurement.

import XCTest
@testable import xctperf_test


final class xctperf_testTests: XCTestCase {
    func testExample() {
        measure {
            for _ in 0 ... 10_000 {
                XCTAssertEqual("abc", "abc")
            }
        }
    }


    static var allTests = [
        ("testExample", testExample),
    ]
}

[swift-ci]

Comment by Andrew Bennett (JIRA)

@briancroom this might be something you're interested in, based on other similar tickets.

[briancroom]

Hi bnut (JIRA User), based on the screenshot, it looks like you are seeing this problem when using Xcode’s XCTest, rather than swift-corelibs-xctest. Could you confirm?

[swift-ci]

Comment by Andrew Bennett (JIRA)

Yes, that's correct @briancroom - sorry if you were the wrong person to ping.

I created the target with SwiftPM using swift package init which adds some linux support, and then using swift package generate-xcodeproj. I don't think this would lead it to use swift-corelibs-xctest though.

I've verified this by looking at the `XCTest` module, which contains:

public var XCT_UI_TESTING_AVAILABLE: Int32 { get }
//
//  Copyright (c) 2014-2015 Apple Inc. All rights reserved.
//

Which doesn't seem to be in the corelibs version.

[briancroom]

Hey bnut (JIRA User), sorry for the delay. No worries, I'm a fine person to ask about this!

Yeah, swift-corelibs-xctest is generally only used on non-Apple platforms, so this would be Xcode's version of XCTest that you're using here. I haven't seen this particular behavior before, but it does seem like a bug. Would you be willing to file a Radar about this on https://bugreport.apple.com and attach an Xcode project which reproduces the problem?

[swift-ci]

Comment by Andrew Bennett (JIRA)

Thanks for the feedback @briancroom, I've created a report at https://bugreport.apple.com/web/?problemID=46153989

Please let me know if you need more information.