Currently the test suites in swift-corelibs-foundation's and swift-package-manager use different scripts/Xcode projects for testing, but both do the following:

• On OS X, Objective-C XCTest is executed via $(DEVELOPER_DIR)/Platforms/MacOSX.platform/Developer/Library/Xcode/Agents/xctest. This program gathers up all the test cases and runs them.
• On Linux, XCTest is executed via XCTMain(), called from within a main.swift file. Both foundation and package-manager use different systems to ensure this file is not run on Linux.

How will this change as a result of this proposal, if at all?

Ultimately we want swift-corelibs-foundation to be able to build, run, and test itself with swiftpm.

@parkera On Linux? Or on OS X as well?

On Linux - the plan with the core libraries is to provide their features where we do not have the Darwin equivalents.

Does this accommodate projects with an integration test suite that might run more slowly, require a second process (e.g. database), require nonstandard build config, etc.?

Perhaps such tests should not be run as part of the standard package test suite? If so, how would that play out in the proposed directory structure?

How are perf tests handled? Is that:

1. A separate command that should be proposed and discussed totally separately? swift build --perftest?
2. A flag to test? swift build --test --performance-only (or swift build --test --with-performance)
3. Do they run as part of --test?
4. Do they live in the test folder?

Does this accommodate projects with an integration test suite that might run more slowly, require a second process (e.g. database), require nonstandard build config, etc.?

Perhaps such tests should not be run as part of the standard package test suite? If so, how would that play out in the proposed directory structure?

This is a concern of mine as well. Tests that require to be run on a physical device or on a specific platform would also fall into this category.

Although I think it should be sufficient for the first version of this to allow opting out of certain test cases within this directory structure. Maybe by having a static var on XCTestCase subclasses:

class FooDeviceTests: XCTestCase {
  static var excludeFromPackageManagerTests = true
}

(As I commented above) I don't love the idea of coupling swiftpm to corelibs-xctest. Adding something like XCTestCaseProvider.excludeFromPackageManagerTests would then also couple corelibs-xctest to concepts that only exist in swiftpm, which I think is undesirable.

Theoretically, you could exclude certain test cases from swiftpm using existing concepts:

class FooDeviceTests: XCTestCase {
    var allTests: [(String -> () -> Void)] {
        var tests = [(String -> () -> Void)]()
        if !Process.arguments.contains("swift-build") {
            tests.append(("test_onPhysicalDevice", test_onPhysicalDevice))
        }
        return tests
    }
}

Going forward, we should have test frameworks like swift-corelibs-xctest and Spectre determine how to aggregate tests. When left to implement their own aggregation schemes, testing frameworks can innovate: the way JUnit 4 implements aggregation is different from how RSpec uses test filters; both have pros and cons. Asserting too much control in how tests are run in the package manager stifles these frameworks' opportunity to develop new concepts.

The command-line arguments part of the proposal, introduced in 68e618c, is great. It sounds like it would make filtering easy to implement, either in swift-corelibs-xctest or other frameworks:

// swift build -t --exclude broken

// MyTestingFramework parses command-line arguments to
// determine which tests to run or not run.
class MyBrokenTests: MyTestingFrameworkTestCase {
    // These tests don't work yet, so I tag them as 'broken'.
    var tags: [String] { ['broken'] }
    func test_myTestThatDoesntPassYet() { /* ... */ }    
}

Currently the test suites in swift-corelibs-foundation's and swift-package-manager use different scripts/Xcode projects for testing, but both do the following:

• On OS X, Objective-C XCTest is executed via $(DEVELOPER_DIR)/Platforms/MacOSX.platform/Developer/Library/Xcode/Agents/xctest. This program gathers up all the test cases and runs them.
• On Linux, XCTest is executed via XCTMain(), called from within a main.swift file. Both foundation and package-manager use different systems to ensure this file is not run on Linux.

How will this change as a result of this proposal, if at all?

Long term a goal is to remove the divergence here, but there is no timeline for it.

Does this accommodate projects with an integration test suite that might run more slowly, require a second process (e.g. database), require nonstandard build config, etc.?

Perhaps such tests should not be run as part of the standard package test suite? If so, how would that play out in the proposed directory structure?

There is no proposal for this at this time. However, we intend to support other test frameworks that conform to a protocol vended by swiftpm, it seems easy enough to design the protocol with these considerations in mind (it could signify the end of a test with a closure to allow asynchronous running mechanisms, and start-end functions could be provided to allow control of external test processes).

How are perf tests handled? Is that:

1. A separate command that should be proposed and discussed totally separately? swift build --perftest?

From the top of my head, I don’t see why it would need to be separated.

1. A flag to test? swift build --test --performance-only (or swift build --test --with-performance)

We could certainly allow this kind of specification. Currently the proposal says it will accept the name of the test case on the command line and that seems sufficient initially.

1. Do they run as part of --test?

I would think: yes.

1. Do they live in the test folder?

I would think: yes.

Although I think it should be sufficient for the first version of this to allow opting out of certain test cases within this directory structure. Maybe by having a static var on XCTestCase subclasses:

class FooDeviceTests: XCTestCase {
  static var excludeFromPackageManagerTests = true
}

It seems to me that #ifs in the Package.swift would be a better way to exclude targets for different target platforms.

Going forward, we should have test frameworks like swift-corelibs-xctest and Spectre determine how to aggregate tests. When left to implement their own aggregation schemes, testing frameworks can innovate: the way JUnit 4 implements aggregation is different from how RSpec uses test filters; both have pros and cons. Asserting too much control in how tests are run in the package manager stifles these frameworks' opportunity to develop new concepts.

If you feel we are missing any understandings here, please feel free to enlighten us. I am certainly less of an expert in this area than I'd like.

Can we take this discussion over to a mailing list? Presumable the package manager mailing list, since it's not really in the purview of the general swift-evolution list?

If you feel we are missing any understandings here, please feel free to enlighten us.

Oops, sorry if my comment came across as sanctimonious here! 😝

My intention was simply to point out that baking into the package manager the ability to include or exclude performance tests, depending on how that was implemented, may make it harder for test frameworks to build cool new features. I don't think anyone's missing any understandings; based on your comments above I think we're all on the same page! 👍

Can we take this discussion over to a mailing list?

Sounds fine to me!

Perhaps such tests should not be run as part of the standard package test suite? If so, how would that play out in the proposed directory structure?

There is no proposal for this at this time. However, we intend to support other test frameworks that conform to a protocol vended by swiftpm, it seems easy enough to design the protocol with these considerations in mind

I’m less concerned about the details of how the test framework is invoked, and more about being about to put all the things in Tests that belong there, without worrying about that tripping up SwiftPM. That means:

1. being able to have multiple modules under Tests, some of which are not run by SwiftPM by default,
2. having the ability to run those non-default test modules via SwiftPM, and
3. having the ability to put directory structures under Tests that SwiftPM does not know how to deal with at all, and run through some different mechanism altogether.

It’s unclear to me from the wording of the current proposal whether this is covered. It sounds like SwiftPM would find everything named *.swift (or maybe *Test.swift) under the test directory and run it as a test?

If anyone has any further comments then we should take them to swift-build-dev@swift.org. I will finish by replying to the last two comments. The intent for comments here is more to correct things, it was my mistake to direct comments here initially.

Oops, sorry if my comment came across as sanctimonious here! 😝

Not at all, I seek merely to ensure we have considered all aspects.

I’m less concerned about the details of how the test framework is invoked, and more about being about to put all the things in Tests that belong there, without worrying about that tripping up SwiftPM. That means:

1. being able to have multiple modules under Tests, some of which are not run by SwiftPM by default,
2. having the ability to run those non-default test modules via SwiftPM, and
3. having the ability to put directory structures under Tests that SwiftPM does not know how to deal with at all, and run through some different mechanism altogether.
   It’s unclear to me from the wording of the current proposal whether this is covered. It sounds like SwiftPM would find everything named *.swift (or maybe *Test.swift) under the test directory and run it as a test?

Excluding targets will be possible as a result of future work on Package.swift.