Open Radar

 
Summary:
UI tests often fail on iOS Simulator because XCUIApplication sometimes fails to terminate between test cases or if -terminate method is called in testcase code, setUp or tearDown. Also quite often tests fail to take snapshot or load accessibility.

The more tests are run in sequence, the greater is probability of false negatives.

Please see log files attached with 2 different reasons of failures.

Steps to Reproduce:
1. Install Xcode 7.3, iOS Simulator 9.2.
2. Create new iOS project with "Hello World" iOS app (single view app is enough) with UI tests enabled.
3. Add sample test code:

- (void)testExample {
    for (int i = 0; i < 500; i++)
    {
        NSLog(@"%@", @(i));
        XCUIApplication* app = [XCUIApplication new];

        [app launch];
        XCTAssertTrue(app.exists);

        [app terminate];
        XCTAssertFalse(app.exists);
    }
}

4. Run tests.
5. Wait until tests fail and observe the result (usually it doesn't take more than 2-3 minutes, sometimes up to 5-15 minutes).

Expected Results:
Tests should not fail.
The app should always be terminated and launched correctly.

Actual Results:
Tests randomly fail because the XCUIApplication state is reported to be incorrect.

Sample log:

2016-04-05 11:16:26.893 XCTRunner[2361:104360] 32
    t =   117.44s     Launch com.company.ttt
    t =   119.13s         Waiting for accessibility to load
    t =   120.79s         Wait for app to idle
    t =   120.94s     Snapshot accessibility hierarchy for com.company.ttt
    t =   120.96s     Terminate <XCUIApplicationProcess: 0x7fb4dd8130f0 com.company.ttt (2487)>
2016-04-05 11:16:30.566 XCTRunner[2361:104360] 33
    t =   121.11s     Launch com.company.ttt
    t =   122.80s         Waiting for accessibility to load
    t =   124.43s         Wait for app to idle
    t =   124.57s     Snapshot accessibility hierarchy for com.company.ttt
    t =   124.60s     Terminate <XCUIApplicationProcess: 0x7fb4dd910ea0 com.company.ttt (2501)>
    t =   184.70s         Assertion Failure: UI Testing Failure - App state for <XCUIApplicationProcess: 0x7fb4dd910ea0 com.company.ttt (2501)> is XCApplicationStateRunningActive (3), still not XCApplicationStateNotRunning (1)
/Users/sdunets/dev/galaxy-ios-app-copy/tttUITests/tttUITests.m:37: error: -[tttUITests testExample] : UI Testing Failure - App state for <XCUIApplicationProcess: 0x7fb4dd910ea0 com.company.ttt (2501)> is XCApplicationStateRunningActive (3), still not XCApplicationStateNotRunning (1)
    t =   184.70s     Tear Down
Test Case '-[tttUITests testExample]' failed (184.705 seconds).
Test Suite 'tttUITests' failed at 2016-04-05 11:17:34.164.
	 Executed 1 test, with 1 failure (0 unexpected) in 184.705 (184.709) seconds


Test session log:
	/Users/sdunets/Library/Developer/Xcode/DerivedData/ttt-haconrdeuzthclatsvllzbadkwlu/Logs/Test/D7810813-9A4E-4042-B417-185C35220B03/Session-2016-04-05_11:14:15-BONZJR.log

Test Suite 'tttUITests.xctest' failed at 2016-04-05 11:17:34.165.
	 Executed 1 test, with 1 failure (0 unexpected) in 184.705 (184.711) seconds
Test Suite 'All tests' failed at 2016-04-05 11:17:34.165.
	 Executed 1 test, with 1 failure (0 unexpected) in 184.705 (184.713) seconds





Version:
Version 7.3 (7D175), OS X 10.11.3 (15D21)

Notes:


Configuration:
Xcode 7.3, any iOS Simulator with iOS 9.2, 9.3