Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite screenshot tests to use new syntax #6300

Merged

Conversation

rablador
Copy link
Contributor

@rablador rablador commented May 29, 2024

We should update our screenshot tests to use the new syntax we use for UI tests.

This PR also adds replaces the location selection screenshot with a custom list selection screenshot, and includes https://linear.app/mullvad/issue/IOS-684/update-screenshots-to-use-pq-tunnels.


This change is Reviewable

@rablador rablador added the iOS Issues related to iOS label May 29, 2024
Copy link

linear bot commented May 29, 2024

@rablador rablador force-pushed the update-location-view-screenshot-automation-ios-645 branch from c55f8a7 to 640ead4 Compare May 29, 2024 15:33
@rablador rablador self-assigned this May 30, 2024
@rablador rablador force-pushed the update-location-view-screenshot-automation-ios-645 branch 2 times, most recently from 3cd581a to bbc3f96 Compare May 30, 2024 08:37
@rablador rablador marked this pull request as ready for review May 30, 2024 08:42
@rablador rablador force-pushed the update-location-view-screenshot-automation-ios-645 branch 2 times, most recently from 7dd44f5 to 880b1f3 Compare May 30, 2024 09:52
Copy link
Collaborator

@niklasberglund niklasberglund left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 17 files reviewed, 4 unresolved discussions (waiting on @rablador)


ios/MullvadVPNUITests/Pages/EditCustomListLocationsPage.swift line 61 at r1 (raw file):

        }

        app.navigationBars[navigationBarName].buttons.firstMatch.tap()

This class EditCustomListLocationsPage is now for both add and edit, so the action context is required. Maybe it should be broken out into two page classes? There are a few differences between the edit and add views even though they also have much in common.


ios/MullvadVPNUITests/Screenshots/ScreenshotTests.swift line 37 at r1 (raw file):

    }

    func testTakeScreenshotOfSecuredConnection() throws {

Should this screenshot still be captured? My interpretation of https://linear.app/mullvad/issue/IOS-684/update-screenshots-to-use-pq-tunnels is that this screenshot should be replaced by the screenshot with a quantum secured connection. It's nice to still have this around, but the drawback is that it opens up for possible mistakes if all screenshots should be used except this one.


ios/MullvadVPNUITests/Screenshots/ScreenshotTests.swift line 112 at r1 (raw file):

        CustomListPage(app)
            .renameCustomList(name: "Low latency locations")

nit: would be nice as a constant since it is reused


ios/TestPlans/MullvadVPNScreenshots.xctestplan line 31 at r1 (raw file):

        "SettingsMigrationTests",
        "SettingsTests"
      ],

Make sure to also uncheck "Automatically include new tests" for the ScreenshotTests test plan, otherwise new tests will be included in the plan.

image.png

@rablador rablador force-pushed the update-location-view-screenshot-automation-ios-645 branch from 880b1f3 to cc8d80b Compare June 3, 2024 07:44
Copy link
Contributor Author

@rablador rablador left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 18 files reviewed, 3 unresolved discussions (waiting on @niklasberglund)


ios/MullvadVPNUITests/Pages/EditCustomListLocationsPage.swift line 61 at r1 (raw file):

Previously, niklasberglund (Niklas Berglund) wrote…

This class EditCustomListLocationsPage is now for both add and edit, so the action context is required. Maybe it should be broken out into two page classes? There are a few differences between the edit and add views even though they also have much in common.

Right now the difference is basically the navigation title. We don't use inheritance much in the app, but I think a new AddCustomListLocationsPage could simply inherit EditCustomListLocationsPage and override the back tap function.


ios/MullvadVPNUITests/Screenshots/ScreenshotTests.swift line 37 at r1 (raw file):

Previously, niklasberglund (Niklas Berglund) wrote…

Should this screenshot still be captured? My interpretation of https://linear.app/mullvad/issue/IOS-684/update-screenshots-to-use-pq-tunnels is that this screenshot should be replaced by the screenshot with a quantum secured connection. It's nice to still have this around, but the drawback is that it opens up for possible mistakes if all screenshots should be used except this one.

Yeah, you're probably right. I'll remove this test.


ios/MullvadVPNUITests/Screenshots/ScreenshotTests.swift line 112 at r1 (raw file):

Previously, niklasberglund (Niklas Berglund) wrote…

nit: would be nice as a constant since it is reused

Yep


ios/TestPlans/MullvadVPNScreenshots.xctestplan line 31 at r1 (raw file):

Previously, niklasberglund (Niklas Berglund) wrote…

Make sure to also uncheck "Automatically include new tests" for the ScreenshotTests test plan, otherwise new tests will be included in the plan.

image.png

Done.

Copy link
Collaborator

@niklasberglund niklasberglund left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: 0 of 18 files reviewed, 3 unresolved discussions

Copy link
Collaborator

@niklasberglund niklasberglund left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 18 files reviewed, 4 unresolved discussions (waiting on @rablador)


ios/MullvadVPNUITests/Screenshots/ScreenshotTests.swift line 33 at r2 (raw file):

        if TunnelControlPage(app).connectionIsSecured {
            TunnelControlPage(app)
                .tapDisconnectButton()

I think it would be easier to follow the code if this reset is done in testTakeScreenshotOfQuantumSecuredConnection teardown.

Copy link
Contributor Author

@rablador rablador left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 18 files reviewed, 4 unresolved discussions (waiting on @niklasberglund)


ios/MullvadVPNUITests/Screenshots/ScreenshotTests.swift line 33 at r2 (raw file):

Previously, niklasberglund (Niklas Berglund) wrote…

I think it would be easier to follow the code if this reset is done in testTakeScreenshotOfQuantumSecuredConnection teardown.

I agree, but that also requires each test to go back to the home screen when they're done. On startup that's where we begin each time. The real "problem" here is that the tests aren't completely stand-alone, but it felt unnecessarily thorough/slow to uninstall the app between each of them.

Copy link
Collaborator

@niklasberglund niklasberglund left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 18 files reviewed, 4 unresolved discussions (waiting on @rablador)


ios/MullvadVPNUITests/Screenshots/ScreenshotTests.swift line 33 at r2 (raw file):

Previously, rablador (Jon Petersson) wrote…

I agree, but that also requires each test to go back to the home screen when they're done. On startup that's where we begin each time. The real "problem" here is that the tests aren't completely stand-alone, but it felt unnecessarily thorough/slow to uninstall the app between each of them.

It looks like testTakeScreenshotOfQuantumSecuredConnection is the only test case connecting to a relay, so this code will only run after testTakeScreenshotOfQuantumSecuredConnection, so it's kind of its teardown but instead it is in the next test cases' setup.

Copy link
Contributor Author

@rablador rablador left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 18 files reviewed, 4 unresolved discussions (waiting on @niklasberglund)


ios/MullvadVPNUITests/Screenshots/ScreenshotTests.swift line 33 at r2 (raw file):

Previously, niklasberglund (Niklas Berglund) wrote…

It looks like testTakeScreenshotOfQuantumSecuredConnection is the only test case connecting to a relay, so this code will only run after testTakeScreenshotOfQuantumSecuredConnection, so it's kind of its teardown but instead it is in the next test cases' setup.

Discussed offline. Will inherit from LoggedInWithTimeUITestCase to make tests independent of each other (results in a logout if logged in before each test).

@rablador rablador force-pushed the update-location-view-screenshot-automation-ios-645 branch from cc8d80b to 3f19850 Compare June 3, 2024 11:10
Copy link
Collaborator

@niklasberglund niklasberglund left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: 0 of 18 files reviewed, 4 unresolved discussions

@buggmagnet buggmagnet requested a review from niklasberglund June 3, 2024 13:58
Copy link
Contributor

@buggmagnet buggmagnet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 12 of 17 files at r1, 5 of 6 files at r2, all commit messages.
Reviewable status: 17 of 18 files reviewed, 5 unresolved discussions (waiting on @niklasberglund and @rablador)


ios/MullvadVPNUITests/Screenshots/ScreenshotTests.swift line 25 at r3 (raw file):

    func testTakeScreenshotOfQuantumSecuredConnection() throws {
        // We can't close banners in the screenshot tests due to how the NotificationController view

Is it because the notification is not accessible ?
Can we make it accessible in that case ? That seems like a silly limitation to have.

Copy link
Contributor Author

@rablador rablador left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 17 of 18 files reviewed, 5 unresolved discussions (waiting on @buggmagnet and @niklasberglund)


ios/MullvadVPNUITests/Screenshots/ScreenshotTests.swift line 25 at r3 (raw file):

Previously, buggmagnet wrote…

Is it because the notification is not accessible ?
Can we make it accessible in that case ? That seems like a silly limitation to have.

It's indeed silly. There's a PR for it here: https://linear.app/mullvad/issue/IOS-704/make-notifications-accessible

Copy link
Contributor

@buggmagnet buggmagnet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 1 of 1 files at r3.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on @niklasberglund)

@buggmagnet buggmagnet force-pushed the update-location-view-screenshot-automation-ios-645 branch from 3f19850 to b875c1b Compare June 4, 2024 06:53
@buggmagnet buggmagnet merged commit 5fc0b40 into main Jun 4, 2024
5 of 7 checks passed
@buggmagnet buggmagnet deleted the update-location-view-screenshot-automation-ios-645 branch June 4, 2024 06:57
Copy link

github-actions bot commented Jun 4, 2024

🚨 End to end tests failed. Please check the failed workflow run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
iOS Issues related to iOS
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants