Skip to content

Latest commit

 

History

History
135 lines (108 loc) · 8.52 KB

ui-tests.md

File metadata and controls

135 lines (108 loc) · 8.52 KB
Raw

UI Tests

WordPress for iOS has UI acceptance tests for critical user flows through the app, such as login, signup, and publishing a post. The tests use mocked network requests with WireMock, defined in the API-Mocks folder in the project's root.

Test Coverage

The following flows are covered/planned to be covered by UI tests. Tests that are covered will be checked.

  1. Unified Login
    • WPCom Account:
      • Log in and log out using WPCom Account
      • Log in and log out Self Hosted Site
      • Invalid Password WPCom Account
      • Add Self Hosted Site after WPCom Login
    • Apple/Google:
      • Log in using Apple Account
      • Log in using Google Account
    • iCloud Keychain:
      • Log in using credentials saved in Keychain
    • Magic Link:
      • Email Magic Link Login
  2. Sign Up:
    • Email Sign Up - Create New Site
    • Email Sign Up - Link to Self-Hosted Site
  3. NUX
    • Create New Site
    • Set Up Blogging Reminders on First Post
  4. Posting/Editing/Gutenberg
    • Post:
      • Publish Text Post
      • Publish Basic Public Post with Category and Tag
      • Add and Remove Featured Image
      • Add Gallery Block
      • Add Media Blocks (Image, Video and Audio)
      • Create Scheduled Post
    • Pages:
      • Create Page from Layout
      • Create Blank Page
  5. Notifications
    • View Notification - Comment, Follow and Like Notifications
    • Reply from Notification
    • Like Notification
  6. Blogging Reminders
    • Set Up New Blogging Reminders from Post Publish Prompt
    • Set Up Scheduled Story Post (iPhone Only)
  7. Stats
    • Insights Stats Load Properly
    • Years Stats Load Properly
  8. Reader
    • View Last Post
    • View Last Post in Safari
    • Add Comment to Post
    • Follow New Topic on Discover Tab
    • Save a Post
    • Like a Post
  9. Jetpack Settings
    • Open and View Jetpack Settings Options
    • Search and View a Plugin
  10. View Site
    • View Site from My Site Screen
    • Update Site and Validate Changes
  11. Dashboard (Jetpack Only)
    • Free to Paid Plans Card
    • Pages Card Header Navigation
    • Activity Log Card Header Navigation
  12. Navigation
    • Load People Screen
    • Tab Bar Navigation (Reader and Notification tabs)
    • Domains Navigation (Jetpack Only)
  13. Support Screen/Help
    • Support Forums Loaded during Login
    • Contact Us Loaded during Login

Running tests

Note that due to the mock server setup, tests cannot be run on physical devices right now.

  1. Follow the build instructions (steps 1-5) to clone the project, install the dependencies, and open the project in Xcode.
  2. rake mocks to start a local mock server.
  3. With the Jetpack scheme selected in Xcode, open the Test Navigator and select the JetpackUITests test plan.
  4. Run the tests on a simulator.

Adding tests

When adding a new UI test, consider:

  • Whether to test a user flow (to accomplish a task or goal) or a specific feature (e.g. boundary testing).
  • What screens are being tested (defined as screen objects in Screens).
  • Are there repeated flows across tests (defined in Flows).
  • What network requests are made during the test (defined in API-Mocks).

Tests classes are grouped together in Tests

When you add a new test, you may need to add new screens, methods, and flows. We use page objects and method chaining for clarity in our tests. Wherever possible, use an existing accessibilityIdentifier (or add one to the app) instead of a string to select a UI element on the screen. This ensures tests can be run regardless of the device language.

Design Guidelines

Naming convention

  • Use unique names with namespaces for accessibility identifiers, e.g. sidebar_reader to make it easy to find elements with a single query.
  • When creating new tests, use the testActionFeature format for the test name to make it easier to see what the test is for: e.g. testCreateScheduledPost()
  • When creating new methods, use the actionObject format: e.g. closePostSettings()
  • For assert methods, use the verifyWhatToVerify format: e.g. verifyPostExists()
  • Note that there’s a common global method assertScreenIsLoaded() that can be used to assert all screens

Performance

  • Use firstMatch in queries when you expect only one element to be visible. It's a ~10x improvement (but it's not as important in absolute numbers)
  • Reduce the number of queries. It's about 20-50% faster to find an element by a unique accessibility identifier with a single query to first find its parent and then the element.
  • Consolidate related tests into a single test to eliminate the unnecessary app startups that take a large portion of test time

Mocking

Adding or updating network mocks

When you add a test (or when the app changes), the request definitions for WireMock need to be updated in API-Mocks/. You can read WireMock’s documentation here.

If you are unsure what network requests need to be mocked for a test, an easy way to find out is to run the app through Proxyman or Charles Proxy and observe the required requests.

Currently, the project does not apply strict mock matching criteria, this means that if there are unmatched requests that are not being used by the test itself. The test should still work although errors like this can be seen in the logs:

unmatched request

When adding a new test however, it is recommended that all requests are matched so that the logs are not further cluttered with Request was not matched like in the above screenshot.

Using stateful behavior for mocks

  1. Add the new scenario in scenarios.json
  2. Fetch and reset scenario during SetUp() in the test class containing the test, e.g. seen on Notification Test
  3. Update JSON mappings to contain the following 3 new attributes, scenarioName, requiredScenarioState and newScenarioState, and the response matching the state of the scenario, e.g. seen on Notification Test

Tips and tricks on using mocks

  • When getting the same request with the same header but a different request body to return different responses, experiment with using different matchers. From some experimenting, I would recommend using matchesJsonPath which is used to differentiate Create Page and Create Post.
  • Use verbose to debug errors, this can be updated by adding the verbose parameter when starting the WireMock server (don’t forget the slash to not break the command).
  • If there are no errors on the console, but the mocks don’t work as expected, check out the app’s logs for errors. Sometimes it could be that the JSON mapping is not parsed correctly.