Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove ExVCR #1523

Merged
merged 4 commits into from
Feb 17, 2023
Merged

Remove ExVCR #1523

merged 4 commits into from
Feb 17, 2023

Conversation

thecristen
Copy link
Collaborator

Summary of changes

Asana Ticket: No ticket.

A bit of a reversion of #463. See that PR for background on what the ExVCR package is and why it was added as a tool as part of our Elixir unit testing. TLDR We tried it as a way to record V3 API responses that happened to be made in the course of unit tests, to prevent problems when the underlying data had changed yet a test expects a certain value.

Ultimately it is probably useful for simple tests. However I (any maybe other devs, I don't know), often ran into problems running the test suite where I would get this error message for tests I was not actively working on or editing:

** (ExVCR.RequestNotMatchError) Request did not match with any one in the current cassette: fixture/vcr_cassettes/[name redacted].json.
     Delete the current cassette with [mix vcr.delete] and re-record.

Which is fine when it happens infrequently... but it happens many times.

In a way the error message exposed a gap in my understanding of how I thought this should work - wouldn't it just append new responses to the existing cassette, ignore it in favor of what's already recorded, or overwrite the recording for me?

From what I can tell, the library has some open issues around cassettes (the recorded responses) possibly not being handled optimally:

And I'm a bit wary that ExVCR might have issues with other tools we use in our testing (see issues for Bypass).

I think part of the problem, is that for some of our tests, including a few we were already using ExVCR to assist with, there are many underlying requests made to the V3 API. For example loading a certain page might load a route with data derived from a query for a schedule which in turn might fetch a trip which then we'll follow up with a request for its vehicle. That's a lot of requests to coordinate, but an hour later that same test is going to fetch a different schedule, ... I'm theorizing that this is a use case that's beyond ExVCR's ability to gracefully handle, but is very common across our entire codebase.

There are other ways to improve our tests - ideally unit tests are isolated to the concept that's actually being tested, without being dependent on the details of the underlying data. Maybe the page load test only cares that there's a %Trip{} being passed to the conn, and we shouldn't test for that by looking for a certain headway text/route name/etc on the page response.

So overall I think it was a nice experiment but ExVCR seemed to be getting in the way more than it was helping. Open to counterarguments on that. This PR deletes everything related to our usage of the library (the :exvcr dependency itself, our own exvcr_helpers module, etc).


General checks

  • Are the changes organized into self-contained commits with descriptive and well-formatted commit messages? This is a good practice that can facilitate easier reviews.
  • Testing. Do the changes include relevant passing updates to tests? This includes updating screenshots. Preferably tests are run locally to verify that there are no test failures created by these changes, before opening a PR.
  • Tech debt. Have you checked for tech debt you can address in the area you're working in? This can be a good time to address small issues, or create Asana tickets for larger issues.

- remove dependency from other modules
- refactor tests away from using ExVCRHelpers
technically ran `mix deps.clean --unused --unlock` so it removed a few other unused things too
@thecristen thecristen requested review from a team and amaisano as code owners February 16, 2023 19:04
@thecristen thecristen requested a review from i0sea February 16, 2023 19:04
@amaisano
Copy link
Contributor

I am very likely not a good match for this PR - I've never used this test directly.

Copy link
Contributor

@i0sea i0sea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like TransferTest is erroring now but seems unrelated

@thecristen thecristen merged commit 3b3dc44 into master Feb 17, 2023
@thecristen thecristen deleted the remove_exvcr branch February 17, 2023 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants