-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extremely slow in jest testing env #50
Comments
Hi @ms88privat, Would you please tell me how long it takes to run your test? Yeah, being a bit slow is totally expected behavior, as it needs to launch browser, load the page, go through steps, capture screenshot, store it and compare it. All these processes take a significant amount of time. I've already tried to optimised this process by storing the captured image in memory instead of hard disk and using the fastest image comparison library. I am happy to help more when I know more details about your issue. Nima |
@ms88privat : Please let me know node and jest version you are running your test on |
Are you using jest as the test runner? I'm on node 7.6.0 and jest 20.0.4. But I did some research and it is just about the file system access. I didn't use your library for it and I didn't use Let me try upgrading jest. 3 sec per test would be fine, but 20+ sec is not :) |
Upgrading jest didn't solve it. |
So it isn't related to your library. I'm just curios if it is working fast enough on your side. But I didn't believed so because of the extreme timeout option you did set in your example. That's why I opened this issue. But if it is working on your side, feel free to close this one |
Hi @ms88privat: Thanks for providing more info. I am wondering why on jest env should cause this issue for you. As I said I am not seeing this issue in my local env and docker. I will close this issue but lets keep posting here till we understand what is the root cause as it might be the case for other peoples. I will also update the extreme timeout on jest example. That was my arbitrary number when I was developing this package. Nima |
I see, it looks like the resolution makes the difference. Please try using "1920x1080" instead of your default 800x600 or so. |
What is the image size you getting by I am actually trying Can you try it with |
@ms88privat : Would you please run you test with
this should give you the timestamp of each step execution. Please post the outcome here |
So it looks like the test itself is "fast" but the whole test case did take 30s. It is really slow after the tab closes and before the image comparison (confirms my suspicions about the file reading step). The image itself is 26KB. I created a fresh app with "create-react-app". Node and everything else is up2date. My test setup looks like this: import Differencify from 'differencify';
const differencify = new Differencify({ debug: true });
describe('tests differencify', () => {
it('validate github page appear correctly', async () => {
await differencify
.init()
.resize({ width: 1600, height: 1200 })
.goto('http://localhost:3000/')
.capture()
.toMatchSnapshot()
.close()
.end();
});
}); Here the debug test output: |
Just tested the same code on coworker's computer. Same result :( But yeah, its a problem with Jest somehow... Just reading an image files does also not work, without using your library at all. Thanks anyway! 👍 |
I just saw the same pattern in your test-fail.png in the readme? Only one test, which takes 17s, but the whole test suite took almost 40s? |
Ok, Now I see where does the problem come from. It is actually the image comparison part that takes a long time. Just tried to make all my tests fail locally and yeah, it takes a long time for every single test to be compared with the original screenshot. I already have some ideas around this. I am using Jimp as the image processing lib which uses pixelmatch for image comparison. Jimp also does perceptual hash comparison of generated images which I guess will be the most time-consuming bit. I will do a benchmark on this and will update the result here. |
Ok, here is my the result on running first test of jest-example.test.js Disabling perceptual hash comparison Disabling perceptual hash comparison and no writing diff image to disk So basically it is writing to disk that takes a long time in case of failure. As you might already know if you pass |
As an improvement, I am happy to put
|
Meantime we can look into how to improve image write to disk functionality. I just played with some Jimp config and I managed to reduce |
Thanks for further investigating this. But for me, if I just import |
@ms88privat : Interesting! Would you be able to verify the same behavior with pixelmatch using with/without Jest? |
pixelmatch is fast on both. But for pixelmatch to work I need to read the images to compare, which is slow again on jest... |
@ms88privat : Ok then you confirm that it is actually reading image from the disk is the slow path! This is pretty odd to me. Image size on disk is around 27kb which I expect |
maybe related to this? jestjs/jest#2925 |
Why is it so slow in jest? You can also see it in the examples, which had to increase the testing timeout to be able to finish the test. Totally unusable :(
I tried to pinpoint it and it is the fs.readStream it looks like. I will investigate further later on. Just opened this issue as a reference.
The text was updated successfully, but these errors were encountered: