-
|
Do you guys use some sort of docker container to run the spark related stuff. I was thinking of running a docker container with jupyter notebooks, have the sqlglot repo mounted and run the codebase there for testing. Or do you have another approach that is more handy? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 1 reply
-
|
we just run it using pyspark and virtual env, so no need for a container |
Beta Was this translation helpful? Give feedback.
-
|
But in the build agent those spark tests are skipped right (85 tests)? So you would only run those tests locally, by having spark installed along with java on your local environment? |
Beta Was this translation helpful? Give feedback.
-
|
the tests are only skipped if you pass SKIP_INTEGRATION=1 |
Beta Was this translation helpful? Give feedback.
we just run it using pyspark and virtual env, so no need for a container