Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cannot install python packages in the start task #146

Open
sbirmole211038 opened this issue Jan 12, 2023 · 2 comments
Open

cannot install python packages in the start task #146

sbirmole211038 opened this issue Jan 12, 2023 · 2 comments
Labels
question Question

Comments

@sbirmole211038
Copy link

hey guys, i am currently facing an issue where i have built an etl pipeline in azure data factory where i use blob trigger to activate the custom task on batch account, i have set up start task for the nodes where i install python and pip and other libraries but python and pip gets installed however the packages don't and the code throws an error saying package missing i have tried using requirements.txt and also whl files for the packages but got no luck, can anyone help me with this issue, thank you in advance
image
the image above is an screenshot of start task command
i have checked that pip and python gets installed by logging in the vd(virtual desktop) and trying commands in the cmd however the python package don't get installed, and even after installing them in the vd as per my knowledge it creates new instance every time the job runs hence i cant run the code even if i install it in the vd it shows the same error i am guessing it uses new instance every time

@staer
Copy link
Contributor

staer commented Jan 26, 2023

Have you verified at the end of your start task that the packages are installed correctly? Basically at the end of the task run a `cmd /c "pip list" and that should go to stdout of the start task which you can ensure that the packages are installed. Is the code you are running where the packages are missing part of the start task or a regular scheduled task to run on the VM?

@staer staer added the question Question label Jan 26, 2023
@alfpark
Copy link
Contributor

alfpark commented Jan 30, 2023

I don't think this is a valid start task. You're trying to run multiple cmd /c executions. You either need to chain these together in one cmd /c command or use a windows command/batch script.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Question
Projects
None yet
Development

No branches or pull requests

3 participants