Skip to content
This repository has been archived by the owner on Jan 4, 2022. It is now read-only.

Problem executing my jobs to cluster dbx deploy --jobs=project_name fails #80

Open
echarso opened this issue Nov 17, 2021 · 6 comments
Open

Problem executing my jobs to cluster dbx deploy --jobs=project_name fails #80

echarso opened this issue Nov 17, 2021 · 6 comments

Comments

@echarso
Copy link

echarso commented Nov 17, 2021 •
edited
Loading

I am keep getting this error when using this project.

response = verify_rest_response(response, endpoint)
File "/Users/charalampossouris/miniforge3/lib/python3.9/site-packages/mlflow/utils/rest_utils.py", line 137, in verify_rest_response
raise MlflowException("%s. Response body: '%s'" % (base_msg, response.text))
mlflow.exceptions.MlflowException: API request to endpoint was successful but the response body was not in a valid JSON format. Response body: '<!doctype html><title>Databricks - Sign In</title>

<script src="/login/login.01615f82.js"></script>'

I am almost confident that my DATABRICKS-HOST and DATABRICKS-TOKEN are correct and placed in the right places.

I will continue debugging it but I was hoping for some hints :)

When executing "databricks workspace list" I can see the workspaces, which indicates to me that my credentials are correctly placed.
@renardeinside
Copy link
Contributor

hi @echarso ,
I guess the root cause is in the host path - please take a look at this section.

@echarso
Copy link
Author

echarso commented Nov 17, 2021 •
edited
Loading

Thanks @renardeinside for your answer. I just tried it and didn't work :(
Should I have a cluster in my deployment file ?
I am using the standard deployment file, the one provided when doing pip install

@echarso
Copy link
Author

echarso commented Nov 17, 2021

I am using Databricks in AWS and I have Mac M1...

@renardeinside
Copy link
Contributor

renardeinside commented Nov 17, 2021 •
edited
Loading

Let's try to debug this on your local machine first.

Could you please provide the following:

  • Databricks CLI version: databricks --version
  • dbx CLI version: dbx --version
  • MLflow version: mlflow --version

Also, please provide the full error log, the truncated part unfortunately is not providing enough info.

@echarso
Copy link
Author

echarso commented Nov 18, 2021

Hi again!
Thanks for taking time on this :)
Databricks Version 0.16.2
DataBricks eXtensions aka dbx, version ~> 0.2.1
mlflow, version 1.14.1

I must admit that now it works from the githubs actions .
Not locally though.

[dbx][2021-11-18 11:01:20.404] Starting new deployment for environment default
[dbx][2021-11-18 11:01:20.404] Using configuration from the environment variables
Traceback (most recent call last):
File "/Users/-/miniforge3/bin/dbx", line 8, in
sys.exit(cli())
File "/Users/-/miniforge3/lib/python3.9/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/Users/-/miniforge3/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Users/-/miniforge3/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/-/miniforge3/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/-/miniforge3/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Users/-/miniforge3/lib/python3.9/site-packages/dbx/commands/deploy.py", line 122, in deploy
api_client = prepare_environment(environment)
File "/Users/-/miniforge3/lib/python3.9/site-packages/dbx/utils/common.py", line 334, in prepare_environment
experiment: Optional[mlflow.entities.Experiment] = mlflow.get_experiment_by_name(environment_data["workspace_dir"])
File "/Users/-/miniforge3/lib/python3.9/site-packages/mlflow/tracking/fluent.py", line 795, in get_experiment_by_name
return MlflowClient().get_experiment_by_name(name)
File "/Users/-/miniforge3/lib/python3.9/site-packages/mlflow/tracking/client.py", line 419, in get_experiment_by_name
return self._tracking_client.get_experiment_by_name(name)
File "/Users/-/miniforge3/lib/python3.9/site-packages/mlflow/tracking/_tracking_service/client.py", line 128, in get_experiment_by_name
return self.store.get_experiment_by_name(name)
File "/Users/-/miniforge3/lib/python3.9/site-packages/mlflow/store/tracking/rest_store.py", line 319, in get_experiment_by_name
raise e
File "/Users/-/miniforge3/lib/python3.9/site-packages/mlflow/store/tracking/rest_store.py", line 306, in get_experiment_by_name
response_proto = self._call_endpoint(GetExperimentByName, req_body)
File "/Users/-/miniforge3/lib/python3.9/site-packages/mlflow/store/tracking/rest_store.py", line 55, in _call_endpoint
return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
File "/Users/-/miniforge3/lib/python3.9/site-packages/mlflow/utils/rest_utils.py", line 170, in call_endpoint
response = verify_rest_response(response, endpoint)
File "/Users/-/miniforge3/lib/python3.9/site-packages/mlflow/utils/rest_utils.py", line 137, in verify_rest_response
raise MlflowException("%s. Response body: '%s'" % (base_msg, response.text))
mlflow.exceptions.MlflowException: API request to endpoint was successful but the response body was not in a valid JSON format. Response body: '<!doctype html><title>Databricks - Sign In</title>

<script src="/login/login.01615f82.js"></script>'

@renardeinside
Copy link
Contributor

hi @echarso ,
please try out the following:

  1. update to dbx 0.2.2
  2. verify that your provided hostname are in variables: DATABRICKS_HOST & DATABRICKS_TOKEN inside your CI pipeline and your local environment.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@renardeinside @echarso