how to update athena results through requests package

[megetron3]

hey all,

on a local moto_proxy server i try to send some post requests to update athena results.
to do so using moto examples:

executing this without success

requests.post(
    "http://motoapi.amazonaws.com/moto-api/static/athena/query-results"
)  

returns:
INFO 2024-02-26 16:17:26,091 - POST http://motoapi.amazonaws.comhttp://motoapi.amazonaws.com/moto-api/static/athena/query-results
ERROR 2024-02-26 16:17:26,091 - 'NoneType' object is not callable
127.0.0.1 - - [26/Feb/2024 16:17:26] code 502, message Bad Gateway
127.0.0.1 - - [26/Feb/2024 16:17:26] "POST http://motoapi.amazonaws.com/moto-api/static/athena/query-results HTTP/1.1" 502 -

and in verbose mode:
INFO 2024-02-26 16:24:06,213 - POST http://motoapi.amazonaws.comhttp://motoapi.amazonaws.com/moto-api/static/athena/query-results
DEBUG 2024-02-26 16:24:06,213 - 	body	b''
DEBUG 2024-02-26 16:24:06,213 - 	headers	{'Host': 'motoapi.amazonaws.com', 'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '0'}
ERROR 2024-02-26 16:24:06,213 - 'NoneType' object is not callable
127.0.0.1 - - [26/Feb/2024 16:24:06] code 502, message Bad Gateway




requests.post(
    "http://localhost:5005/moto-api/static/athena/query-results",
)  

returns:
INFO 2024-02-26 16:18:38,812 - POST http://localhost:5005/moto-api/static/athena/query-results
ERROR 2024-02-26 16:18:38,812 - 'AWSPreparedRequest' object has no attribute 'environ'
127.0.0.1 - - [26/Feb/2024 16:18:38] code 502, message Bad Gateway
127.0.0.1 - - [26/Feb/2024 16:18:38] "POST /moto-api/static/athena/query-results HTTP/1.1" 502 -

what do i do wrong?

[bblommers]

Hi @megetron3!

Which version of Moto are you on?

I think I resolved this bug a few days ago. There hasn't been a proper release yet, but the latest dev release should work just fine:
https://pypi.org/project/moto/5.0.3.dev23/

[megetron3]

thank you very much! seems like the upgrade didnt happend. once upgrading it solved the issue for me:)

[bblommers]

That's great - happy to hear it works now. Thank you for letting me know!

[megetron3]

@bblommers thanks for the quick response.
upgrading to the dev version still gives the same result.

please note that i call moto proxy. if switching to a moto server everything works just fine and requests.post retuens the expeced 201 status code.

does the moto proxy should support athena results api?

[bblommers]

It should work with the Proxy, but I don't think there are any tests for that particular scenario. So it may very well just be broken. I'll try to have a look later today.

[bblommers]

I cannot reproduce this unfortunately. Note that the endpoint should be http://motoapi.amazonaws.com/moto-api/static/athena/query-results for it to work with the Proxy - sending a request to localhost:5005/moto-api/... via the Proxy will not work.

See this PR as an example: https://github.com/getmoto/moto/pull/7395/files
The tests in test_athena_server_api.py verify the full flow (starting an execution, posting data, verifying that data is returned via get_query_results, etc), and those tests also pass when running against the proxy. (I just had to modify them slightly to ensure that the HTTP requests would use the Proxy.)

[megetron3]

please note that your tests has the @mock_aws decorator, which actually will not reach to the proxy server.
adding this decorator to my tests will make them pass without error but the proxy doesnt receive the request at all.

hopefully, in order reproduce this maybe try to raise the moto proxy server:
moto_proxy -H 0.0.0.0

and execute the following pytest test against the proxy server:

def test_athena():
    expected_results = {
        "account_id": "123456789012",  # This is the default - can be omitted
        "region": "us-east-1",  # This is the default - can be omitted
        "results": [
            {
                "rows": [{"Data": [{"VarCharValue": "1"}]}],
                "column_info": [{
                    "CatalogName": "string",
                    "SchemaName": "string",
                    "TableName": "string",
                    "Name": "string",
                    "Label": "string",
                    "Type": "string",
                    "Precision": 123,
                    "Scale": 123,
                    "Nullable": "NOT_NULL",
                    "CaseSensitive": True,
                }],
            },
        ],
    }
    proxies = {"http": "http://localhost:5005", "https": "http://localhost:5005"}

    config_url = "http://motoapi.amazonaws.com/moto-api/static/athena/query-results"
    resp = requests.post(config_url, json=expected_results, proxies=proxies, verify=False)

    print("resp", resp.text) # <= returns 502 error . the moto proxy server logs should show the exact error 
    assert resp.status_code == 201 # Fails

after exeuting this test, the request seems to be seen in the proxy logs as expected but for some reason the request url in the logs has a domain + the url i sent.
so instead of seeing in logs a post to
http://motoapi.amazonaws.com/moto-api/static/athena/query-results
i get a post to:
http://motoapi.amazonaws.comhttp://motoapi.amazonaws.com/moto-api/static/athena/query-results

INFO 2024-02-27 19:23:47,444 - POST http://motoapi.amazonaws.comhttp://motoapi.amazonaws.com/moto-api/static/athena/query-results
ERROR 2024-02-27 19:23:47,445 - 'NoneType' object is not callable
127.0.0.1 - - [27/Feb/2024 19:23:47] code 502, message Bad Gateway
127.0.0.1 - - [27/Feb/2024 19:23:47] "POST http://motoapi.amazonaws.com/moto-api/static/athena/query-results HTTP/1.1" 502 -

[bblommers]

Good catch!

Without the mock_aws-decorator it still succeeds for me though - see https://github.com/getmoto/moto/pull/7401/files

Edit: And I can see the requests come through in the Proxy logs, with verbose logging it looks like this:

INFO 2024-02-27 18:46:23,720 - POST http://motoapi.amazonaws.com/moto-api/static/athena/query-results
DEBUG 2024-02-27 18:46:23,720 - 	body	b'{"results": [{"rows": [{"Data": [{"VarCharValue": "1"}]}], "column_info": [{"CatalogName": "string", "SchemaName": "string", "TableName": "string", "Name": "string", "Label": "string", "Type": "string", "Precision": 123, "Scale": 123, "Nullable": "NOT_NULL", "CaseSensitive": true}]}]}'
DEBUG 2024-02-27 18:46:23,720 - 	headers	{'Host': 'motoapi.amazonaws.com', 'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate, br, zstd', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '285', 'Content-Type': 'application/json'}
DEBUG 2024-02-27 18:46:23,757 - 	=====RESPONSE========
DEBUG 2024-02-27 18:46:23,757 - 	(201, {}, '')

[megetron3]

please note that even when i dont set the athena results and execute this

    client = boto3.client("athena", endpoint_url='http://localhost:5000', region_name="us-east-1")
    details = client.get_query_execution(QueryExecutionId="any_id")

to the moto proxy or to the server, i expect zero results while getting:

127.0.0.1 - - [26/Feb/2024 17:15:23] "POST / HTTP/1.1" 500 -
Error on request:
Traceback (most recent call last):
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/werkzeug/serving.py", line 362, in run_wsgi
    execute(self.server.app)
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/werkzeug/serving.py", line 323, in execute
    application_iter = app(environ, start_response)
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/moto/moto_server/werkzeug_app.py", line 257, in __call__
    return backend_app(environ, start_response)
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/flask/app.py", line 1488, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/flask/app.py", line 1466, in wsgi_app
    response = self.handle_exception(e)
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/flask_cors/extension.py", line 176, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/flask/app.py", line 872, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/flask_cors/extension.py", line 176, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/flask/app.py", line 870, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/flask/app.py", line 855, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/moto/core/utils.py", line 111, in __call__
    result = self.callback(request, request.url, dict(request.headers))
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/moto/core/responses.py", line 255, in dispatch
    return cls()._dispatch(*args, **kwargs)
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/moto/core/responses.py", line 464, in _dispatch
    return self.call_action()
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/moto/core/responses.py", line 561, in call_action
    response = method()
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/moto/athena/responses.py", line 58, in get_query_execution
    execution = self.athena_backend.get_query_execution(exec_id)
  File "/Users/guy.eliyahu/repo/integration_tests/venv/lib/python3.10/site-packages/moto/athena/models.py", line 225, in get_query_execution
    return self.executions[exec_id]
KeyError: 'any_id'

[bblommers]

get_query_execution will only return query executions that have started, so start_query_execution would have to be called first.
As long as an execution has been started, the get_query_execution method should work without any other configuration.

To get the custom results that you uploaded via the MotoAPI, use the get_query_results method instead.