Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Error] AWS HTTP error: Client error: POST resulted in a 410 Gone response #9

Open
mikebronner opened this issue Jul 29, 2021 · 38 comments
Open

[Error] AWS HTTP error: Client error: POST resulted in a 410 Gone response #9

mikebronner opened this issue Jul 29, 2021 · 38 comments
Labels
bug Something isn't working

Comments

@mikebronner
Copy link
Contributor

mikebronner commented Jul 29, 2021 •
edited
Loading

Thank you for this package! I'm using it in conjunction with Laravel Vapor.
I'm receiving the following error intermittently, multiple times per day:

Aws\ApiGatewayManagementApi\Exception\ApiGatewayManagementApiException Error executing "PostToConnection" on "https://xxxxx.execute-api.us-east-1.amazonaws.com/yyyyy/@connections/DOvZ1cdTIAMCLtA%3D"; AWS HTTP error: Client error: `POST https://xxxxx.execute-api.us-east-1.amazonaws.com/yyyyy/@connections/DOvZ1cdTIAMCLtA%3D` resulted in a `410 Gone` response Unable to parse error information from response - Error parsing JSON: Syntax error
     vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:195 Aws\WrappedHttpHandler::parseError
     vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:97 Aws\WrappedHttpHandler::Aws\{closure}
     vendor/guzzlehttp/promises/src/Promise.php:204 GuzzleHttp\Promise\Promise::callHandler
     vendor/guzzlehttp/promises/src/Promise.php:169 GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}
     vendor/guzzlehttp/promises/src/RejectedPromise.php:42 GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}
     vendor/guzzlehttp/promises/src/TaskQueue.php:48 GuzzleHttp\Promise\TaskQueue::run
     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:158 GuzzleHttp\Handler\CurlMultiHandler::tick
     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:183 GuzzleHttp\Handler\CurlMultiHandler::execute
     vendor/guzzlehttp/promises/src/Promise.php:248 GuzzleHttp\Promise\Promise::invokeWaitFn
     vendor/guzzlehttp/promises/src/Promise.php:224 GuzzleHttp\Promise\Promise::waitIfPending
     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList
     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending
     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList
     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending
     vendor/guzzlehttp/promises/src/Promise.php:62 GuzzleHttp\Promise\Promise::wait
     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:58 Aws\AwsClient::execute
     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:86 Aws\AwsClient::__call
     vendor/georgeboot/laravel-echo-api-gateway/src/ConnectionRepository.php:23 Georgeboot\LaravelEchoApiGateway\ConnectionRepository::sendMessage
     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:140 Georgeboot\LaravelEchoApiGateway\Handler::sendMessage
     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:66 Georgeboot\LaravelEchoApiGateway\Handler::handleMessage
     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:32 Georgeboot\LaravelEchoApiGateway\Handler::handleWebsocket
     vendor/bref/bref/src/Event/ApiGateway/WebsocketHandler.php:19 Bref\Event\ApiGateway\WebsocketHandler::handle
     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:60 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handleWebsocketEvent
     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:52 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handle
     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}
     vendor/laravel/framework/src/Illuminate/Container/Util.php:40 Illuminate\Container\Util::unwrapIfClosure
     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93 Illuminate\Container\BoundMethod::callBoundMethod
     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37 Illuminate\Container\BoundMethod::call
     vendor/laravel/framework/src/Illuminate/Container/Container.php:651 Illuminate\Container\Container::call
     vendor/laravel/framework/src/Illuminate/Console/Command.php:136 Illuminate\Console\Command::execute
     vendor/symfony/console/Command/Command.php:299 Symfony\Component\Console\Command\Command::run
     vendor/laravel/framework/src/Illuminate/Console/Command.php:121 Illuminate\Console\Command::run
     vendor/symfony/console/Application.php:978 Symfony\Component\Console\Application::doRunCommand
     vendor/symfony/console/Application.php:295 Symfony\Component\Console\Application::doRun
     vendor/symfony/console/Application.php:167 Symfony\Component\Console\Application::run
     vendor/laravel/framework/src/Illuminate/Console/Application.php:92 Illuminate\Console\Application::run
     vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129 Illuminate\Foundation\Console\Kernel::handle
     artisan:43 [main]

Caused by: GuzzleHttp\Exception\ClientException Client error: `POST https://xxxxx.execute-api.us-east-1.amazonaws.com/yyyyy/@connections/DOvZ1cdTIAMCLtA%3D` resulted in a `410 Gone` response
     vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113 GuzzleHttp\Exception\RequestException::create
     vendor/guzzlehttp/guzzle/src/Middleware.php:69 GuzzleHttp\Middleware::GuzzleHttp\{closure}
     vendor/guzzlehttp/promises/src/Promise.php:204 GuzzleHttp\Promise\Promise::callHandler
     vendor/guzzlehttp/promises/src/Promise.php:153 GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}
     vendor/guzzlehttp/promises/src/TaskQueue.php:48 GuzzleHttp\Promise\TaskQueue::run
     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:158 GuzzleHttp\Handler\CurlMultiHandler::tick
     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:183 GuzzleHttp\Handler\CurlMultiHandler::execute
     vendor/guzzlehttp/promises/src/Promise.php:248 GuzzleHttp\Promise\Promise::invokeWaitFn
     vendor/guzzlehttp/promises/src/Promise.php:224 GuzzleHttp\Promise\Promise::waitIfPending
     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList
     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending
     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList
     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending
     vendor/guzzlehttp/promises/src/Promise.php:62 GuzzleHttp\Promise\Promise::wait
     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:58 Aws\AwsClient::execute
     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:86 Aws\AwsClient::__call
     vendor/georgeboot/laravel-echo-api-gateway/src/ConnectionRepository.php:23 Georgeboot\LaravelEchoApiGateway\ConnectionRepository::sendMessage
     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:140 Georgeboot\LaravelEchoApiGateway\Handler::sendMessage
     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:66 Georgeboot\LaravelEchoApiGateway\Handler::handleMessage
     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:32 Georgeboot\LaravelEchoApiGateway\Handler::handleWebsocket
     vendor/bref/bref/src/Event/ApiGateway/WebsocketHandler.php:19 Bref\Event\ApiGateway\WebsocketHandler::handle
     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:60 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handleWebsocketEvent
     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:52 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handle
     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}
     vendor/laravel/framework/src/Illuminate/Container/Util.php:40 Illuminate\Container\Util::unwrapIfClosure
     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93 Illuminate\Container\BoundMethod::callBoundMethod
     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37 Illuminate\Container\BoundMethod::call
     vendor/laravel/framework/src/Illuminate/Container/Container.php:651 Illuminate\Container\Container::call
     vendor/laravel/framework/src/Illuminate/Console/Command.php:136 Illuminate\Console\Command::execute
     vendor/symfony/console/Command/Command.php:299 Symfony\Component\Console\Command\Command::run
     vendor/laravel/framework/src/Illuminate/Console/Command.php:121 Illuminate\Console\Command::run
     vendor/symfony/console/Application.php:978 Symfony\Component\Console\Application::doRunCommand
     vendor/symfony/console/Application.php:295 Symfony\Component\Console\Application::doRun
     vendor/symfony/console/Application.php:167 Symfony\Component\Console\Application::run
     vendor/laravel/framework/src/Illuminate/Console/Application.php:92 Illuminate\Console\Application::run
     vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129 Illuminate\Foundation\Console\Kernel::handle
     artisan:43 [main]

I believe this has the side effect that the Laravel Echo event will not update the site when this error happens. I'm unclear as to what the cause might be.
@mikebronner mikebronner changed the title [Error] AWS HTTP error: Client error: POST https://xxxxxxx.execute-api.us-east-1.amazonaws.com/yyyyyyy/@connections/DOvZ1cdTIAMCLtA%3D resulted in a 410 Gone response Unable to parse error information from response [Error] AWS HTTP error: Client error: POST resulted in a 410 Gone response Jul 29, 2021
@georgeboot
Copy link
Owner

Hi Mike, thanks for reporting!

I believe this is due to the connection being (unexpectedly) closed by the client. This is currently not handled by Echo.

I will investigate and see if I can fix this for you.

@georgeboot georgeboot added the bug Something isn't working label Jul 29, 2021
georgeboot added a commit that referenced this issue Jul 29, 2021
@georgeboot
Handle dropped connections #9
e41a068
@georgeboot
Copy link
Owner

georgeboot commented Jul 29, 2021 •
edited
Loading

@mikebronner can you please test the dev-master to verify this issue has been fixed now (e41a068)? Once I can confirm, I will tag a new version.

@mikebronner
Copy link
Contributor Author

Hi @georgeboot, thank you so much for the ultra-quick response! :) I will update and monitor over the next few days and report back. Thank you!

@mikebronner
Copy link
Contributor Author

mikebronner commented Jul 29, 2021 •
edited
Loading

Hi @georgeboot It appears we are still getting them. These might only be occurring when we are deploying, and not on every deployment. I wonder if this happens when vapor deploys midstream of an Echo event?

Update: may not be related to deployments after all. The latest error happened 5 minutes after the deployment completed.

@georgeboot
Copy link
Owner

Yeah I think I am catching the exception incorrectly. But it's hard to catch because AWS doesn't document that topic very good (to my knowledge).

I will look further tomorrow.

@mikebronner
Copy link
Contributor Author

mikebronner commented Jul 29, 2021 •
edited
Loading

NP, thank you for all your help with this!

Further Update:

This may be related to queued events on vapor. I have several failed queue jobs that triggered this error. I will keep monitoring this.

@georgeboot
Copy link
Owner

Mind trying the latest master? AWS SDK has a funny way of handling exceptions but I believe it should be correct this time.

@mikebronner
Copy link
Contributor Author

Absolutely, we'll get it in our next deployment today and keep monitoring. :)

@georgeboot
Copy link
Owner

@mikebronner is the issue fixed now?

@semsphy
Copy link

semsphy commented Aug 3, 2021 •
edited
Loading

@mikebronner is the issue fixed now?

Nope. I just switched to dev-master and redeployed, I still got the error message in my log.

ERROR: Error executing "PostToConnection" on "https://xxxxx-api.ap-southeast-1.amazonaws.com/dev/@connections/DfbxWdFeyQ0CGKg%3D"; AWS HTTP error: Client error: `xxxxxx` resulted in a `403 Forbidden` response:
{"message":"Forbidden"}

@georgeboot
Copy link
Owner

That is a different error. You previously got a 410 Gone, the one you are getting now is a 403 Forbidden.

@semsphy
Copy link

semsphy commented Aug 3, 2021

That is a different error. You previously got a 410 Gone, the one you are getting now is a 403 Forbidden.

Whoop, let's me re-test and let's you know.

@semsphy
Copy link

semsphy commented Aug 3, 2021

@georgeboot , confirmed, it is fixed. I got not error now.

@mikebronner
Copy link
Contributor Author

@georgeboot I installed the update about 2 hrs ago, haven't seen the error come back yet, will continue to monitor.

@mikebronner
Copy link
Contributor Author

It appears that I might still be getting the error. Here is the full stack trace:

Aws\ApiGatewayManagementApi\Exception\ApiGatewayManagementApiException Error executing "PostToConnection" on "https://wxj1pjq938.execute-api.us-east-1.amazonaws.com/production/@connections/Df14mfUXIAMCIyg%3D"; AWS HTTP error: Client error: `POST https://wxj1pjq938.execute-api.us-east-1.amazonaws.com/production/@connections/Df14mfUXIAMCIyg%3D` resulted in a `410 Gone` response Unable to parse error information from response - Error parsing JSON: Syntax error      vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:195 Aws\WrappedHttpHandler::parseError     vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:97 Aws\WrappedHttpHandler::Aws\{closure}     vendor/guzzlehttp/promises/src/Promise.php:204 GuzzleHttp\Promise\Promise::callHandler     vendor/guzzlehttp/promises/src/Promise.php:169 GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}     vendor/guzzlehttp/promises/src/RejectedPromise.php:42 GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}     vendor/guzzlehttp/promises/src/TaskQueue.php:48 GuzzleHttp\Promise\TaskQueue::run     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:158 GuzzleHttp\Handler\CurlMultiHandler::tick     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:183 GuzzleHttp\Handler\CurlMultiHandler::execute     vendor/guzzlehttp/promises/src/Promise.php:248 GuzzleHttp\Promise\Promise::invokeWaitFn     vendor/guzzlehttp/promises/src/Promise.php:224 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:62 GuzzleHttp\Promise\Promise::wait     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:58 Aws\AwsClient::execute     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:86 Aws\AwsClient::__call     vendor/georgeboot/laravel-echo-api-gateway/src/ConnectionRepository.php:27 Georgeboot\LaravelEchoApiGateway\ConnectionRepository::sendMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:139 Georgeboot\LaravelEchoApiGateway\Handler::sendMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:117 Georgeboot\LaravelEchoApiGateway\Handler::subscribe     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:69 Georgeboot\LaravelEchoApiGateway\Handler::handleMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:31 Georgeboot\LaravelEchoApiGateway\Handler::handleWebsocket     vendor/bref/bref/src/Event/ApiGateway/WebsocketHandler.php:19 Bref\Event\ApiGateway\WebsocketHandler::handle     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:60 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handleWebsocketEvent     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:52 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handle     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}     vendor/laravel/framework/src/Illuminate/Container/Util.php:40 Illuminate\Container\Util::unwrapIfClosure     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93 Illuminate\Container\BoundMethod::callBoundMethod     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37 Illuminate\Container\BoundMethod::call     vendor/laravel/framework/src/Illuminate/Container/Container.php:651 Illuminate\Container\Container::call     vendor/laravel/framework/src/Illuminate/Console/Command.php:136 Illuminate\Console\Command::execute     vendor/symfony/console/Command/Command.php:299 Symfony\Component\Console\Command\Command::run     vendor/laravel/framework/src/Illuminate/Console/Command.php:121 Illuminate\Console\Command::run     vendor/symfony/console/Application.php:978 Symfony\Component\Console\Application::doRunCommand     vendor/symfony/console/Application.php:295 Symfony\Component\Console\Application::doRun     vendor/symfony/console/Application.php:167 Symfony\Component\Console\Application::run     vendor/laravel/framework/src/Illuminate/Console/Application.php:92 Illuminate\Console\Application::run     vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129 Illuminate\Foundation\Console\Kernel::handle     artisan:43 [main] 

Caused by: GuzzleHttp\Exception\ClientException Client error: `POST https://wxj1pjq938.execute-api.us-east-1.amazonaws.com/production/@connections/Df14mfUXIAMCIyg%3D` resulted in a `410 Gone` response      vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113 GuzzleHttp\Exception\RequestException::create     vendor/guzzlehttp/guzzle/src/Middleware.php:69 GuzzleHttp\Middleware::GuzzleHttp\{closure}     vendor/guzzlehttp/promises/src/Promise.php:204 GuzzleHttp\Promise\Promise::callHandler     vendor/guzzlehttp/promises/src/Promise.php:153 GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}     vendor/guzzlehttp/promises/src/TaskQueue.php:48 GuzzleHttp\Promise\TaskQueue::run     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:158 GuzzleHttp\Handler\CurlMultiHandler::tick     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:183 GuzzleHttp\Handler\CurlMultiHandler::execute     vendor/guzzlehttp/promises/src/Promise.php:248 GuzzleHttp\Promise\Promise::invokeWaitFn     vendor/guzzlehttp/promises/src/Promise.php:224 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:62 GuzzleHttp\Promise\Promise::wait     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:58 Aws\AwsClient::execute     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:86 Aws\AwsClient::__call     vendor/georgeboot/laravel-echo-api-gateway/src/ConnectionRepository.php:27 Georgeboot\LaravelEchoApiGateway\ConnectionRepository::sendMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:139 Georgeboot\LaravelEchoApiGateway\Handler::sendMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:117 Georgeboot\LaravelEchoApiGateway\Handler::subscribe     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:69 Georgeboot\LaravelEchoApiGateway\Handler::handleMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:31 Georgeboot\LaravelEchoApiGateway\Handler::handleWebsocket     vendor/bref/bref/src/Event/ApiGateway/WebsocketHandler.php:19 Bref\Event\ApiGateway\WebsocketHandler::handle     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:60 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handleWebsocketEvent     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:52 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handle     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}     vendor/laravel/framework/src/Illuminate/Container/Util.php:40 Illuminate\Container\Util::unwrapIfClosure     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93 Illuminate\Container\BoundMethod::callBoundMethod     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37 Illuminate\Container\BoundMethod::call     vendor/laravel/framework/src/Illuminate/Container/Container.php:651 Illuminate\Container\Container::call     vendor/laravel/framework/src/Illuminate/Console/Command.php:136 Illuminate\Console\Command::execute     vendor/symfony/console/Command/Command.php:299 Symfony\Component\Console\Command\Command::run     vendor/laravel/framework/src/Illuminate/Console/Command.php:121 Illuminate\Console\Command::run     vendor/symfony/console/Application.php:978 Symfony\Component\Console\Application::doRunCommand     vendor/symfony/console/Application.php:295 Symfony\Component\Console\Application::doRun     vendor/symfony/console/Application.php:167 Symfony\Component\Console\Application::run     vendor/laravel/framework/src/Illuminate/Console/Application.php:92 Illuminate\Console\Application::run     vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129 Illuminate\Foundation\Console\Kernel::handle     artisan:43 [main]

@georgeboot georgeboot reopened this Aug 3, 2021
@georgeboot
Copy link
Owner

georgeboot commented Aug 3, 2021 •
edited
Loading

Grr. Why does AWS make it so hard to mock errors 🙄

Will dive in tomorrow and hopefully fix this once and for real. Apologies for the hassle!

@mikebronner
Copy link
Contributor Author

No fret, @georgeboot, thanks for all your work on this! Also no rush, I don't think these are causing issues?

You mentioned the solution is to close the connection and drop the pending message. Is it technically possible to resubmit the message, if it failed, to an open connection, or is the new connection information not available at that point?

@georgeboot
Copy link
Owner

georgeboot commented Aug 3, 2021 •
edited
Loading

Is it technically possible to resubmit the message, if it failed, to an open connection, or is the new connection information not available at that point?

Nope. Connections can only be initiated from the client side.

I believe pusher will also just ignore the message.

Once I have implemented client-reconnects (#2), this will no longer be an issue as the old connections details will be purged and Echo will only see the new connection.

@semsphy
Copy link

semsphy commented Aug 5, 2021

It appears that I might still be getting the error. Here is the full stack trace:

Aws\ApiGatewayManagementApi\Exception\ApiGatewayManagementApiException Error executing "PostToConnection" on "https://wxj1pjq938.execute-api.us-east-1.amazonaws.com/production/@connections/Df14mfUXIAMCIyg%3D"; AWS HTTP error: Client error: `POST https://wxj1pjq938.execute-api.us-east-1.amazonaws.com/production/@connections/Df14mfUXIAMCIyg%3D` resulted in a `410 Gone` response Unable to parse error information from response - Error parsing JSON: Syntax error      vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:195 Aws\WrappedHttpHandler::parseError     vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:97 Aws\WrappedHttpHandler::Aws\{closure}     vendor/guzzlehttp/promises/src/Promise.php:204 GuzzleHttp\Promise\Promise::callHandler     vendor/guzzlehttp/promises/src/Promise.php:169 GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}     vendor/guzzlehttp/promises/src/RejectedPromise.php:42 GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}     vendor/guzzlehttp/promises/src/TaskQueue.php:48 GuzzleHttp\Promise\TaskQueue::run     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:158 GuzzleHttp\Handler\CurlMultiHandler::tick     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:183 GuzzleHttp\Handler\CurlMultiHandler::execute     vendor/guzzlehttp/promises/src/Promise.php:248 GuzzleHttp\Promise\Promise::invokeWaitFn     vendor/guzzlehttp/promises/src/Promise.php:224 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:62 GuzzleHttp\Promise\Promise::wait     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:58 Aws\AwsClient::execute     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:86 Aws\AwsClient::__call     vendor/georgeboot/laravel-echo-api-gateway/src/ConnectionRepository.php:27 Georgeboot\LaravelEchoApiGateway\ConnectionRepository::sendMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:139 Georgeboot\LaravelEchoApiGateway\Handler::sendMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:117 Georgeboot\LaravelEchoApiGateway\Handler::subscribe     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:69 Georgeboot\LaravelEchoApiGateway\Handler::handleMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:31 Georgeboot\LaravelEchoApiGateway\Handler::handleWebsocket     vendor/bref/bref/src/Event/ApiGateway/WebsocketHandler.php:19 Bref\Event\ApiGateway\WebsocketHandler::handle     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:60 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handleWebsocketEvent     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:52 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handle     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}     vendor/laravel/framework/src/Illuminate/Container/Util.php:40 Illuminate\Container\Util::unwrapIfClosure     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93 Illuminate\Container\BoundMethod::callBoundMethod     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37 Illuminate\Container\BoundMethod::call     vendor/laravel/framework/src/Illuminate/Container/Container.php:651 Illuminate\Container\Container::call     vendor/laravel/framework/src/Illuminate/Console/Command.php:136 Illuminate\Console\Command::execute     vendor/symfony/console/Command/Command.php:299 Symfony\Component\Console\Command\Command::run     vendor/laravel/framework/src/Illuminate/Console/Command.php:121 Illuminate\Console\Command::run     vendor/symfony/console/Application.php:978 Symfony\Component\Console\Application::doRunCommand     vendor/symfony/console/Application.php:295 Symfony\Component\Console\Application::doRun     vendor/symfony/console/Application.php:167 Symfony\Component\Console\Application::run     vendor/laravel/framework/src/Illuminate/Console/Application.php:92 Illuminate\Console\Application::run     vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129 Illuminate\Foundation\Console\Kernel::handle     artisan:43 [main] 

Caused by: GuzzleHttp\Exception\ClientException Client error: `POST https://wxj1pjq938.execute-api.us-east-1.amazonaws.com/production/@connections/Df14mfUXIAMCIyg%3D` resulted in a `410 Gone` response      vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113 GuzzleHttp\Exception\RequestException::create     vendor/guzzlehttp/guzzle/src/Middleware.php:69 GuzzleHttp\Middleware::GuzzleHttp\{closure}     vendor/guzzlehttp/promises/src/Promise.php:204 GuzzleHttp\Promise\Promise::callHandler     vendor/guzzlehttp/promises/src/Promise.php:153 GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}     vendor/guzzlehttp/promises/src/TaskQueue.php:48 GuzzleHttp\Promise\TaskQueue::run     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:158 GuzzleHttp\Handler\CurlMultiHandler::tick     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:183 GuzzleHttp\Handler\CurlMultiHandler::execute     vendor/guzzlehttp/promises/src/Promise.php:248 GuzzleHttp\Promise\Promise::invokeWaitFn     vendor/guzzlehttp/promises/src/Promise.php:224 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:62 GuzzleHttp\Promise\Promise::wait     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:58 Aws\AwsClient::execute     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:86 Aws\AwsClient::__call     vendor/georgeboot/laravel-echo-api-gateway/src/ConnectionRepository.php:27 Georgeboot\LaravelEchoApiGateway\ConnectionRepository::sendMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:139 Georgeboot\LaravelEchoApiGateway\Handler::sendMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:117 Georgeboot\LaravelEchoApiGateway\Handler::subscribe     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:69 Georgeboot\LaravelEchoApiGateway\Handler::handleMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:31 Georgeboot\LaravelEchoApiGateway\Handler::handleWebsocket     vendor/bref/bref/src/Event/ApiGateway/WebsocketHandler.php:19 Bref\Event\ApiGateway\WebsocketHandler::handle     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:60 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handleWebsocketEvent     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:52 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handle     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}     vendor/laravel/framework/src/Illuminate/Container/Util.php:40 Illuminate\Container\Util::unwrapIfClosure     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93 Illuminate\Container\BoundMethod::callBoundMethod     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37 Illuminate\Container\BoundMethod::call     vendor/laravel/framework/src/Illuminate/Container/Container.php:651 Illuminate\Container\Container::call     vendor/laravel/framework/src/Illuminate/Console/Command.php:136 Illuminate\Console\Command::execute     vendor/symfony/console/Command/Command.php:299 Symfony\Component\Console\Command\Command::run     vendor/laravel/framework/src/Illuminate/Console/Command.php:121 Illuminate\Console\Command::run     vendor/symfony/console/Application.php:978 Symfony\Component\Console\Application::doRunCommand     vendor/symfony/console/Application.php:295 Symfony\Component\Console\Application::doRun     vendor/symfony/console/Application.php:167 Symfony\Component\Console\Application::run     vendor/laravel/framework/src/Illuminate/Console/Application.php:92 Illuminate\Console\Application::run     vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129 Illuminate\Foundation\Console\Kernel::handle     artisan:43 [main]

Hi @mikebronner , I faced this problem when I broadcasted my message. How about you?

@mikebronner
Copy link
Contributor Author

@semsphy I believe that is the case as well. Is there another scenario this could happen? (Just asking since this package is for event broadcasting, and I didn't know if I maybe missed something to troubleshoot.)

@semsphy
Copy link

semsphy commented Aug 6, 2021

@mikebronner , indeed, there is only this scenario. I just wanted to confirm your scenario. :)

@georgeboot
Copy link
Owner

Can you try the dev-master again? Tried something different in 957ce4f

@semsphy
Copy link

semsphy commented Aug 6, 2021 

[PRODUCTION]: 2021-08-06 12:41:19🚨ERROR: Error executing "PostToConnection" on "https://xxxx.execute-api.ap-southeast-1.amazonaws.com/prod/@connections/DfyBYdK2SQ0CGCA%3D"; AWS HTTP error: Client error: `POST https://xxxxx.execute-api.ap-southeast-1.amazonaws.com/prod/@connections/DfyBYdK2SQ0CGCA%3D` resulted in a `410 Gone` response Unable to parse error information from response - Error parsing JSON: Syntax error 
[{"exception":"[object] (Aws\\ApiGatewayManagementApi\\Exception\\ApiGatewayManagementApiException(code: 0): Error executing \"PostToConnection\" on \"https://xxxxx.execute-api.ap-southeast-1.amazonaws.com/prod/@connections/DfyBYdK2SQ0CGCA%3D\"; AWS HTTP error: Client error: `POST https://xxxxx.execute-api.ap-southeast-1.amazonaws.com/prod/@connections/DfyBYdK2SQ0CGCA%3D` resulted in a `410 Gone` response Unable to parse error information from response - Error parsing JSON: Syntax error at /var/task/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:195)
[previous exception] [object] (GuzzleHttp\\Exception\\ClientException(code: 410): Client error: `POST https://xxxxx.execute-api.ap-southeast-1.amazonaws.com/prod/@connections/DfyBYdK2SQ0CGCA%3D` resulted in a `410 Gone` response at /var/task/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113)"}]```

@mikebronner
Copy link
Contributor Author

Just pushed the new update, will keep monitoring

@georgeboot
Copy link
Owner

georgeboot commented Aug 6, 2021 •
edited
Loading

Yeah so oke I see what is happening. Thanks for testing for me by the way, appreciate it!

Regardless of what type of error the API returns, it already breaks when trying to set-up the ApiGatewayManagementApiException. I thought perhaps we can just catch the GuzzleException but we cant.

I will open up a issue at the PHP SDK and make sure this gets fixed in there. aws/aws-sdk-php#2285

@semsphy
Copy link

semsphy commented Aug 6, 2021 •
edited
Loading

I saw the creator of bref, he handled different way.

use Bref\Websocket\SimpleWebsocketClient;

$websocketClient = SimpleWebsocketClient::create(getenv('API_GATEWAY_ID'), getenv('AWS_REGION'), 'prod',);
foreach ($this->connectionStorage->getAllConnections() as $connectionId) {
    $websocketClient->message($connectionId, $message);
}

@georgeboot
Copy link
Owner

I saw the creator of bref, he handled different way.

use Bref\Websocket\SimpleWebsocketClient;

$websocketClient = SimpleWebsocketClient::create(getenv('API_GATEWAY_ID'), getenv('AWS_REGION'), 'prod',);
foreach ($this->connectionStorage->getAllConnections() as $connectionId) {
    $websocketClient->message($connectionId, $message);
}

yeah he uses https://github.com/async-aws/aws. Might do that as well, as the main AWS SDK can be a bit of a pain in the ass sometimes.

@mikebronner
Copy link
Contributor Author

Hi guys, I just wanted to check in on this. I know there is the issue opened with AWS. @georgeboot have you been able to make any progress implementing async-aws? This is becoming critical for us, we are seeing around 14000 errors due to this each month, impacting our user-experience significantly.

Thanks again for all your hard work on this package. :)

@georgeboot
Copy link
Owner

@mikebronner can you perhaps help with Sam's request in aws/aws-sdk-php#2285 (comment)? I'm just too busy to do all the debugging work now.

As soon as AWS SDK fix the issue, this package should have the fix as well. Moving the async-aws is an option, but likely more work.

@mikebronner
Copy link
Contributor Author

@georgeboot sure thing, I'll give it a shot.

@mikebronner
Copy link
Contributor Author

@georgeboot quick question: can you point me where to set debug => true, per Sam's request? Is there a way I can do it so that composer won't overwrite it on update?

@georgeboot
Copy link
Owner

I believe what Sam wants is the following:

  1. Create a websocket API
  2. Establish a connection and write down the connectionId
  3. Close the connection
  4. Send a message to the closed connection using a client with debug = true and capture the full output

Setting debug=true needs to be set here:

laravel-echo-api-gateway/src/ConnectionRepository.php

Lines 17 to 20 in 957ce4f

$this->apiGatewayManagementApiClient = new ApiGatewayManagementApiClient(array_merge($config['connection'], [
'version' => '2018-11-29',
'endpoint' => "https://{$config['api']['id']}.execute-api.{$config['connection']['region']}.amazonaws.com/{$config['api']['stage']}/",
]));

As you already have the package deployed, you could inspect the DynamoDB table to obtain a connectionId. Wait till it disconnects and send the message from your local computer (using live key ID + secret). Thats the easiest I can think of.

Thanks for helping!

@mikebronner
Copy link
Contributor Author

@georgeboot Thanks for the hints. We have not been able to replicate the problem, we always get 403 Forbidden when trying to access a closed connection.

@georgeboot
Copy link
Owner

@mikebronner are you able to post to a valid connection? 403 indicates unauthorised, which is a different error.

@mikebronner
Copy link
Contributor Author

mikebronner commented Oct 5, 2021 •
edited
Loading

@georgeboot Yes, but the connection is already gone. We were looking through our logs, and the code for the package, and think we might have an idea. Our AWS guy is thinking that this is due to ConsistentReads defaulting to false. What if you did this:

    public function getConnectionIdsForChannel(string ...$channels): Collection
    {
        $promises = collect($channels)->map(fn($channel) => $this->dynamoDb->queryAsync([
            "ConsistentRead" => true,
            'TableName' => $this->table,
            'IndexName' => 'lookup-by-channel',
            'KeyConditionExpression' => 'channel = :channel',
            'ExpressionAttributeValues' => [
                ':channel' => ['S' => $channel],
            ],
        ]))->toArray();

        $responses = Utils::all($promises)->wait();

        return collect($responses)
            ->flatmap(fn($result) => $result['Items'])
            ->map(fn($item) => $item['connectionId']['S'])
            ->unique();
    }

And then also further down where you run ->query(). That will make sure the DynamoDB instances are synced up and all have the same connections configured.

This is just an observation, since we cannot reproduce it under set conditions, and our thinking is that it is due to the asynchronous nature of DynamoDB instances, and getting that to stay in sync with our app.

I'll submit this in a PR and test.

@georgeboot
Copy link
Owner

Could be, but sounds like a race condition to me. Happens a bit too much for that, but perhaps you're right.

If happily accept your PR if it fixes the issue 😊

@mikebronner
Copy link
Contributor Author

I put the PR patch in my composer and will be testing, I'll let you know. :)

@jdavidbakr
Copy link
Contributor

I was getting this same issue when I was trying to implement presence channels. I think this change to src/ConnectionRepository.php should fix this issue - it seems there may be multiple ways that the exception is being thrown.

https://github.com/georgeboot/laravel-echo-api-gateway/pull/25/files

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jdavidbakr @semsphy @georgeboot @mikebronner