Runs in Docker, but not in Amazon ECS

[MaffooClock]

I've been fighting this problem for many hours. I have been trying to deploy the Docker image to Amazon ECS, which is successful, but it will not respond on port 3000.

I can shell into the container and verify that node is running:

/home/node # ps -eaf
PID   USER     TIME  COMMAND
    1 node      0:00 tini -- node bin/www
    8 node      0:02 node bin/www
   15 root      0:00 /managed-agents/execute-command/amazon-ssm-agent
   33 root      0:00 /managed-agents/execute-command/ssm-agent-worker
   41 root      0:00 /managed-agents/execute-command/ssm-session-worker ecs-execute-command-cukrysyhd4jgufzao223v5l3xe
   52 root      0:00 /bin/sh
   53 root      0:00 ps -eaf

...and that it is listening on TCP 3000, too:

/home/node # netstat -pant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 172.30.8.162:41308      172.30.22.203:443       ESTABLISHED 41/ssm-session-work
tcp        0      0 172.30.8.162:36062      172.30.3.159:6379       ESTABLISHED -
tcp        0      0 172.30.8.162:45130      172.30.22.203:443       ESTABLISHED 33/ssm-agent-worker
tcp        0      0 172.30.8.162:41622      172.30.65.24:5432       ESTABLISHED -
tcp        0      0 :::3000                 :::*                    LISTEN      -
tcp        0      0 ::ffff:172.30.8.162:3000 ::ffff:192.168.20.83:50404 ESTABLISHED -
tcp        0      0 ::ffff:172.30.8.162:3000 ::ffff:172.108.145.141:19915 ESTABLISHED -

In fact, you can even see two remote clients have established connections to TCP 3000 (one is a browser on my workstation, the other is a load balancer attempting to run a health check), but the node process just isn't responding.

The logs show the correct output:

{"level":30,"time":1721840619925,"pid":8,"hostname":"ip-172-30-8-162.ec2.internal","msg":"Server listening on port 3000"}
{"level":30,"time":1721840619924,"pid":8,"hostname":"ip-172-30-8-162.ec2.internal","msg":"Successfully set up DB. Starting server..."}
{"level":30,"time":1721840619894,"pid":8,"hostname":"ip-172-30-8-162.ec2.internal","msg":"Connected to OIDC issuer"}
{"level":30,"time":1721840619866,"pid":8,"hostname":"ip-172-30-8-162.ec2.internal","msg":"Setting up DB..."}
{"level":30,"time":1721840619865,"pid":8,"hostname":"ip-172-30-8-162.ec2.internal","msg":"Starting server under production environment..."}

But no matter what, pointing my browser to http://172.30.8.162:3000/app just hangs until it finally times out. Same with using curl -- the command stalls until it times out, with no output or errors:

$ curl --verbose http://172.30.8.162:3000/app
*   Trying 172.30.8.162:3000...
* Connected to 172.30.8.162 (172.30.8.162) port 3000
> GET /app HTTP/1.1
> Host: 172.30.8.162:3000
> User-Agent: curl/8.5.0
> Accept: */*
>
### ☝️ Hangs right here 

I even cloned the repo and rebuilt the production version into a new image (had to re-generate the package-lock.json because some of the dependencies were out of date and caused npm to fail), but the behavior was the same.

But again: if I run it in Docker, it runs just fine.

I'm not suggesting this is a problem with Blink itself (hence starting this discussion rather than a new issue), but obviously there's a difference between Docker and Amazon's Fargate platform. But I have other node-based apps running on Fargate, no problem, so I'm not sure why this one is different 🤷🏻‍♂️

Anyone have any insight? Or at least any suggestions on how to troubleshoot this further?

[MaffooClock]

Oh, and I just thought of another thing to try: sending an HTTP request from within the container itself.

/home/node # wget -S -O - http://localhost:3000/app
> Connecting to localhost:3000 (127.0.0.1:3000)
> 

And as with curl from my workstation, the wget request just hangs until it times out. Weird 🤔

[MaffooClock]

And one more thing...

While connected to the running container, I decided to start a second instance of the node application, but on a different port (which worked). Then, from another host, I made a curl request to the alternate port, but the result was the same as always.

But here's what I observed: when I tried to CTRL+C to terminate the second node process, it spit out Received shutdown signal. Closing connections... as expected, but kept running. The only way I could kill it was to kill my own session with the container, then start a new one and run kill -9 on the process ID.

At the same time, I had an SSH connection to a remote host, from which I was running curl to send that request to that alternate port... when I kill -9'd the second node process, the curl process exited with "curl: (52) Empty reply from server".

So that seems to indicate that the TCP connection (shown in netstat with an "ESTABLISHED" status) was being handled by the node process, it just wasn't doing anything with it.

I see this project is using express, so I wonder if there are any configuration options or method calls before server.listen() that make a difference in this behavior?

[MaffooClock]

As a test, I connected to the running container and created a file as bin/test:

const express = require( 'express' );
const app = express();

app.get( '/', ( req, res, next ) => {
  res.sendStatus( 200 );
});

app.listen( 8080, '0.0.0.0' );

...then executed node bin/test, and was able to get a response on port 8080. So that tells me there's maybe something screwy with a middleware? Not sure how to debug that since there are no errors...

[MaffooClock]

AHA! I think I found the problem:

blink/app.js

Line 40 in 8619fa7

.use(rateLimit.short)

I started commenting out use()'s, one at a time, until the node app responded (I kept manually executing a second PORT=8080 node bin/app on the running container as I tested each one).

So, for whatever reason, the express-rate-limit middleware causes the the Express server to hang indefinitely? And why it does this on Fargate and not in Docker is still a mystery.

Upon closer inspection, I found that if I comment out the store property of the limiter object, the app runs.

blink/middlewares/rate-limiter.js

Lines 9 to 19 in 8619fa7

	exports.short = rateLimit({
	legacyHeaders: true,
	windowMs: ms(process.env.RATE_LIMIT_SHORT_WINDOW),
	max: process.env.RATE_LIMIT_SHORT_MAX - 0,
	keyGenerator: req =>
	(req[JWT_AUTH_PROPERTY] || {}).sub || (req.user || {}).id || req.ip,
	store: new RedisStore({
	prefix: 'rate-limit-short',
	sendCommand
	})
	})

Apparently there's beef with the rate limiter and my Redis server? So, I ran a redis-cli test connection to my Redis instance from an EC2 instance that is on the same subnet... no response.

But it worked on my local Docker, which was configured to use the Postgres and Redis services in AWS (I have a site-to-site VPN to AWS, so I can access all resources privately), which misled me to believe it was a Docker vs. Fargate problem. So the problem with Fargate is that one of the ways to enable communication with other AWS services is to have endpoints for those services added to the VPC -- I'd forgotten to add endpoints for ElastiCache and RDS.

TL;DR:
So this WHOLE TIME, Blink had stalled on the rate limiter trying to make a connection to the Redis server. It should throw an exception instead of hanging indefinitely. I'll open an issue for this, at least...