Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable healthcheck for sql connection #3521

Merged
merged 4 commits into from
Aug 13, 2024
Merged

Conversation

thoniTUB
Copy link
Collaborator

@thoniTUB thoniTUB commented Aug 9, 2024

HealthCheck-Endpoint, when DB is down:

{
  "HikariPool-1.pool.ConnectivityCheck" : {
    "healthy" : false,
    "message" : "HikariPool-1 - Connection is not available, request timed out after 30015ms.",
    "error" : {
      "type" : "java.sql.SQLTransientConnectionException",
      "message" : "HikariPool-1 - Connection is not available, request timed out after 30015ms.",
      "stack" : [ "com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)", "com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181)", "com.zaxxer.hikari.metrics.dropwizard.CodahaleHealthChecker$ConnectivityHealthCheck.check(CodahaleHealthChecker.java:95)", "com.codahale.metrics.health.HealthCheck.execute(HealthCheck.java:374)", "com.codahale.metrics.health.HealthCheckRegistry.runHealthChecks(HealthCheckRegistry.java:184)", "io.dropwizard.metrics.servlets.HealthCheckServlet.runHealthChecks(HealthCheckServlet.java:177)", "io.dropwizard.metrics.servlets.HealthCheckServlet.doGet(HealthCheckServlet.java:146)", "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:500)", "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587)", "io.dropwizard.metrics.servlets.AdminServlet.service(AdminServlet.java:154)", "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587)", "org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)", "org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)", "io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:46)", "io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:40)", "org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)", "org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)", "org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)", "org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)", "org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)", "org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)", "org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)", "org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)", "org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)", "org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)", "io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)", "org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)", "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)", "io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26)", "org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:46)", "org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)", "org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)", "org.eclipse.jetty.server.Server.handle(Server.java:563)", "org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)", "org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)", "org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)", "org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)", "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)", "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)", "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)", "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)", "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)", "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)", "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)", "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)", "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)", "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)", "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)", "java.base/java.lang.Thread.run(Thread.java:840)" ],
      "cause" : {
        "type" : "org.postgresql.util.PSQLException",
        "message" : "Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.",
        "stack" : [ "org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:342)", "org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54)", "org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:253)", "org.postgresql.Driver.makeConnection(Driver.java:434)", "org.postgresql.Driver.connect(Driver.java:291)", "com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)", "com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:359)", "com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201)", "com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:470)", "com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:733)", "com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712)", "java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)", "java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)", "java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)", "java.base/java.lang.Thread.run(Thread.java:840)" ],
        "cause" : {
          "type" : "java.net.ConnectException",
          "message" : "Connection timed out: getsockopt",
          "stack" : [ "java.base/sun.nio.ch.Net.pollConnect(Native Method)", "java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)", "java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:554)", "java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602)", "java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)", "java.base/java.net.Socket.connect(Socket.java:633)", "org.postgresql.core.PGStream.createSocket(PGStream.java:243)", "org.postgresql.core.PGStream.<init>(PGStream.java:98)", "org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:132)", "org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:258)", "org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54)", "org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:253)", "org.postgresql.Driver.makeConnection(Driver.java:434)", "org.postgresql.Driver.connect(Driver.java:291)", "com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)", "com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:359)", "com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201)", "com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:470)", "com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:733)", "com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712)", "java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)", "java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)", "java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)", "java.base/java.lang.Thread.run(Thread.java:840)" ]
        }
      }
    },
    "duration" : 30015,
    "timestamp" : "2024-08-09T14:03:12.811+02:00"
  },
  "deadlocks" : {
    "healthy" : true,
    "duration" : 0,
    "timestamp" : "2024-08-09T14:03:12.811+02:00"
  }
}

when it is up again

{
  "HikariPool-1.pool.ConnectivityCheck" : {
    "healthy" : true,
    "duration" : 38,
    "timestamp" : "2024-08-09T14:06:46.497+02:00"
  },
  "deadlocks" : {
    "healthy" : true,
    "duration" : 0,
    "timestamp" : "2024-08-09T14:06:46.497+02:00"
  }
}

When the DB is down, the health check request returns after the connection/connectivity timeout is over (default 30 s).

The reconnect only works if the database connection could be established upon server start.

@awildturtok
Copy link
Collaborator

Sind rot fyi

@thoniTUB thoniTUB enabled auto-merge August 13, 2024 11:04
@thoniTUB thoniTUB merged commit 21d5916 into develop Aug 13, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants