source-custom/forked "An unknown error occurred. (HTTP 504)", "Failed to find output files from connector within timeout of 9 minute(s). Is the connector still running?"

[jeremy-ingenuity]

Connector Name

source-custom/forked with modifications

Connector Version

source-clockify forked 0.4.1 and source-sonar-cloud 0.2.2

What step the error happened?

During the sync

Relevant information

This connector (source-clockify-custom from yml) works at least a day ago but now even after retesting, it would return an HTTP 504 error

As for the actual sync, it would return the following error:

Relevant log output

2024-11-21 13:21:20 platform > Failed to update workload status to launched. Workload may be reprocessed on restart.
java.lang.Throwable: Client error : 410 Gone {"message":"Heartbeat a workload in a terminal state"}
        at io.airbyte.workload.api.client.generated.WorkloadApi.workloadLaunched(Unknown Source)
        at io.airbyte.workload.launcher.client.WorkloadApiClient.updateStatusToLaunched(Unknown Source)
        at io.airbyte.workload.launcher.pipeline.handlers.SuccessHandler.accept(Unknown Source)
        at io.airbyte.workload.launcher.pipeline.LaunchPipeline$buildPipeline$2.invoke(Unknown Source)
        at io.airbyte.workload.launcher.pipeline.LaunchPipeline$buildPipeline$2.invoke(Unknown Source)
        at io.airbyte.workload.launcher.pipeline.LaunchPipeline.buildPipeline$lambda$1(Unknown Source)
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(Unknown Source)
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(Unknown Source)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(Unknown Source)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(Unknown Source)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(Unknown Source)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(Unknown Source)
        at reactor.core.publisher.Operators$ScalarSubscription.request(Unknown Source)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(Unknown Source)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(Unknown Source)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(Unknown Source)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(Unknown Source)
        at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Unknown Source)
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(Unknown Source)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(Unknown Source)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(Unknown Source)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(Unknown Source)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(Unknown Source)
        at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(Unknown Source)
        at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(Unknown Source)
        at reactor.core.publisher.Mono.subscribe(Unknown Source)
        at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(Unknown Source)
        at reactor.core.scheduler.ImmediateScheduler$ImmediateSchedulerWorker.schedule(Unknown Source)
        at reactor.core.publisher.MonoSubscribeOn.subscribeOrReturn(Unknown Source)
        at reactor.core.publisher.Mono.subscribe(Unknown Source)
        at reactor.core.publisher.Mono.subscribeWith(Unknown Source)
        at reactor.core.publisher.Mono.subscribe(Unknown Source)
        at io.airbyte.workload.launcher.pipeline.LaunchPipeline.accept(Unknown Source)
        at io.airbyte.workload.launcher.pipeline.consumer.LauncherMessageConsumer.consume(Unknown Source)
        at io.airbyte.workload.launcher.pipeline.consumer.LauncherMessageConsumer.consume(Unknown Source)
        at io.airbyte.commons.temporal.queue.QueueActivityImpl.consume(Unknown Source)
        at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(Unknown Source)
        at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(Unknown Source)
        at io.temporal.common.interceptors.ActivityInboundCallsInterceptorBase.execute(Unknown Source)
        at io.temporal.opentracing.internal.OpenTracingActivityInboundCallsInterceptor.execute(Unknown Source)
        at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(Unknown Source)
        at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(Unknown Source)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(Unknown Source)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(Unknown Source)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(Unknown Source)
        at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

2024-11-21 13:21:20 platform > Pipeline completed for workload: 61102587-464a-4bea-b783-3ef96d439c80_212_4_check.
2024-11-21 13:21:20 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=20, successiveCompleteFailures=5, totalCompleteFailures=5, successivePartialFailures=0, totalPartialFailures=0)
 Backoff before next attempt: 13 minutes 30 seconds
2024-11-21 13:21:20 platform > Failing job: 212, reason: Job failed after too many retries for connection 86d552aa-2b39-4d9c-b7e4-676543ac278e

Contribute

• Yes, I want to contribute

[MoralesPablo]

Same here!

[jeremy-ingenuity]

It seems to work now, all my custom connectors (source) are now working fine (healthy)