Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide a sample application using Spanner emulator #515

Open
magiccrafter opened this issue Mar 23, 2022 · 8 comments
Open

Provide a sample application using Spanner emulator #515

magiccrafter opened this issue Mar 23, 2022 · 8 comments
Labels

Comments

@magiccrafter
Copy link
Contributor

Hi,

I'm getting the following exception after running a simple integration test. I've tried various combinations and I always hit the same error. I'm not sure if it is a configuration issue on my end or a bug since the information during the debugging was not sufficient to come up to a conclusion on what exactly the problem is.
It would be super beneficial if there was a sample app using Spanner + Spring Boot + R2DBC + Testcontainers (Spanner Emulator). The same way we have it for Postgres for example.

java.lang.AssertionError: expectation "expectComplete" failed (expected: onComplete(); actual: onError(com.google.cloud.spanner.SpannerException: UNAUTHENTICATED: com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.))

	at reactor.test.MessageFormatter.assertionError(MessageFormatter.java:115)
	at reactor.test.MessageFormatter.failPrefix(MessageFormatter.java:104)
	at reactor.test.MessageFormatter.fail(MessageFormatter.java:73)
	at reactor.test.MessageFormatter.failOptional(MessageFormatter.java:88)
	at reactor.test.DefaultStepVerifierBuilder.lambda$expectComplete$4(DefaultStepVerifierBuilder.java:336)
	at reactor.test.DefaultStepVerifierBuilder$SignalEvent.test(DefaultStepVerifierBuilder.java:2218)
	at reactor.test.DefaultStepVerifierBuilder$DefaultVerifySubscriber.onSignal(DefaultStepVerifierBuilder.java:1490)
	at reactor.test.DefaultStepVerifierBuilder$DefaultVerifySubscriber.onExpectation(DefaultStepVerifierBuilder.java:1438)
	at reactor.test.DefaultStepVerifierBuilder$DefaultVerifySubscriber.onError(DefaultStepVerifierBuilder.java:1091)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.checkTerminated(FluxFlatMap.java:842)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.drainLoop(FluxFlatMap.java:608)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.drain(FluxFlatMap.java:588)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onError(FluxFlatMap.java:451)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
	at reactor.core.publisher.Operators.error(Operators.java:198)
	at reactor.core.publisher.FluxError.subscribe(FluxError.java:43)
	at reactor.core.publisher.Flux.subscribe(Flux.java:8469)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
	at reactor.core.publisher.FluxUsingWhen$UsingWhenSubscriber.deferredError(FluxUsingWhen.java:398)
	at reactor.core.publisher.FluxUsingWhen$RollbackInner.onComplete(FluxUsingWhen.java:475)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onComplete(Operators.java:2058)
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:133)
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:144)
	at com.google.cloud.spanner.r2dbc.v2.DatabaseClientReactiveAdapter$1.onSuccess(DatabaseClientReactiveAdapter.java:320)
	at com.google.api.core.ApiFutures$1.onSuccess(ApiFutures.java:73)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1139)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
	Suppressed: com.google.cloud.spanner.SpannerException: UNAUTHENTICATED: com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
		at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:284)
		at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:170)
		at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:110)
		at com.google.cloud.spanner.SessionPool$WaiterFuture.pollUninterruptiblyWithTimeout(SessionPool.java:1558)
		at com.google.cloud.spanner.SessionPool$WaiterFuture.get(SessionPool.java:1529)
		at com.google.cloud.spanner.SessionPool$WaiterFuture.get(SessionPool.java:1506)
		at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:66)
		at com.google.cloud.spanner.SessionPool$PooledSessionFuture.get(SessionPool.java:1307)
		at com.google.cloud.spanner.SessionPool$AutoClosingReadContext$1.internalNext(SessionPool.java:289)
		at com.google.cloud.spanner.SessionPool$AutoClosingReadContext$1.next(SessionPool.java:252)
		at com.google.cloud.spanner.AsyncResultSetImpl$ProduceRowsCallable.call(AsyncResultSetImpl.java:340)
		at com.google.cloud.spanner.AsyncResultSetImpl$ProduceRowsCallable.call(AsyncResultSetImpl.java:334)
		at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
		at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
		at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
		... 5 more
	Caused by: com.google.cloud.spanner.SpannerException: UNAUTHENTICATED: com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
		at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:284)
		at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:170)
		at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:110)
		at com.google.cloud.spanner.SessionPool$SessionConsumerImpl.onSessionCreateFailure(SessionPool.java:2315)
		at com.google.cloud.spanner.SessionClient$BatchCreateSessionsRunnable.run(SessionClient.java:140)
		... 6 more
	Caused by: com.google.cloud.spanner.SpannerException: UNAUTHENTICATED: com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
		at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:284)
		at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:61)
		at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:181)
		at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.get(GapicSpannerRpc.java:1676)
		at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.batchCreateSessions(GapicSpannerRpc.java:1399)
		at com.google.cloud.spanner.SessionClient.internalBatchCreateSessions(SessionClient.java:299)
		at com.google.cloud.spanner.SessionClient.access$000(SessionClient.java:38)
		at com.google.cloud.spanner.SessionClient$BatchCreateSessionsRunnable.run(SessionClient.java:137)
		... 6 more
	Caused by: java.util.concurrent.ExecutionException: com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
		at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588)
		at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:567)
		at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.get(GapicSpannerRpc.java:1670)
		... 10 more
	Caused by: com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
		at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:71)
		at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
		at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
		at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
		at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
		at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1133)
		at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
		at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1277)
		at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
		at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
		at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:564)
		at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:534)
		at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
		at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
		at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
		at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
		at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
		at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
		at com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
		at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463)
		at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427)
		at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460)
		at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
		at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
		at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
		at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
		at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
		at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
		... 6 more
	Caused by: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
		at io.grpc.Status.asRuntimeException(Status.java:535)
		... 23 more

I've created this simple minimalistic app with just the dependencies mentioned above to narrow down the perimeter:
https://github.com/magiccrafter/spanner-spring-boot-r2jdbc-app

I've followed the Credentials section in the README without luck.
https://github.com/GoogleCloudPlatform/cloud-spanner-r2dbc#authentication

In the absence of explicit authentication options, Application Default Credentials will be automatically inferred from the environment in which the application is running, unless the connection is in plain-text, indicating the use of Cloud Spanner emulator.

@elefeint
Copy link
Contributor

@magiccrafter Which authentication methods have you tried -- service account or default credentials?

Does this sample work for you while logged in with Application Default Credentials?

Having a sample going against the emulator is a good idea.

@elefeint elefeint added question Further information is requested P2 labels Mar 23, 2022
@magiccrafter
Copy link
Contributor Author

@elefeint Thanks for the quick reply. I've checked all samples in GoogleCloudPlatform/cloud-spanner-r2dbc and https://www.testcontainers.org/modules/gcloud/. Sadly they didn't help much in my case.
I haven't tried yet validating the code against any real spanner database. Once I find my way to make it work I'm more than happy to contribute a sample of some integration tests.

@magiccrafter
Copy link
Contributor Author

magiccrafter commented Mar 23, 2022

Which authentication methods have you tried -- service account or default credentials?

I've added usePlainText=true to the r2dbc URL which in theory we should bypass the credentials during the integration testing. After doing some lib debugging it seems that the credentials provider is correctly configured to NoCredentials (SpannerConnectionFactoryProvider#extractCredentials)

@elefeint
Copy link
Contributor

You would also have to set up an environment variable export SPANNER_EMULATOR_HOST=localhost:9010, so that the Spanner client library picks up the non-production host, similar to jdbc driver

Upvote #200 for allowing programmatic customization of emulator.

@magiccrafter
Copy link
Contributor Author

@elefeint Splendid. Many thanks for this hint. It works now. I was misled by this example: https://github.com/saturnism/testcontainers-gcloud-examples/blob/main/springboot/spanner-example/src/test/java/com/example/springboot/spanner/SpannerIntegrationTests.java
It may be the case that it was working before when gcp spring client starters were under the Spring Projects umbrella.

@elefeint
Copy link
Contributor

Spring Cloud GCP already has the change that's needed in #200, so a configuration property is sufficient there. As it happens, our team supports both projects.

@magiccrafter
Copy link
Contributor Author

Here is the workaround I came up with to bypass the requirement for SPANNER_EMULATOR_HOST env variable prior to the integration tests run:

@Testcontainers
@ExtendWith(SystemStubsExtension.class)
@ExtendWith(SpringExtension.class)
@SpringBootTest
@ActiveProfiles("it")
@DirtiesContext
public class SpannerIT {

    static final String PROJECT_ID = "nv-local";
    static final String INSTANCE_ID = "test-instance";
    static final String DATABASE_NAME = "trades";

    @Container
    private static final SpannerEmulatorContainer spannerContainer =
        new SpannerEmulatorContainer(
            DockerImageName.parse("gcr.io/cloud-spanner-emulator/emulator").withTag("1.4.1"));

    @SystemStub
    private static EnvironmentVariables environmentVariables;

    @Autowired
    ConnectionFactory connectionFactory;

    @DynamicPropertySource
    static void properties(DynamicPropertyRegistry r) {
        environmentVariables.set("SPANNER_EMULATOR_HOST", spannerContainer.getEmulatorGrpcEndpoint());
        r.add("spring.r2dbc.url", () -> "r2dbc:cloudspanner://" +
            "/projects/" + PROJECT_ID + "/instances/" + INSTANCE_ID + "/databases/" + DATABASE_NAME);
    }

This way, we can have the Spanner emulator in Testcontainers without starting the spanner emulator separately and mapping a concrete port. The complete source code can be found here.

@elefeint Many thanks for pointing out the direction. After successfully embedding the above in my POC app, I will contribute a different sample with the above setup. It is excellent that the spanner emulator doesn't have to be running to write or run integration tests locally or anywhere else.

@elefeint
Copy link
Contributor

That's very cool, thank you!
I am going to rename this issue and leave it open.

@elefeint elefeint changed the title UNAUTHENTICATED: Request is missing required authentication credential Provide a sample application using Spanner emulator Mar 24, 2022
@elefeint elefeint added documentation and removed question Further information is requested labels Mar 24, 2022
@burkedavison burkedavison added the enhancement New feature or request label Feb 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants