Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] MaxMessageSize Reached When Querying ICEBERG Table with Kyuubi 1.9 #6594

Closed
3 of 4 tasks
zhanghe-git opened this issue Aug 7, 2024 · 2 comments
Closed
3 of 4 tasks

[Bug] MaxMessageSize Reached When Querying ICEBERG Table with Kyuubi 1.9 #6594

zhanghe-git opened this issue Aug 7, 2024 · 2 comments
Labels
kind:bug This is a clearly a bug priority:major

Comments

@zhanghe-git
Copy link

zhanghe-git commented Aug 7, 2024 •
edited
Loading

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the bug

I encountered an issue when querying an ICEBERG table using a self-built Kyuubi (version 1.9.1) with Spark.

The query involves retrieving a single row of data that is 100MB in size.

While the query executes successfully on the Spark side, the Kyuubi side reports a MaxMessageSize reached error.

I attempted to resolve this by adjusting the kyuubi.frontend.thrift.max.message.size parameter, but the error persists.

It seems that some max size limit is being reached in Kyuubi. How can this be adjusted?

Additional information: When using Alibaba Cloud's EMR with its built-in Kyuubi (version 1.7.1), the query returns successfully. (My kyuubi-defaults configuration is consistent with Alibaba Cloud's setup.)

Affects Version(s)

1.9.1

Kyuubi Server Log Output

2024-08-07 16:29:53.048 INFO KyuubiSessionManager-exec-pool: Thread-178 org.apache.kyuubi.operation.ExecuteStatement: Processing he.zhang's query[05f5c252-9420-43b2-81d1-f9b7596f343b]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.113 seconds
2024-08-07 16:29:53.059 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.operation.ExecuteStatement: Processing he.zhang's query[05f5c252-9420-43b2-81d1-f9b7596f343b]: FINISHED_STATE -> CLOSED_STATE, time taken: 0.124 seconds
2024-08-07 16:29:53.080 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:05 F5 C2 52 94 20 43 B2 81 D1 F9 B7 59 6F 34 3B, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side
2024-08-07 16:29:53.090 INFO KyuubiSessionManager-exec-pool: Thread-179 org.apache.kyuubi.operation.ExecuteStatement: Processing he.zhang's query[4619304f-68e8-41d6-94e6-64b8d70913c5]: PENDING_STATE -> RUNNING_STATE, statement:
select * from db8822.zlc_test_person_003  where id='ee4c83fa-4e8a-4a12-b3ce-b055a2ab595b'
2024-08-07 16:29:58.094 INFO KyuubiSessionManager-exec-pool: Thread-179 org.apache.kyuubi.operation.ExecuteStatement: Query[4619304f-68e8-41d6-94e6-64b8d70913c5] in RUNNING_STATE
2024-08-07 16:30:02.205 INFO KyuubiRestFrontendService-182 org.apache.kyuubi.server.http.authentication.AuthenticationAuditLogger: user=anonymous(auth:BASIC)	ip=10.69.11.32	proxyIp=null	method=GET	uri=/api/v1/sessions/count	params=null	protocol=HTTP/1.1	status=200
2024-08-07 16:30:03.097 INFO KyuubiSessionManager-exec-pool: Thread-179 org.apache.kyuubi.operation.ExecuteStatement: Query[4619304f-68e8-41d6-94e6-64b8d70913c5] in RUNNING_STATE
2024-08-07 16:30:03.692 INFO KyuubiSessionManager-exec-pool: Thread-179 org.apache.kyuubi.operation.ExecuteStatement: Query[4619304f-68e8-41d6-94e6-64b8d70913c5] in FINISHED_STATE
2024-08-07 16:30:03.692 INFO KyuubiSessionManager-exec-pool: Thread-179 org.apache.kyuubi.operation.ExecuteStatement: Processing he.zhang's query[4619304f-68e8-41d6-94e6-64b8d70913c5]: RUNNING_STATE -> FINISHED_STATE, time taken: 10.602 seconds
2024-08-07 16:30:04.141 ERROR KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Error fetching results:
org.apache.kyuubi.shaded.thrift.transport.TTransportException: MaxMessageSize reached
	at org.apache.kyuubi.shaded.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
	at org.apache.kyuubi.shaded.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
	at org.apache.kyuubi.shaded.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
	at org.apache.kyuubi.shaded.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
	at org.apache.kyuubi.shaded.thrift.transport.TTransport.readAll(TTransport.java:109)
	at org.apache.kyuubi.shaded.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:434)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TBinaryColumn$TBinaryColumnStandardScheme.read(TBinaryColumn.java:431)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TBinaryColumn$TBinaryColumnStandardScheme.read(TBinaryColumn.java:411)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TBinaryColumn.read(TBinaryColumn.java:345)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TColumn.standardSchemeReadValue(TColumn.java:317)
	at org.apache.kyuubi.shaded.thrift.TUnion$TUnionStandardScheme.read(TUnion.java:228)
	at org.apache.kyuubi.shaded.thrift.TUnion$TUnionStandardScheme.read(TUnion.java:217)
	at org.apache.kyuubi.shaded.thrift.TUnion.read(TUnion.java:138)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TRowSet$TRowSetStandardScheme.read(TRowSet.java:718)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TRowSet$TRowSetStandardScheme.read(TRowSet.java:670)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TRowSet.read(TRowSet.java:580)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TFetchResultsResp$TFetchResultsRespStandardScheme.read(TFetchResultsResp.java:495)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TFetchResultsResp$TFetchResultsRespStandardScheme.read(TFetchResultsResp.java:463)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TFetchResultsResp.read(TFetchResultsResp.java:385)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$FetchResults_result$FetchResults_resultStandardScheme.read(TCLIService.java:16912)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$FetchResults_result$FetchResults_resultStandardScheme.read(TCLIService.java:16897)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$FetchResults_result.read(TCLIService.java:16844)
	at org.apache.kyuubi.shaded.thrift.TServiceClient.receiveBase(TServiceClient.java:88)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Client.recv_FetchResults(TCLIService.java:548)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Client.FetchResults(TCLIService.java:535)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$fetchResults$1(KyuubiSyncThriftClient.scala:437)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$withLockAcquiredAsyncRequest$2(KyuubiSyncThriftClient.scala:167)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
2024-08-07 16:30:04.146 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.operation.ExecuteStatement: Processing he.zhang's query[4619304f-68e8-41d6-94e6-64b8d70913c5]: FINISHED_STATE -> CLOSED_STATE, time taken: 11.056 seconds
2024-08-07 16:30:04.147 WARN KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.operation.ExecuteStatement: Error closing THandleIdentifier(guid:46 19 30 4F 68 E8 41 D6 94 E6 64 B8 D7 09 13 C5, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38): MaxMessageSize reached
org.apache.kyuubi.shaded.thrift.transport.TTransportException: MaxMessageSize reached
	at org.apache.kyuubi.shaded.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
	at org.apache.kyuubi.shaded.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
	at org.apache.kyuubi.shaded.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
	at org.apache.kyuubi.shaded.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
	at org.apache.kyuubi.shaded.thrift.transport.TTransport.readAll(TTransport.java:109)
	at org.apache.kyuubi.shaded.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464)
	at org.apache.kyuubi.shaded.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:362)
	at org.apache.kyuubi.shaded.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:245)
	at org.apache.kyuubi.shaded.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Client.recv_CloseOperation(TCLIService.java:502)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:489)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$closeOperation$1(KyuubiSyncThriftClient.scala:413)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$withLockAcquiredAsyncRequest$2(KyuubiSyncThriftClient.scala:167)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
2024-08-07 16:30:04.469 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Session [SessionHandle [b921325e-0066-4cde-8c9e-8ca43275c2d0]] disconnected without closing properly, close it now
2024-08-07 16:30:04.470 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.session.KyuubiSessionManager: he.zhang's KyuubiSessionImpl with SessionHandle [b921325e-0066-4cde-8c9e-8ca43275c2d0] is closed, current opening sessions 0
2024-08-07 16:30:04.474 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.operation.LaunchEngine: Processing he.zhang's query[9f58a672-629e-4b98-8c3a-f1fb756fd1ca]: FINISHED_STATE -> CLOSED_STATE, time taken: 43.801 seconds
2024-08-07 16:30:04.516 ERROR KyuubiTBinaryFrontendHandler-Pool: Thread-169 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Failed closing session
org.apache.kyuubi.KyuubiSQLException: Error while cleaning up the engine resources
	at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.closeSession(KyuubiSyncThriftClient.scala:237)
	at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:281)
	at org.apache.kyuubi.session.SessionManager.closeSession(SessionManager.scala:134)
	at org.apache.kyuubi.session.KyuubiSessionManager.closeSession(KyuubiSessionManager.scala:129)

Kyuubi Engine Log Output

No response

Kyuubi Server Configurations

No response

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

  • Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
  • No. I cannot submit a PR at this time.
@zhanghe-git zhanghe-git added kind:bug This is a clearly a bug priority:major labels Aug 7, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 7, 2024

Hello @zhanghe-git,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.

@pan3793 pan3793 changed the title [Bug] MaxMessageSize Reached When Querying ICEBERG Table with Kyuubi 1.9.1 [Bug] MaxMessageSize Reached When Querying ICEBERG Table with Kyuubi 1.9 Aug 7, 2024
@pan3793
Copy link
Member

pan3793 commented Aug 7, 2024

This is caused by Thrift upgrading, from 0.9.3-1 to 0.16.0.

We should make the thrift client's max message size configurable

pan3793 added a commit to pan3793/kyuubi that referenced this issue Aug 20, 2024
@pan3793
[KYUUBI apache#6594] Port HIVE-26633: Make thrift client maxMessageSi…
ff6b521 
…ze configurable
@pan3793 pan3793 mentioned this issue Aug 20, 2024
Closed
4 tasks
pan3793 added a commit to pan3793/kyuubi that referenced this issue Aug 21, 2024
@pan3793
[KYUUBI apache#6594] Port HIVE-26633: Make thrift client maxMessageSi…
e4841c8 
…ze configurable
pan3793 added a commit that referenced this issue Aug 27, 2024
@pan3793
[KYUUBI #6594] Port HIVE-26633: Make thrift client maxMessageSize con…
eb03be6 
…figurable

Fix #6594.

This PR ports HIVE-26633(apache/hive#3674): Make thrift client maxMessageSize configurable to fix a regression after upgrading Thrift 0.16 in 1.9.0.

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

---

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6631 from pan3793/thrift-max-size.

Closes #6594

e4841c8 [Cheng Pan] [KYUUBI #6594] Port HIVE-26633: Make thrift client maxMessageSize configurable

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 11de72f)
Signed-off-by: Cheng Pan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind:bug This is a clearly a bug priority:major
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[KYUUBI #6594] Port HIVE-26633: Make thrift client maxMessageSize configurable pan3793/kyuubi
2 participants
@zhanghe-git @pan3793