Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement] [Bug] Modify the logic in the beeline script that specifies the KYUUBI_HOME environment variable. #5987

Closed
3 of 4 tasks
Emor-nj opened this issue Jan 17, 2024 · 0 comments
Closed
3 of 4 tasks

[Improvement] [Bug] Modify the logic in the beeline script that specifies the KYUUBI_HOME environment variable. #5987

Emor-nj opened this issue Jan 17, 2024 · 0 comments
Labels
kind:bug This is a clearly a bug priority:major

Comments

@Emor-nj
Copy link
Contributor

Emor-nj commented Jan 17, 2024 •
edited by pan3793
Loading

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the bug

When using Kyuubi in my company, there is a scenario where we create a symbolic link from ${KYUUBI_HOME}/bin/beeline to /usr/bin/kyuubi-beeline. This allows distinguishing its usage from Hive's beeline, and users can globally use kyuubi-beeline without specifying a directory. However, when executing kyuubi-beeline, an error occurs, stating that it cannot find load-kyuubi-env.sh. The specific error message is as follows:

/bin/kyuubi-beeline: line 25: //bin/load-kyuubi-env.sh: No such file or directory.

Personally, I believe the logic of exporting KYUUBI_HOME in ${KYUUBI_HOME}/bin/beeline should be modified. If the environment has already declared KYUUBI_HOME, it should use the pre-declared KYUUBI_HOME instead of forcefully specifying the parent directory of the current one.

Affects Version(s)

master/1.8.0

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

No response

Kyuubi Server Configurations

No response

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

  • Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
  • No. I cannot submit a PR at this time.
@Emor-nj Emor-nj added kind:bug This is a clearly a bug priority:major labels Jan 17, 2024
@Emor-nj Emor-nj mentioned this issue Jan 17, 2024
Closed
4 tasks
pan3793 pushed a commit that referenced this issue Jan 17, 2024
@Emor-nj @pan3793
[KYUUBI #5987] Enhance KYUUBI_HOME detection in shell script to handl…
0c3d2c8 
…e softlink cases

# 🔍 Description
## Issue References 🔗

This pull request fixes #5987
Modify the logic in the beeline script that specifies the KYUUBI_HOME environment variable.

## Describe Your Solution 🔧

 If the environment has already declared KYUUBI_HOME, it should use the pre-declared KYUUBI_HOME instead of forcefully specifying the parent directory of the current one.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️
1. set soft link:
 `ln -s /opt/soft/kyuubi/kyuubi/bin/beeline /usr/bin/kyuubi-beeline`
2. set env like:
`export KYUUBI_HOME=/opt/soft/kyuubi/kyuubi`
3. run kyuubi-beeline:
`kyuubi-beeline -u "jdbc:hive2://xx.xx.xx.xx:2181,xx.xx.xx.xx:2181,xx.xx.xx.xx:2181/tmp;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;principal=hive/_HOSTxxx"`
4. command will fail:
`/bin/kyuubi-beeline: line 27: //bin/load-kyuubi-env.sh: No such file or directory
Error: JAVA_HOME IS NOT SET! CANNOT PROCEED.`

#### Behavior With This Pull Request 🎉
1. set soft link:
 `ln -s /opt/soft/kyuubi/kyuubi/bin/beeline /usr/bin/kyuubi-beeline`
2. set env like:
`export KYUUBI_HOME=/opt/soft/kyuubi/kyuubi`
3. run kyuubi-beeline:
`kyuubi-beeline -u "jdbc:hive2://xx.xx.xx.xx:2181,xx.xx.xx.xx:2181,xx.xx.xx.xx:2181/tmp;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;principal=hive/_HOSTxxx"`
4. command will success
#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #5988 from Emor-nj/kyuubi-beeline.

Closes #5987

3a988e0 [Emor-nj] fix KYUUBI_HOME is always empty in load-kyuubi-env.sh
cc70387 [Emor-nj] fix load-kyuubi-env.sh
a101428 [Emor-nj] Remove unnecessary exports.
f48db3f [Emor-nj] uniformly modify all related scripts.
35215b9 [Emor-nj] Modify the logic in the beeline script that specifies the KYUUBI_HOME environment variable.

Authored-by: Emor-nj <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 7b38889)
Signed-off-by: Cheng Pan <[email protected]>
pan3793 added a commit to pan3793/kyuubi that referenced this issue Jan 22, 2024
@pan3793
[KYUUBI apache#5987] Always export KYUUBI_HOME in load-kyuubi-env.sh
a4e40b6
@pan3793 pan3793 mentioned this issue Jan 22, 2024
Closed
4 tasks
pan3793 added a commit that referenced this issue Jan 22, 2024
@pan3793
[KYUUBI #5987] Always export KYUUBI_HOME in load-kyuubi-env.sh
ac388d1 
# 🔍 Description
## Issue References 🔗

This is a follow-up of #5987, after applying #5987, I found a regression, the Kyuubi Server Java process can not see KYUUBI_HOME env if we don't declare KYUUBI_HOME as an env var explicitly, this breaks the main resources jar discovery of the engine, and eventually fails the engine bootstrap.

## Describe Your Solution 🔧

Restore `export KYUUBI_HOME` logic in `load-kyuubi-env.sh`

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Build and start Kyuubi Server.
```
export SPARK_HOME=/Users/chengpan/app/spark-3.5.0-bin-hadoop3
build/dist --spark-provided --flink-provided --hive-provided
dist/bin/kyuubi run
```

Open session to launch a Spark engine
```
dist/bin/beeline -u 'jdbc:hive2://0.0.0.0:10009/default'
```

#### Behavior Without This Pull Request ⚰️

A wrong `kyuubi-spark-sql-engine_2.12-1.9.0-SNAPSHOT.jar` location was found.
```
/Users/chengpan/app/spark-3.5.0-bin-hadoop3/bin/spark-submit \
        ...
        --proxy-user chengpan /Users/chengpan/Projects/apache-kyuubi/dist/jars/externals/kyuubi-spark-sql-engine/target/kyuubi-spark-sql-engine_2.12-1.9.0-SNAPSHOT.jar
```

#### Behavior With This Pull Request 🎉

The correct `kyuubi-spark-sql-engine_2.12-1.9.0-SNAPSHOT.jar` location was found.
```
/Users/chengpan/app/spark-3.5.0-bin-hadoop3/bin/spark-submit \
        ...
        --proxy-user chengpan /Users/chengpan/Projects/apache-kyuubi/dist/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.9.0-SNAPSHOT.jar
```

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6004 from pan3793/5987-followup.

Closes #5987

a4e40b6 [Cheng Pan] [KYUUBI #5987] Always export KYUUBI_HOME in load-kyuubi-env.sh

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
pan3793 added a commit that referenced this issue Jan 22, 2024
@pan3793
[KYUUBI #5987] Always export KYUUBI_HOME in load-kyuubi-env.sh
6397fb6 
# 🔍 Description
## Issue References 🔗

This is a follow-up of #5987, after applying #5987, I found a regression, the Kyuubi Server Java process can not see KYUUBI_HOME env if we don't declare KYUUBI_HOME as an env var explicitly, this breaks the main resources jar discovery of the engine, and eventually fails the engine bootstrap.

## Describe Your Solution 🔧

Restore `export KYUUBI_HOME` logic in `load-kyuubi-env.sh`

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Build and start Kyuubi Server.
```
export SPARK_HOME=/Users/chengpan/app/spark-3.5.0-bin-hadoop3
build/dist --spark-provided --flink-provided --hive-provided
dist/bin/kyuubi run
```

Open session to launch a Spark engine
```
dist/bin/beeline -u 'jdbc:hive2://0.0.0.0:10009/default'
```

#### Behavior Without This Pull Request ⚰️

A wrong `kyuubi-spark-sql-engine_2.12-1.9.0-SNAPSHOT.jar` location was found.
```
/Users/chengpan/app/spark-3.5.0-bin-hadoop3/bin/spark-submit \
        ...
        --proxy-user chengpan /Users/chengpan/Projects/apache-kyuubi/dist/jars/externals/kyuubi-spark-sql-engine/target/kyuubi-spark-sql-engine_2.12-1.9.0-SNAPSHOT.jar
```

#### Behavior With This Pull Request 🎉

The correct `kyuubi-spark-sql-engine_2.12-1.9.0-SNAPSHOT.jar` location was found.
```
/Users/chengpan/app/spark-3.5.0-bin-hadoop3/bin/spark-submit \
        ...
        --proxy-user chengpan /Users/chengpan/Projects/apache-kyuubi/dist/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.9.0-SNAPSHOT.jar
```

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6004 from pan3793/5987-followup.

Closes #5987

a4e40b6 [Cheng Pan] [KYUUBI #5987] Always export KYUUBI_HOME in load-kyuubi-env.sh

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit ac388d1)
Signed-off-by: Cheng Pan <[email protected]>
zhaohehuhu pushed a commit to zhaohehuhu/incubator-kyuubi that referenced this issue Feb 5, 2024
@Emor-nj @zhaohehuhu
[KYUUBI apache#5987] Enhance KYUUBI_HOME detection in shell script to…
5a230f6 
… handle softlink cases

# 🔍 Description
## Issue References 🔗

This pull request fixes apache#5987
Modify the logic in the beeline script that specifies the KYUUBI_HOME environment variable.

## Describe Your Solution 🔧

 If the environment has already declared KYUUBI_HOME, it should use the pre-declared KYUUBI_HOME instead of forcefully specifying the parent directory of the current one.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️
1. set soft link:
 `ln -s /opt/soft/kyuubi/kyuubi/bin/beeline /usr/bin/kyuubi-beeline`
2. set env like:
`export KYUUBI_HOME=/opt/soft/kyuubi/kyuubi`
3. run kyuubi-beeline:
`kyuubi-beeline -u "jdbc:hive2://xx.xx.xx.xx:2181,xx.xx.xx.xx:2181,xx.xx.xx.xx:2181/tmp;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;principal=hive/_HOSTxxx"`
4. command will fail:
`/bin/kyuubi-beeline: line 27: //bin/load-kyuubi-env.sh: No such file or directory
Error: JAVA_HOME IS NOT SET! CANNOT PROCEED.`

#### Behavior With This Pull Request 🎉
1. set soft link:
 `ln -s /opt/soft/kyuubi/kyuubi/bin/beeline /usr/bin/kyuubi-beeline`
2. set env like:
`export KYUUBI_HOME=/opt/soft/kyuubi/kyuubi`
3. run kyuubi-beeline:
`kyuubi-beeline -u "jdbc:hive2://xx.xx.xx.xx:2181,xx.xx.xx.xx:2181,xx.xx.xx.xx:2181/tmp;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;principal=hive/_HOSTxxx"`
4. command will success
#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes apache#5988 from Emor-nj/kyuubi-beeline.

Closes apache#5987

3a988e0 [Emor-nj] fix KYUUBI_HOME is always empty in load-kyuubi-env.sh
cc70387 [Emor-nj] fix load-kyuubi-env.sh
a101428 [Emor-nj] Remove unnecessary exports.
f48db3f [Emor-nj] uniformly modify all related scripts.
35215b9 [Emor-nj] Modify the logic in the beeline script that specifies the KYUUBI_HOME environment variable.

Authored-by: Emor-nj <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
zhaohehuhu pushed a commit to zhaohehuhu/incubator-kyuubi that referenced this issue Feb 5, 2024
@pan3793 @zhaohehuhu
[KYUUBI apache#5987] Always export KYUUBI_HOME in load-kyuubi-env.sh
2499914 
# 🔍 Description
## Issue References 🔗

This is a follow-up of apache#5987, after applying apache#5987, I found a regression, the Kyuubi Server Java process can not see KYUUBI_HOME env if we don't declare KYUUBI_HOME as an env var explicitly, this breaks the main resources jar discovery of the engine, and eventually fails the engine bootstrap.

## Describe Your Solution 🔧

Restore `export KYUUBI_HOME` logic in `load-kyuubi-env.sh`

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Build and start Kyuubi Server.
```
export SPARK_HOME=/Users/chengpan/app/spark-3.5.0-bin-hadoop3
build/dist --spark-provided --flink-provided --hive-provided
dist/bin/kyuubi run
```

Open session to launch a Spark engine
```
dist/bin/beeline -u 'jdbc:hive2://0.0.0.0:10009/default'
```

#### Behavior Without This Pull Request ⚰️

A wrong `kyuubi-spark-sql-engine_2.12-1.9.0-SNAPSHOT.jar` location was found.
```
/Users/chengpan/app/spark-3.5.0-bin-hadoop3/bin/spark-submit \
        ...
        --proxy-user chengpan /Users/chengpan/Projects/apache-kyuubi/dist/jars/externals/kyuubi-spark-sql-engine/target/kyuubi-spark-sql-engine_2.12-1.9.0-SNAPSHOT.jar
```

#### Behavior With This Pull Request 🎉

The correct `kyuubi-spark-sql-engine_2.12-1.9.0-SNAPSHOT.jar` location was found.
```
/Users/chengpan/app/spark-3.5.0-bin-hadoop3/bin/spark-submit \
        ...
        --proxy-user chengpan /Users/chengpan/Projects/apache-kyuubi/dist/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.9.0-SNAPSHOT.jar
```

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes apache#6004 from pan3793/5987-followup.

Closes apache#5987

a4e40b6 [Cheng Pan] [KYUUBI apache#5987] Always export KYUUBI_HOME in load-kyuubi-env.sh

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
zhaohehuhu pushed a commit to zhaohehuhu/incubator-kyuubi that referenced this issue Mar 21, 2024
@Emor-nj @zhaohehuhu
[KYUUBI apache#5987] Enhance KYUUBI_HOME detection in shell script to…
2cb82dd 
… handle softlink cases

# 🔍 Description
## Issue References 🔗

This pull request fixes apache#5987
Modify the logic in the beeline script that specifies the KYUUBI_HOME environment variable.

## Describe Your Solution 🔧

 If the environment has already declared KYUUBI_HOME, it should use the pre-declared KYUUBI_HOME instead of forcefully specifying the parent directory of the current one.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️
1. set soft link:
 `ln -s /opt/soft/kyuubi/kyuubi/bin/beeline /usr/bin/kyuubi-beeline`
2. set env like:
`export KYUUBI_HOME=/opt/soft/kyuubi/kyuubi`
3. run kyuubi-beeline:
`kyuubi-beeline -u "jdbc:hive2://xx.xx.xx.xx:2181,xx.xx.xx.xx:2181,xx.xx.xx.xx:2181/tmp;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;principal=hive/_HOSTxxx"`
4. command will fail:
`/bin/kyuubi-beeline: line 27: //bin/load-kyuubi-env.sh: No such file or directory
Error: JAVA_HOME IS NOT SET! CANNOT PROCEED.`

#### Behavior With This Pull Request 🎉
1. set soft link:
 `ln -s /opt/soft/kyuubi/kyuubi/bin/beeline /usr/bin/kyuubi-beeline`
2. set env like:
`export KYUUBI_HOME=/opt/soft/kyuubi/kyuubi`
3. run kyuubi-beeline:
`kyuubi-beeline -u "jdbc:hive2://xx.xx.xx.xx:2181,xx.xx.xx.xx:2181,xx.xx.xx.xx:2181/tmp;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;principal=hive/_HOSTxxx"`
4. command will success
#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes apache#5988 from Emor-nj/kyuubi-beeline.

Closes apache#5987

3a988e0 [Emor-nj] fix KYUUBI_HOME is always empty in load-kyuubi-env.sh
cc70387 [Emor-nj] fix load-kyuubi-env.sh
a101428 [Emor-nj] Remove unnecessary exports.
f48db3f [Emor-nj] uniformly modify all related scripts.
35215b9 [Emor-nj] Modify the logic in the beeline script that specifies the KYUUBI_HOME environment variable.

Authored-by: Emor-nj <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
zhaohehuhu pushed a commit to zhaohehuhu/incubator-kyuubi that referenced this issue Mar 21, 2024
@pan3793 @zhaohehuhu
[KYUUBI apache#5987] Always export KYUUBI_HOME in load-kyuubi-env.sh
5b6171b 
# 🔍 Description
## Issue References 🔗

This is a follow-up of apache#5987, after applying apache#5987, I found a regression, the Kyuubi Server Java process can not see KYUUBI_HOME env if we don't declare KYUUBI_HOME as an env var explicitly, this breaks the main resources jar discovery of the engine, and eventually fails the engine bootstrap.

## Describe Your Solution 🔧

Restore `export KYUUBI_HOME` logic in `load-kyuubi-env.sh`

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Build and start Kyuubi Server.
```
export SPARK_HOME=/Users/chengpan/app/spark-3.5.0-bin-hadoop3
build/dist --spark-provided --flink-provided --hive-provided
dist/bin/kyuubi run
```

Open session to launch a Spark engine
```
dist/bin/beeline -u 'jdbc:hive2://0.0.0.0:10009/default'
```

#### Behavior Without This Pull Request ⚰️

A wrong `kyuubi-spark-sql-engine_2.12-1.9.0-SNAPSHOT.jar` location was found.
```
/Users/chengpan/app/spark-3.5.0-bin-hadoop3/bin/spark-submit \
        ...
        --proxy-user chengpan /Users/chengpan/Projects/apache-kyuubi/dist/jars/externals/kyuubi-spark-sql-engine/target/kyuubi-spark-sql-engine_2.12-1.9.0-SNAPSHOT.jar
```

#### Behavior With This Pull Request 🎉

The correct `kyuubi-spark-sql-engine_2.12-1.9.0-SNAPSHOT.jar` location was found.
```
/Users/chengpan/app/spark-3.5.0-bin-hadoop3/bin/spark-submit \
        ...
        --proxy-user chengpan /Users/chengpan/Projects/apache-kyuubi/dist/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.9.0-SNAPSHOT.jar
```

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes apache#6004 from pan3793/5987-followup.

Closes apache#5987

a4e40b6 [Cheng Pan] [KYUUBI apache#5987] Always export KYUUBI_HOME in load-kyuubi-env.sh

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind:bug This is a clearly a bug priority:major
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[KYUUBI #5987] Enhance KYUUBI_HOME detection in shell script to handle softlink cases Emor-nj/kyuubi
1 participant
@Emor-nj