Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CELEBORN-977] Support RocksDB as recover DB backend #1913

Closed
wants to merge 8 commits into from

Conversation

cxzl25
Copy link
Contributor

@cxzl25 cxzl25 commented Sep 14, 2023

What changes were proposed in this pull request?

Why are the changes needed?

LevelDB does not support mac arm version.

java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /private/var/folders/tc/r2n_8g6j4731h7clfqwntg880000gn/T/libleveldbjni-64-1-4616234670453989010.8: dlopen(/private/var/folders/tc/r2n_8g6j4731h7clfqwntg880000gn/T/libleveldbjni-64-1-4616234670453989010.8, 0x0001): tried: '/private/var/folders/tc/r2n_8g6j4731h7clfqwntg880000gn/T/libleveldbjni-64-1-4616234670453989010.8' (fat file, but missing compatible architecture (have 'x86_64,i386', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/private/var/folders/tc/r2n_8g6j4731h7clfqwntg880000gn/T/libleveldbjni-64-1-4616234670453989010.8' (no such file), '/private/var/folders/tc/r2n_8g6j4731h7clfqwntg880000gn/T/libleveldbjni-64-1-4616234670453989010.8' (fat file, but missing compatible architecture (have 'x86_64,i386', need 'arm64'))]
  	at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:182)
  	at org.fusesource.hawtjni.runtime.Library.load(Library.java:140)
  	at org.fusesource.leveldbjni.JniDBFactory.<clinit>(JniDBFactory.java:48)
  	at org.apache.celeborn.service.deploy.worker.shuffledb.LevelDBProvider.initLevelDB(LevelDBProvider.java:49)
  	at org.apache.celeborn.service.deploy.worker.shuffledb.DBProvider.initDB(DBProvider.java:30)
  	at org.apache.celeborn.service.deploy.worker.storage.StorageManager.<init>(StorageManager.scala:197)
  	at org.apache.celeborn.service.deploy.worker.Worker.<init>(Worker.scala:109)
  	at org.apache.celeborn.service.deploy.worker.Worker$.main(Worker.scala:734)
  	at org.apache.celeborn.service.deploy.worker.Worker.main(Worker.scala)

The released leveldbjni-all for org.fusesource.leveldbjni does not support AArch64 Linux, we need to use org.openlabtesting.leveldbjni.

See https://issues.apache.org/jira/browse/HADOOP-16614

Does this PR introduce any user-facing change?

No.

How was this patch tested?

local test

@codecov
Copy link

codecov bot commented Sep 14, 2023

Codecov Report

Merging #1913 (6700a3b) into main (e0c00ec) will decrease coverage by 0.00%.
Report is 5 commits behind head on main.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1913      +/-   ##
==========================================
- Coverage   46.61%   46.61%   -0.00%     
==========================================
  Files         164      164              
  Lines       10247    10256       +9     
  Branches      937      937              
==========================================
+ Hits         4776     4780       +4     
- Misses       5158     5160       +2     
- Partials      313      316       +3     
Files Changed Coverage Δ
...cala/org/apache/celeborn/common/CelebornConf.scala 87.50% <100.00%> (+0.05%) ⬆️

... and 3 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@cxzl25 cxzl25 changed the title [CELEBORN-977] Support RocksDB as recover DB Backend [CELEBORN-977] Support RocksDB as recover DB backend Sep 14, 2023
@pan3793
Copy link
Member

pan3793 commented Sep 15, 2023

LICENSE related stuff need to update

LICENSE Outdated Show resolved Hide resolved
@pan3793 pan3793 closed this in beed2a8 Sep 19, 2023
@pan3793
Copy link
Member

pan3793 commented Sep 19, 2023

Merged to main

@waitinfuture
Copy link
Contributor

LGTM, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants