-
Notifications
You must be signed in to change notification settings - Fork 44
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* feat(spark): Refactoring datasources (#514) ### Reason for this PR By moving datasources under `org.apache.spark.sql` we are able to access private Spark API. Last time when I was trying to fully migrate datasources to V2 it was a blocker. Detailed motivation is in #493 ### What changes are included in this PR? Mostly refactoring. ### Are these changes tested? Unit tests are passed I manually checked the generated JARs:  ### Are there any user-facing changes? Mostly not because `GarDataSource` was left under the same package. Close #493 * feat(dev): Add release and verify scripts (#507) Reason for this PR Add scripts for developer or release manager to easily release version or verify a version. What changes are included in this PR? Add release and verify scripts related document is updated to website, see Update the release and verify document, and add development document incubator-graphar-website#18 Are these changes tested? yes Are there any user-facing changes? no --------- Signed-off-by: acezen <[email protected]> * chore: Bump to version v0.12.0 (Round 1) (#517) Signed-off-by: acezen <[email protected]> * chore: Add CHANGELOG.md (#513) Signed-off-by: acezen <[email protected]> * Introduce buf - v2 - buf.gen - buf On branch format-definition-dev Your branch is up to date with 'origin/format-definition-dev'. Changes to be committed: new file: buf.gen.yaml new file: buf.yaml modified: format/adjacent_list.proto modified: format/edge_info.proto modified: format/graph_info.proto modified: format/property_group.proto modified: format/types.proto modified: format/vertex_info.proto --------- Signed-off-by: acezen <[email protected]> Co-authored-by: Weibin Zeng <[email protected]>
- Loading branch information
1 parent
cbc7a6c
commit 22aa41f
Showing
59 changed files
with
963 additions
and
149 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| version: v2 | ||
| managed: | ||
| enabled: true | ||
| disable: | ||
| - file_option: java_package | ||
| plugins: | ||
| # Python classes | ||
| - remote: buf.build/protocolbuffers/python:v27.1 | ||
| out: pyspark/graphar_pyspark/proto/ | ||
| # Python headers for IDEs and MyPy | ||
| - remote: buf.build/protocolbuffers/pyi | ||
| out: pyspark/graphar_pyspark/proto/ | ||
| # Cpp | ||
| - remote: buf.build/protocolbuffers/cpp:v27.1 | ||
| out: cpp/src/proto | ||
| # Java | ||
| - remote: buf.build/protocolbuffers/java:v27.1 | ||
| out: maven-projects/info/src/main/java/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| version: v2 | ||
| modules: | ||
| - path: format |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| #!/bin/bash | ||
| # | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
|
|
||
| # A script to download test data for GraphAr | ||
|
|
||
| if [ -n "${GAR_TEST_DATA}" ]; then | ||
| if [[ ! -d "$GAR_TEST_DATA" ]]; then | ||
| echo "GAR_TEST_DATA is set but the directory does not exist, cloning the test data to $GAR_TEST_DATA" | ||
| git clone https://github.com/apache/incubator-graphar-testing.git "$GAR_TEST_DATA" --depth 1 || true | ||
| fi | ||
| else | ||
| echo "GAR_TEST_DATA is not set, cloning the test data to /tmp/graphar-testing" | ||
| git clone https://github.com/apache/incubator-graphar-testing.git /tmp/graphar-testing --depth 1 || true | ||
| echo "Test data has been cloned to /tmp/graphar-testing, please run" | ||
| echo " export GAR_TEST_DATA=/tmp/graphar-testing" | ||
| fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
|
|
||
| cmake | ||
| conda-forge::arrow-cpp=13.0.0 | ||
| make | ||
| clangxx_linux-64 | ||
| conda-forge::catch2=3.6.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
|
|
||
| maven | ||
| openjdk=11.0.13 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,119 @@ | ||
| #!/usr/bin/env python3 | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
|
|
||
| # Derived from Apache OpenDAL v0.45.1 | ||
| # https://github.com/apache/opendal/blob/5079125/scripts/release.py | ||
|
|
||
| import re | ||
| import subprocess | ||
| from pathlib import Path | ||
|
|
||
| ROOT_DIR = Path(__file__).parent.parent.parent | ||
|
|
||
| def get_package_version(): | ||
| major_version = None | ||
| minor_version = None | ||
| patch_version = None | ||
| major_pattern = re.compile(r'set\s*\(\s*GRAPHAR_MAJOR_VERSION\s+(\d+)\s*\)', re.IGNORECASE) | ||
| minor_pattern = re.compile(r'set\s*\(\s*GRAPHAR_MINOR_VERSION\s+(\d+)\s*\)', re.IGNORECASE) | ||
| patch_pattern = re.compile(r'set\s*\(\s*GRAPHAR_PATCH_VERSION\s+(\d+)\s*\)', re.IGNORECASE) | ||
|
|
||
| file_path = ROOT_DIR / "cpp/CMakeLists.txt" | ||
| with open(file_path, 'r') as file: | ||
| for line in file: | ||
| major_match = major_pattern.search(line) | ||
| minor_match = minor_pattern.search(line) | ||
| patch_match = patch_pattern.search(line) | ||
|
|
||
| if major_match: | ||
| major_version = major_match.group(1) | ||
| if minor_match: | ||
| minor_version = minor_match.group(1) | ||
| if patch_match: | ||
| patch_version = patch_match.group(1) | ||
|
|
||
| if major_version and minor_version and patch_version: | ||
| return f"{major_version}.{minor_version}.{patch_version}" | ||
| else: | ||
| return None | ||
|
|
||
| def archive_source_package(): | ||
| print(f"Archive source package started") | ||
|
|
||
| version = get_package_version() | ||
| assert version, "Failed to get the package version" | ||
| name = f"apache-graphar-{version}-incubating-src" | ||
|
|
||
| archive_command = [ | ||
| "git", | ||
| "archive", | ||
| "--prefix", | ||
| f"apache-graphar-{version}-incubating-src/", | ||
| "-o", | ||
| f"{ROOT_DIR}/dist/{name}.tar.gz", | ||
| "HEAD", | ||
| ] | ||
| subprocess.run( | ||
| archive_command, | ||
| cwd=ROOT_DIR, | ||
| check=True, | ||
| ) | ||
|
|
||
| print(f"Archive source package to dist/{name}.tar.gz") | ||
|
|
||
|
|
||
| def generate_signature(): | ||
| for i in Path(ROOT_DIR / "dist").glob("*.tar.gz"): | ||
| print(f"Generate signature for {i}") | ||
| subprocess.run( | ||
| ["gpg", "--yes", "--armor", "--output", f"{i}.asc", "--detach-sig", str(i)], | ||
| cwd=ROOT_DIR / "dist", | ||
| check=True, | ||
| ) | ||
|
|
||
| for i in Path(ROOT_DIR / "dist").glob("*.tar.gz"): | ||
| print(f"Check signature for {i}") | ||
| subprocess.run( | ||
| ["gpg", "--verify", f"{i}.asc", str(i)], cwd=ROOT_DIR / "dist", check=True | ||
| ) | ||
|
|
||
|
|
||
| def generate_checksum(): | ||
| for i in Path(ROOT_DIR / "dist").glob("*.tar.gz"): | ||
| print(f"Generate checksum for {i}") | ||
| subprocess.run( | ||
| ["sha512sum", str(i.relative_to(ROOT_DIR / "dist"))], | ||
| stdout=open(f"{i}.sha512", "w"), | ||
| cwd=ROOT_DIR / "dist", | ||
| check=True, | ||
| ) | ||
|
|
||
| for i in Path(ROOT_DIR / "dist").glob("*.tar.gz"): | ||
| print(f"Check checksum for {i}") | ||
| subprocess.run( | ||
| ["sha512sum", "--check", f"{str(i.relative_to(ROOT_DIR / 'dist'))}.sha512"], | ||
| cwd=ROOT_DIR / "dist", | ||
| check=True, | ||
| ) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| (ROOT_DIR / "dist").mkdir(exist_ok=True) | ||
| archive_source_package() | ||
| generate_signature() | ||
| generate_checksum() |
Oops, something went wrong.