Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#5982] feat (gvfs-fuse): Implement an Gravitino fileset file system #5983

Closed
wants to merge 71 commits into from

Conversation

diqiu50
Copy link
Contributor

@diqiu50 diqiu50 commented Dec 25, 2024

What changes were proposed in this pull request?

Implement an Gravitino fileset file system, Support mount catalog/schema/fileset to local directory

Why are the changes needed?

Fix: #5982

Does this PR introduce any user-facing change?

No

How was this patch tested?

UT and IT

justinmclean and others added 30 commits December 6, 2024 16:05
### What changes were proposed in this pull request?

Add column commands to the Gravitino CLI

### Why are the changes needed?

Expand Gravitino CLI.

Fix: apache#5383

### Does this PR introduce _any_ user-facing change?

No, but add extra commands.

### How was this patch tested?

locally.
…5676)

### What changes were proposed in this pull request?

1. Release a new docker Hive image to support GCS.
2. Add related test based on the new image.
 
### Why are the changes needed?

For users convenience. 

Fix: apache#5673 

### Does this PR introduce _any_ user-facing change?

N/A

### How was this patch tested?

new IT `CatalogHiveGCSIT`

---------

Co-authored-by: Jerry Shao <[email protected]>
…apache#5740)

### What changes were proposed in this pull request?

fix camelCase method names from `getfieldName` to `getFieldName`


### Why are the changes needed?

`getfieldName` is not standardized in naming


### Does this PR introduce _any_ user-facing change?

none

### How was this patch tested?

TestTableOpsUtils
…tino and paimon tables in the paimon catalog document. (apache#5741)

### What changes were proposed in this pull request?

Fix the binary data type mapping error of the gravitino and paimon
tables in the paimon catalog document.
…e#5690)

### What changes were proposed in this pull request?

This is the first PR of credential vending for Gravitino server to add
credential API , please refer to apache#5682 for the overall workflow.
Examples to use the API:

```java
    Catalog catalog = metalake.loadCatalog(catalogName);
    Credential[] credentials =
        catalog.supportsCredentials().getCredentials();

    Credential credential =
        catalog.supportsCredentials().getCredential(“s3-token”);
```

```java
    Fileset fileset = catalog.asFilesetCatalog().loadFileset(filesetIdent);
    Credential[] credentials =
        fileset.supportsCredentials().getCredentials();
    Credential credential =
       fileset.supportsCredentials().getCredential(“s3-token”);
 
```


### Why are the changes needed?

Fix: apache#5619 

### Does this PR introduce _any_ user-facing change?
no

### How was this patch tested?
using draft PR to do E2E test
…bles in the Gravitino CLI (apache#5709)

### What changes were proposed in this pull request?

Add --sortorder option to display sort order information on Tables.

### Why are the changes needed?

Closes: apache#5279

### Does this PR introduce _any_ user-facing change?

Yes, it adds the --sortorder option to CLI

### How was this patch tested?

Compiled and tested locally.
)

### What changes were proposed in this pull request?

 Add the ability to create a table via the Gravitino CLI.

### Why are the changes needed?

so we can create tables via the CLI.

Fix: apache#5383

### Does this PR introduce _any_ user-facing change?

No, but it expands on CLI commands.

### How was this patch tested?

Tested locally.
…he#5749)

### What changes were proposed in this pull request?

add note for Doris type mapping

### Why are the changes needed?

Doris does support nested type in JDBC

Fix: apache#5098 

### Does this PR introduce _any_ user-facing change?

no

### How was this patch tested?

no
…on plugin (apache#5751)

### What changes were proposed in this pull request?

Add metalake name variable in the `BaseAuthorization::newPlugin()`
params.

### Why are the changes needed?

Fix: apache#5750

### Does this PR introduce _any_ user-facing change?

N/A

### How was this patch tested?

Add ITs.
…ache#5752)

### What changes were proposed in this pull request?

Expand Gravitino CLI support.

### Why are the changes needed?

Add support for all current catalogs.

Fix: apache#5560

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

Tested locally.
### What changes were proposed in this pull request?

Adjust table properties documentation.

### Why are the changes needed?

Add missing information about certain immutable fields and improve
readability.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Previewed the markdown output.
…ient (apache#5646)

### What changes were proposed in this pull request?
Implement expression from java, including:
- Expression.java
- FunctionExpression.java
- NamedReference.java
- UnparsedExpression.java
- literals/

convert to python client, and add unit test for each class.

### Why are the changes needed?

We need to support the expressions in python client

Fix: apache#5201

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Need to pass all unit tests.

---------

Co-authored-by: Xun <[email protected]>
…peError: e.type.match (apache#5762)

### What changes were proposed in this pull request?

Handle unsupported column types.

### Why are the changes needed?

Fix: apache#5707

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Tested by opening a table containing unsupported column types (e.g.,
list, map, struct).

![image](https://github.com/user-attachments/assets/1df87e3b-8142-4001-a257-c99a390b35d9)
…e#5610)

### What changes were proposed in this pull request?

Added simple authentication to the Gravitino CLI

### Why are the changes needed?

To support authentication.

Fix: apache#5384

### Does this PR introduce _any_ user-facing change?

No, but adds authentication to the CLI.

### How was this patch tested?

Tested locally.
### What changes were proposed in this pull request?

Sort options in alphabetical order.

### Why are the changes needed?

Feedback in an unrelated PR.

Fix: # N/A

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Tested locally
### What changes were proposed in this pull request?

Added OAuth support to Gravitino CLI, Note this is supported by the
configuration files.

### Why are the changes needed?

To support OAuth in the Gravitino CLI.

Fix: apache#5384

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Tested locally.
…pache#5701)

### What changes were proposed in this pull request?

support Aliyun oss credential provider 

### Why are the changes needed?

Fix: [# (5625)](apache#5625)

### How was this patch tested?

IcebergRESTOSSIT
### What changes were proposed in this pull request?

Missing param `<T>` is added to the `output` method of `Command.java`.

### Why are the changes needed?

Fix: apache#5772

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?
To verify, please run the following command:
```
./gradlew clean build -x test
```
After running this command, the warning `no @param for <T>` will no
longer appear.
…pache#5766)

### What changes were proposed in this pull request?

Support Kerberos authentication.

### Why are the changes needed?

To add support for Kerberos authentication.

Fix: apache#5384

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Tested locally.
…tion and code structure layout (apache#5835)

### What changes were proposed in this pull request?

1. Implement basic FUSE interfaces.
2. Implement filesystem trait and relation structures.


### Why are the changes needed?

Fix: apache#5734

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

No
� Conflicts:
�	clients/filesystem-fuse/src/main.rs

� Conflicts:
�	clients/filesystem-fuse/src/filesystem.rs
�	clients/filesystem-fuse/src/main.rs
…pache#5878)

### What changes were proposed in this pull request?

Implement a common filesystem layer to handle manage file ids, file name
mappings, and file relationships. and delegate filesystem APIs to
PathFilesystem.

### Why are the changes needed?

Fix: apache#5877

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

Uts
@diqiu50 diqiu50 closed this Dec 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Improvement] Implement the Gravitino fileset file system in gvfs fuse