Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using liberofs #335

Open
cgwalters opened this issue Sep 6, 2024 · 5 comments
Open

Using liberofs #335

cgwalters opened this issue Sep 6, 2024 · 5 comments

Comments

@cgwalters
Copy link
Contributor

It came up in a discussion that perhaps instead of composefs having its own "write EROFS" code we could reuse liberofs from https://github.com/erofs/erofs-utils/

These are just the things I'm thinking about:

  • API stability; the library isn't stable, but we could probably get around that with vendoring
  • Licensing erofs-utils is GPLv2+ or ASL2.0+ but our lib is LGPLv2+ - I guess it works? Yeah, LGPLv2+ should be compatible with ASL2, I was just thinking of ASL2 and GPLv2 being classically incompatible
  • Format stability: this is a big one. Effectively we promise that the input mkcomposefs accepts today will result in a bit for bit compatible output going forward. Now it seems unlikely mkfs.erofs would change what it does by default, but if it ever did we'd need an option to control that.
  • block vs byte based: AFAICS liberofs operates in terms of blocks (which makes total sense) but the public API we've exposed here via lcfs_write_to effectively promises to operate in terms of streaming. I think it wouldn't be too hard to bridge these two, but still a concern.

Combining these things, I think it's potentially viable for us to investigate vendoring portions of liberofs to start.

@hsiangkao
Copy link
Contributor

I could help for composefs after I sort out apis mess...
liberofs could be an alternative first since

  • as you said: format stability
  • I do need time to make sure no memory leak in liberofs itself. For original mkfs.erofs usage likewise, it was considered harmless..

@cgwalters
Copy link
Contributor Author

I do need time to make sure no memory leak in liberofs itself. For original mkfs.erofs usage likewise, it was considered harmless..

Heh yeah, understood...as I like to say in C one needs to work hard not to leak memory, in Rust one needs to work hard to leak memory.

That said, any reason you're not using __attribute__((cleanup()))?

@hsiangkao
Copy link
Contributor

I do need time to make sure no memory leak in liberofs itself. For original mkfs.erofs usage likewise, it was considered harmless..

Heh yeah, understood...as I like to say in C one needs to work hard not to leak memory, in Rust one needs to work hard to leak memory.

Rust is a great thing. Yet Rust is still not quite portable and syntax-stable compared to C and at an early stage of Linux distribution ecosystems (Rust crates vs C libararies).

I don't have enough time to maintain several language implementaions (some other people asked for Go implementation) mainly due to active development. But if any volunteer I'd happy to enbrace them.

That said, any reason you're not using __attribute__((cleanup()))?

Not for any particular reason (as long as --std=gnu89 supports it but not quite sure which version gcc started to support this), I could refactor code to use it.

@cgwalters
Copy link
Contributor Author

I don't have enough time to maintain several language implementaions (some other people asked for Go implementation) mainly due to active development. But if any volunteer I'd happy to enbrace them.

No interest in Go from me, but certainly if I had started composefs it would have been in Rust and I maintain higher level Rust crates here.

That said, I did recently come across https://github.com/ToolmanP/erofs-rs - not sure if you saw that. I only very superficially skimmed the code, it looked decent.

@hsiangkao
Copy link
Contributor

hsiangkao commented Sep 9, 2024

I don't have enough time to maintain several language implementaions (some other people asked for Go implementation) mainly due to active development. But if any volunteer I'd happy to enbrace them.

No interest in Go from me, but certainly if I had started composefs it would have been in Rust and I maintain higher level Rust crates here.

Yeah, I totally understand you aren't interested.
But in term to EROFS perspertive itself, we shouldn't ignore the requirement since EROFS is language-independent(neutral), so there is no difference between C, Rust, and Go.
Just because C is still the most portable language in the current world, but Rust and Go are definitely acceptable (although my leisure bandwidth is limited, life is hard.)

That said, I did recently come across https://github.com/ToolmanP/erofs-rs - not sure if you saw that. I only very superficially skimmed the code, it looked decent.

Yeah, I noticed that, but it still need some time to form a proper one for erofs-utils integration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants