Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for location of facade crates #27783

Closed
alexcrichton opened this issue Aug 13, 2015 · 33 comments · Fixed by #59675
Closed

Tracking issue for location of facade crates #27783

alexcrichton opened this issue Aug 13, 2015 · 33 comments · Fixed by #59675
Labels
B-unstable Blocker: Implemented in the nightly compiler and unstable. C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.

Comments

@alexcrichton
Copy link
Member

alexcrichton commented Aug 13, 2015

We probably don't want to indefinitely have a large number of facade crates which are all unstable, so we should eventually stabilize these crates, fold them back into the standard library, or find a good middle ground in which they can all reside.

The current set of crates considered under this issue are:

  • rustc_unicode
  • libc
  • collections
  • alloc
  • rand

Update 2018-04-05:

  • libc and rand have moved to crates.io
  • collections was merged into alloc
  • rustc_unicode was renamed to std_unicode and later merged into core

This leaves only the alloc crate still unstable, which with the stable core and std crates (plus arguably proc_macro) form the standard library.


Update 2018-06-19:

RFC 2480 proposes stabilizing the alloc crate, which would close this issue.

@alexcrichton alexcrichton added T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. B-unstable Blocker: Implemented in the nightly compiler and unstable. labels Aug 13, 2015
@Gankra
Copy link
Contributor

Gankra commented Aug 13, 2015

It has been noted that the separation between alloc and collections is seemingly arbitrary. However with a robust allocator API, it could be possible to specify collections without any allocator. So e.g. no_std users could provide their own allocator and still use Vec.

Basically I don't think these should be stabilized at all in anything approaching the short term pending the ultimate allocator designs.

@zitsen
Copy link

zitsen commented Dec 16, 2015

Found this through alloc documents.

Is there any news about this issue?
btw, I just want a RawVec for custom data-structures.

@SimonSapin
Copy link
Contributor

RawVec would be great. Any way to allocate memory other than Vec::with_capacity + mem::forget would be nice.

@ghost
Copy link

ghost commented Jan 5, 2016

Is there any news about this issue?
btw, I just want a RawVec for custom data-structures.

RawVec would be great. Any way to allocate memory other than Vec::with_capacity + mem::forget would be nice.

It would be nice if rust officially support some low-level mem layer (in rust way).

@steveklabnik
Copy link
Member

@chao787 that's the alloc crate.

@Shnatsel
Copy link
Member

Shnatsel commented Oct 4, 2016

I've found this issue through compiler error message:

error: use of unstable library feature 'alloc_system': this library is unlikely to be stabilized in its current form or name (see issue #27783)

I'm just trying to opt out of jemalloc and get Rust to compile my binary with system default malloc implementation. Is there a way to do it that would not involve unstable language features, like a compiler switch to get jemalloc out of my binaries and off my lawn?

Rationale: jemalloc has abysmal fork performance (80x slowdown) on default kernel configuration of recent Ubuntu. See issue #36705 for more info.

Opting out of jemalloc is also important for fuzzing, e.g. with afl-fuzz; even on jemalloc-friendly configurations using default malloc gets you 20% more fork performance which is significant for fuzzing workloads, and lets you substitute abusive memory allocators at runtime that catch more errors than default allocator or jemalloc but don't incur the performance penalty of DUMA or AddressSanitizer. See AFL's libdislocator as an example.

@alexcrichton
Copy link
Member Author

@Shnatsel yeah I'd love to explore the possibilities of stabilizing the "please give me the system allocator" intent. Right now it's unfortunately not possible to do that in stable Rust.

The tracking issue for global allocators in general is #27389, but it may be worth spawning off a separate thread of discussion for just declaring the intent to use the system allocator.

@brson
Copy link
Contributor

brson commented Oct 4, 2016

FWIW I'm not in favor of merging the facade crates. Having std be decomposed into reusable components should be useful for those targeting more exotic systems, particularly if I complete my further ambitions to extract all system-specific components of std into their own crates. Ultimately people should be able to create custom stds for whatever weird systems they want be reusing our small self-contained building blocks.

@Shnatsel
Copy link
Member

Shnatsel commented Oct 4, 2016

Decomposed stdlib would be really useful for asm.js target that I will need in the near future and really want to use Rust for.

@whitequark
Copy link
Member

@brson As a person targeting a mildly exotic system, I agree. For example, I would expect that a significant proportion of bare-metal use cases (like mine) would want alloc and collections but nothing beyond.

@SimonSapin
Copy link
Contributor

SimonSapin commented Mar 1, 2017

The current set of crates considered under this issue are:

  • rustc_unicode
  • libc
  • collections
  • alloc
  • rand

I surveyed what’s tracked by this issue when writing #39954. Unless I missed something:

  • libc and rand are now available on crates.io.
  • All functionality in collections is re-exported as stable in std, is deprecated, or has a more specific tracking issue.
  • Some non-deprecated functionality in std_unicode (formerly rustc_unicode) and alloc is re-exported as stable in std.
  • Functionality not available on stable Rust is:
    • alloc::heap
      • When a type T can be written such as mem::align_of::<T>() is the desired alignment, Vec + mem::forget can be used to re-implement allocate, deallocate, and reallocate.
      • As far as I know, no such work-around is possible for reallocate_inplace or usable_size.
      • EMPTY is “known” to be 1 as *mut (), but keeping a separate definition and assuming it keeps matching alloc’s seems fragile
    • alloc::oom
    • alloc::raw_vec
    • std_unicode::derived_property
    • std_unicode::property
    • std_unicode::str::is_utf16 Remove std_unicode::str::is_utf16 #40190
    • std_unicode::str::utf8_char_width Reduce std_unicode’s public API #40189
    • std_unicode::str::Utf16Encoder
  • Additional functionality not available on stable Rust with #![no_std]:
    • alloc::arc
    • alloc::boxed
    • alloc::rc
    • collections
    • std_unicode::char
    • std_unicode::str::UnicodeStr

Some environments might require #![no_std] because they lack threads for example, but still have a memory allocator available.

My subjective opinions:

  • derived_property, property, utf8_char_width, and raw_vec are public in order to be used in other crates. Arguably they’re not something we ever want to stabilize.
  • Utf16Encoder is almost in this category, but it is usefully more general than the stable str::encode_utf16 method since it works on arbitrary char iterators rather than just &str.
  • is_utf16 has never been used in the compiler or standard library since 47e7a05 added it in 2012 “for OS API interop”. It can be replaced with a one-liner: std::char::decode_utf16(s).all(|r| r.is_ok()). I propose removing it: Remove std_unicode::str::is_utf16 #40190

@SimonSapin
Copy link
Contributor

Looking at Utf16Encoder again, it’s based on char::encode_utf16 and so fairly easy to reproduce outside of std, so never mind. (Mostly, it’s only verbose because defining an iterator is verbose.)

I think this leaves two items in need of attention:

  • Stabilize raw memory allocation (alloc::heap, maybe also alloc::oom).
  • Figure out how to provide non-core functionality to #![no_std] crates.

I vaguely recall GC-related concerns about the former, but I think those would also apply to the Vec + mem::forget trick that is used on stable today. There’s even a crate for it: https://crates.io/crates/memalloc

@SimonSapin
Copy link
Contributor

Stabilize raw memory allocation

This is #27700, not sure how I missed that.

@nagisa
Copy link
Member

nagisa commented May 3, 2017

For unicode stuff, now that we can use crates.io crates in rustc build system, could the crates.io crates be used?

@SimonSapin
Copy link
Contributor

The compiler depends on crates from crates.io. Does std? Should it?

@clarfonthey
Copy link
Contributor

Question: is there a reason why Box is re-exported in libcollections but Rc and Arc are not?

@tarcieri
Copy link
Contributor

@alexcrichton might want to remove collections from the list in the description since it's gone as of #42648

@Ericson2314
Copy link
Contributor

Ericson2314 commented Jun 17, 2017

@SimonSapin I think it would be a huge boost to the no-std ecosystem if std was allowed to depend on crates.io (at least official out-of-tree crates).

@su8
Copy link

su8 commented Jun 21, 2017

How do I install libc, can't use the following

extern crate libc;
use libc::{c_char, uint8_t};

@clarfonthey
Copy link
Contributor

@su8 libc is on crates.io now

@clarfonthey
Copy link
Contributor

@alexcrichton the OP should probably be updated

@Mark-Simulacrum Mark-Simulacrum added the C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC label Jul 22, 2017
bors added a commit that referenced this issue Apr 11, 2018
Merge the std_unicode crate into the core crate

[The standard library facade](#27783) has historically contained a number of crates with different roles, but that number has decreased over time. `rand` and `libc` have moved to crates.io, and [`collections` was merged into `alloc`](#42648). Today we have `core` that applies everywhere, `std` that expects a full operating system, and `alloc` in-between that only requires a memory allocator (which can be provided by users)… and `std_unicode`, which doesn’t really have a reason to be separate anymore. It contains functionality based on Unicode data tables that can be large, but as long as relevant functions are not called the tables should be removed from binaries by linkers.

This deprecates the unstable `std_unicode` crate and moves all of its contents into `core`, replacing them with `pub use` reexports. The crate can be removed later. This also removes the `CharExt` trait (replaced with inherent methods in libcore) and `UnicodeStr` trait (merged into `StrExt`). There traits were both unstable and not intended to be used or named directly.

A number of new items are newly-available in libcore and instantly stable there, but only if they were already stable in libstd.

Fixes #49319.
bors added a commit that referenced this issue Apr 12, 2018
Merge the std_unicode crate into the core crate

[The standard library facade](#27783) has historically contained a number of crates with different roles, but that number has decreased over time. `rand` and `libc` have moved to crates.io, and [`collections` was merged into `alloc`](#42648). Today we have `core` that applies everywhere, `std` that expects a full operating system, and `alloc` in-between that only requires a memory allocator (which can be provided by users)… and `std_unicode`, which doesn’t really have a reason to be separate anymore. It contains functionality based on Unicode data tables that can be large, but as long as relevant functions are not called the tables should be removed from binaries by linkers.

This deprecates the unstable `std_unicode` crate and moves all of its contents into `core`, replacing them with `pub use` reexports. The crate can be removed later. This also removes the `CharExt` trait (replaced with inherent methods in libcore) and `UnicodeStr` trait (merged into `StrExt`). There traits were both unstable and not intended to be used or named directly.

A number of new items are newly-available in libcore and instantly stable there, but only if they were already stable in libstd.

Fixes #49319.
@Ericson2314
Copy link
Contributor

OK now that #49698 is merged, I'd like to reiterate my concerns on this general direction of fewer creates behind the facade.

  • There is some tentative consensus that multiple facade creates is good, even necessary, for portability. See Refactor std to improve ease of porting rust-lang-nursery/portability-wg#1. This does not apply to the std_unicode merge.

  • I (and a handful of other) concerned about too big/monolithic crates, and keeping library code unnecessarily in https://github.com/rust-lang/rust/.

  • Code that doesn't use unstable language features can always be moved out in principle.

  • Moving code out is friendlier to alternative Rust implementations, and even for the official one better formalizes the which code the lang team doesn't need to worry extra about.

  • Smaller libraries are easier to read.

  • Independent repos foster parallel development

  • core is getting quite big, especially. IMO It shouldn't contain all pure abstractions, but just the minimal amount to support lang items.

The second concern absolutely does apply to std_unicode merge.std_unicode doesn't appear to, and nobody said anything to the contrary. It's much easier to merge crates than split them apart as things naturally grow entangled when nothing prevents that. So to the extent that the jury is still out on the facade, we should not be "speculatively" merging crates without fully-formed justification.

@Ericson2314
Copy link
Contributor

Ericson2314 commented Apr 12, 2018

Also, on a process level, I'm concerned that rather than getting a simple "we disagree with your points" which would have been fine, I was getting told "your concerns don't apply to this PR", which just isn't true. This leaves me wondering who actually read the discussion before checking off the box, and whether the rules "final comment period" was followed.

Admittedly, I did open with the unrelevant portability concerns, unfortunately sowing confusion I did not intend, but I thought that was cleared up in the end, at least between @SimonSapin and I. @SimonSapin indeed said in his last comment to me #49698 (comment) "I don’t care as much about how source code is organized internally" which at least felt like a "this PR isn't trying to disagree with you" leaving open other resolutions to the issue. But then @withoutboats rejoined with another "[your concern] seems off topic from this PR" #49698 (comment), and @alexcrichton, who had not participated in that discussion since a LGTM before I raised my points, did the r+.

Again, I'm not demanding that everyone agree to with me, but just to that explicit reject my points rather than leaving open that they were just missed. If there is no process violation going on, either FCP is different for PRs than RFCs in ways I wasn't aware of, or I misunderstand FCP overall.

@SimonSapin
Copy link
Contributor

SimonSapin commented Jun 19, 2018

RFC 2480 proposes stabilizing the alloc crate, which would close this issue.

Related:

@glandium
Copy link
Contributor

There is something questionable with using alloc in a no_std context currently: you automatically get a global allocator without adding a #[global_allocator], and that allocator is jemalloc. I would have expected something similar to panic: a compiler error about the missing panic_implementation (which currently still talks about the lang item, but you get the point).

@SimonSapin
Copy link
Contributor

@glandium #36963 should make this disappear, and it only happens for executables. A staticlib for example does get an error message: #51639, since the "default lib allocator" is not jemalloc and is defined in std.

But yeah it’s a good question whether we should change this in the meantime. Maybe it’s not much of a problem in practice?

bors added a commit that referenced this issue Jun 29, 2018
Make the public API of the alloc crate a subset of std

This only affects **unstable** APIs.

I plan to submit an RFC proposing to stabilize the crate. The reason it isn’t stable yet (#27783) is in case we end up merging the standard library crates into one. However the `core` crate is already stable, so if that happens we’ll need to keep it working somehow (likely by making replacing its contents by `pub use` items). We can do the same for `alloc`. This PR will hopefully make this easier, but even if that doesn’t happen consistency with `std` seems good.
@newpavlov
Copy link
Contributor

How about adding getrandom to the OP list? (maybe instead of rand?) Introduction of a simple (overwritable) function for pulling entropy from system randomness source was previously briefly discussed here. See rust-random/getrandom for the current prototype.

@SimonSapin
Copy link
Contributor

RFC 2480 to stabilize the alloc crate was accepted. PR #59675 does so, and closes this issue since alloc was the last crate tracked here. (Others have been stabilized already, moved to crates.io, or merged into other crates.)

SimonSapin added a commit to SimonSapin/rust that referenced this issue Apr 12, 2019
Centril added a commit to Centril/rust that referenced this issue Apr 12, 2019
Centril added a commit to Centril/rust that referenced this issue Apr 13, 2019
Centril added a commit to Centril/rust that referenced this issue Apr 13, 2019
Centril added a commit to Centril/rust that referenced this issue Apr 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
B-unstable Blocker: Implemented in the nightly compiler and unstable. C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging a pull request may close this issue.