Tracking issue for allocation APIs

[alexcrichton]

Current status

Final incarnation of std::heap is being proposed in rust-lang/rfcs#1974, hopefully for stabilization thereafter.

Open questions for stabilization are:

• Is it required to deallocate with the exact size that you allocate with? With the usable_size business we may wish to allow, for example, that you if you allocate with (size, align) you must deallocate with a size somewhere in the range of size...usable_size(size, align). It appears that jemalloc is totally ok with this (doesn't require you to deallocate with a precise size you allocate with) and this would also allow Vec to naturally take advantage of the excess capacity jemalloc gives it when it does an allocation. (although actually doing this is also somewhat orthogonal to this decision, we're just empowering Vec). So far @gankro has most of the thoughts on this.

• Is it required to deallocate with the exact align that you allocate with? Concerns have been raised that allocatores like jemalloc don't require this, and it's difficult to envision an allocator that does require this. (more discussion). @ruuda and @rkruppe look like they've got the most thoughts so far on this.

Original report

This is a tracking issue for the unstable APIs related to allocation. Today this encompasses:

• alloc
• heap_api
• oom

This largely involves just dealing with the stabilization of liballoc itself, but it needs to address issues such as:

• Are we willing to stabilize the exact interface of the allocation API as-is today?
• Do we want to stabilize custom allocators?
• Do we want to stabilize a standalone allocation library, liballoc?
• Should oom be a generally available piece of functionality, and should it be pluggable?

This will likely take a good deal of work to stabilize, and this may be done piecemeal, but this issue should serve as a location for tracking at least these unstable features.

[alexcrichton]

cc @pnkfelix

[SimonSapin]

It’s already possible to indirectly use allocate and deallocate in stable Rust:

fn allocate<T>(count: usize) -> *mut T {
    let mut v = Vec::with_capacity(count);
    let ptr = v.as_mut_ptr();
    std::mem::forget(v);
    ptr
}

fn deallocate<T>(ptr: *mut T, count: usize) {
    std::mem::drop(Vec::from_raw_parts(ptr, 0, count));
}

Any future GC support (mentioned in the #[unstable(feature="heap_api")) message) will need to deal with this.

While this hack has the merit of existing (and enabling many libraries to making themselves available on stable Rust),

• It’s ugly
• It’s hard to discover if you haven’t seen it before
• It’s sometimes awkward to use. For example, Tendril wants to do something like Rc<str> where a reference count is in a header next to (before) that bytes that make up a string. The header needs to be memory-aligned, but the amount of bytes is arbitrary. Tendril ends up calculating the next multiple of size_of::<Header> in order to find out what capacity of Vec<Header> to allocate for a given number of bytes.

While I’m not attached to the details of the current alloc::heap API, I think it’s important for Stable Rust to have a more straightforward way to allocate/deallocate memory.

[SimonSapin]

Random idea for a (more rustic?) RAII-based API for alloc::heap:

/// Allocated but not-necessarily-initialized memory.
struct Buffer {
    ptr: Unique<u8>,
    size: usize,
    align: usize,
}

impl Drop for Buffer {
    /* deallocate */
}

impl Buffer {
    fn new(size: usize, align: usize) -> Result<Self, ()> {
        /* allocate, but avoid undefined behavior by asserting or something
           (maybe skip calling allocate() on `size == 0`?) */
    }

    // Maybe skip those and keep the unsafe functions API?
    fn into_ptr(self) -> *mut u8 { /* forget self */ }
    unsafe fn from_raw_parts(ptr: *mut u8, size: usize, align: usize) -> Self { /* */ }

    fn as_ptr(&self) -> *const u8 { self.ptr }
    fn as_mut_ptr(&mut self) -> *mut u8 { self.ptr }
    fn size(&self) -> usize { self.size }  // Call this len?
    fn align(&self) -> usize { self.align }

    // The caller is responsible for not reading uninitialized memory
    unsafe fn as_slice(&self) -> &[u8] { /* ... */ }  
    unsafe fn as_mut_slice(&mut self) -> &mut [u8] { /* ... */ }

    fn reallocate(&mut self, new_size: usize) -> Result<(), ()> { /* ... */ }
    fn reallocate_in_place(&mut self, new_size: usize) -> Result<(), ()> { /* ... */ }
}

[kamalmarhubi]

It’s already possible to indirectly use allocate and deallocate in stable Rust:

this doesn't get you access to the align parameter though

[SimonSapin]

Not directly, but you can influence alignment by carefully picking T in Vec::<T>::with_capacity.

[kamalmarhubi]

@SimonSapin ooh nice! I'd have to read up on alignment rules to figure out if there's a type that would let me align to a page boundary or not, but nice trick :-)

[pnkfelix]

@kamalmarhubi This stackoverflow answer is probably the most relevant thing that is actually implemented today: http://stackoverflow.com/questions/32428153/how-can-i-align-a-struct-to-a-specifed-byte-boundary

(Longer term we'll presumably put in something better. The Allocator trait I'm proposing has Layout objects to describe blocks of memory that allow one to specify alignment constraints.)

[kamalmarhubi]

@pnkfelix thanks for the link. Sounds like I'm out of luck for page boundary alignment though! I am also unclear if the allocator would hate me for SPLICE_F_GIFT pages to the kernel. I should probably set up my own anonymous mapping for this anyway...

[glaebhoerl]

It took me several seconds to realize that "aligning to a page boundary" actually wasn't a tongue-in-cheek joke about text rendering...