LTO triggers apparent miscompilation on Windows 10 x64

[AlyoshaVasilieva]

I tried this code:

Cargo.toml:

[package]
name = "bmp-minimal"
version = "0.1.0"
authors = ["Malloc Voidstar <1284317+AlyoshaVasilieva@users.noreply.github.com>"]
edition = "2018"

[dependencies]

[profile.release]
lto = true
codegen-units = 1

main.rs:

use std::io::Write;

fn main() {
    let mut out = Vec::with_capacity(51200);
    for val in 0u8..=255 {
        out.write_all(&[val, val, val, 0]).unwrap();
    }

    for _ in (0..1).rev() {
        for _ in 0..1 {
            out.write_all(&[0]).unwrap();
        }
    }
    println!("{} bytes written", out.len());
}

(extremely minimized form of grayscale BMP encoding from image crate)

Command: cargo run --release

I expected to see this happen: 1025 bytes written

Instead, this happened:

memory allocation of 26843545600 bytes failederror: process didn't exit successfully: `target\release\bmp-minimal.exe` (exit code: 0xc0000409, STATUS_STACK_BUFFER_OVERRUN)

Meta

rustc --version --verbose:

rustc 1.45.0 (5c1f21c3b 2020-07-13)
binary: rustc
commit-hash: 5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2
commit-date: 2020-07-13
host: x86_64-pc-windows-msvc
release: 1.45.0
LLVM version: 10.0

It appears to work fine on ARM Linux in my limited testing.

This minimized form works/doesn't work on the following versions:

Rust version	Works?
1.45 (LLVM 10.0)	No
1.44.1	No
1.43.1	Yes
1.42.0	No
1.41.1	No
1.40.0	No
1.39.0	No
1.38.0 (LLVM 9.0)	No
1.37.0	Yes
1.36.0	Yes
1.35.0	Yes
1.34.0	Yes

If the LTO and codegen-units settings are removed it works on all above versions.

I originally noticed this when a project that worked on 1.44.1 suddenly started failing with huge memory allocations on 1.45, so this table specifically applies to this repro, not whatever the underlying bug is.

[LeSeulArtichaut]

@rustbot ping llvm

[rustbot]

Hey LLVM ICE-breakers! This bug has been identified as a good
"LLVM ICE-breaking candidate". In case it's useful, here are some
instructions for tackling these sorts of bugs. Maybe take a look?
Thanks! <3

cc @camelid @comex @cuviper @DutchGhost @hanna-kruppe @hdhoang @heyrutvik @JOE1994 @jryans @mmilenko @nagisa @nikic @Noah-Kennedy @SiavoshZarrasvand @spastorino @vertexclique @vgxbj

[LeSeulArtichaut]

It appears to work fine on ARM Linux in my limited testing.

@rustbot ping windows

[rustbot]

Hey Windows Group! This bug has been identified as a good "Windows candidate".
In case it's useful, here are some instructions for tackling these sorts of
bugs. Maybe take a look?
Thanks! <3

cc @arlosi @danielframpton @gdr-at-ms @kennykerr @luqmana @lzybkr @retep998 @rylev @sivadeilra

[mati865]

Works fine on Windows-gnu:

$ cargo run --release
   Compiling bmp-minimal v0.1.0 (C:\Users\mateusz\AppData\Local\Temp\tmp)
    Finished release [optimized] target(s) in 3.71s
     Running `target\release\bmp-minimal.exe`
1025 bytes written

With MSVC it endlessly allocates memory.

@rustbot modify labels: -O-windows +O-windows-msvc