Move the ruby C build to a separate crate #310
Comments
sgrif
added a commit
to sgrif/rubyfmt
that referenced
this issue
Jun 4, 2021
This commit attempts to clean up the code responsible for compiling Ruby in the build script, as well as get the crate compiling on Windows. For Unix, the main change here is that I've removed any attempts at skipping steps beyond those built into `make`. This should still skip the most expensive parts of the build, but be much more resilient. It is possible (or even likely) that this is going to be slightly slower than what was there before. If this is undesirable, we should move the C build out to a separate crate as outlined in fables-tales#310. On Windows, this goes from the crate not compiling to the crate compiling. This commit does *not* mean that the build on Windows is working, yet. On my machine, it builds but immediately segfaults when run. This is most likely because I have configured Ruby to skip all extensions except ripper. The most likely candidate is `enc` (which unfortunately is also the one I was struggling to get building), but I haven't dug in further. Even though Windows builds segfault, I don't think that should block this PR being merged. I've spent several hours banging my head against this, and need to walk away from it for a while. The fact that it *compiles* is still a step in the right direction. The fact that this is strictly an improvement on what was happening before is why I opted to submit this in this state. The way Windows build tools are set up is... Funky. MSVC intends you to pretty much always build through Visual Studio, and if you install just the build tools you're expected to run through a shortcut it provides which links MSVC DLLs and sets a ton of env vars for you. This is definitely suboptimal, and I couldn't get the crate to build at all when run from the MSVC console. The CC crate seems to handle most of this for us. I still ran into a few issues figuring everything out (Ruby builds in 32 bit by default even when compiled on a 64 bit system for... some reason), and I couldn't some extensions (enc and bigdecimal, possibly others) on 64-bit. But I don't think that's because of the build environment. The other changes were pretty straight forward. There was one instance where we assumed `long` was 64 bits (which isn't true), and the jemallocator crate doesn't build on Windows at the moment, so I've disabled it. Fixes fables-tales#301. Related to fables-tales#304. Related to fables-tales#310.
sgrif
added a commit
to sgrif/rubyfmt
that referenced
this issue
Jun 4, 2021
This commit attempts to clean up the code responsible for compiling Ruby in the build script, as well as get the crate compiling on Windows. For Unix, the main change here is that I've removed any attempts at skipping steps beyond those built into `make`. This should still skip the most expensive parts of the build, but be much more resilient. It is possible (or even likely) that this is going to be slightly slower than what was there before. If this is undesirable, we should move the C build out to a separate crate as outlined in fables-tales#310. On Windows, this goes from the crate not compiling to the crate compiling. This commit does *not* mean that the build on Windows is working, yet. On my machine, it builds but immediately segfaults when run. This is most likely because I have configured Ruby to skip all extensions except ripper. The most likely candidate is `enc` (which unfortunately is also the one I was struggling to get building), but I haven't dug in further. Even though Windows builds segfault, I don't think that should block this PR being merged. I've spent several hours banging my head against this, and need to walk away from it for a while. The fact that it *compiles* is still a step in the right direction. The fact that this is strictly an improvement on what was happening before is why I opted to submit this in this state. The way Windows build tools are set up is... Funky. MSVC intends you to pretty much always build through Visual Studio, and if you install just the build tools you're expected to run through a shortcut it provides which links MSVC DLLs and sets a ton of env vars for you. This is definitely suboptimal, and I couldn't get the crate to build at all when run from the MSVC console. The CC crate seems to handle most of this for us. I still ran into a few issues figuring everything out (Ruby builds in 32 bit by default even when compiled on a 64 bit system for... some reason), and I couldn't some extensions (enc and bigdecimal, possibly others) on 64-bit. But I don't think that's because of the build environment. The other changes were pretty straight forward. There was one instance where we assumed `long` was 64 bits (which isn't true), and the jemallocator crate doesn't build on Windows at the moment, so I've disabled it. The winapi dependency is basically just to get the appropriate `rustc-link-libs` that are needed for Ruby. I'm not sure if the `everything` feature actually slows anything down at runtime or not. It should probably get pared down at some point, but after realizing it's at least 6 or 7 dylibs I punted this for now Fixes fables-tales#301. Related to fables-tales#304. Related to fables-tales#310.
sgrif
added a commit
to sgrif/rubyfmt
that referenced
this issue
Jun 4, 2021
This commit attempts to clean up the code responsible for compiling Ruby in the build script, as well as get the crate compiling on Windows. For Unix, the main change here is that I've removed any attempts at skipping steps beyond those built into `make`. This should still skip the most expensive parts of the build, but be much more resilient. It is possible (or even likely) that this is going to be slightly slower than what was there before. If this is undesirable, we should move the C build out to a separate crate as outlined in fables-tales#310. On Windows, this goes from the crate not compiling to the crate compiling. This commit does *not* mean that the build on Windows is working, yet. On my machine, it builds but immediately segfaults when run. This is most likely because I have configured Ruby to skip all extensions except ripper. The most likely candidate is `enc` (which unfortunately is also the one I was struggling to get building), but I haven't dug in further. Even though Windows builds segfault, I don't think that should block this PR being merged. I've spent several hours banging my head against this, and need to walk away from it for a while. The fact that it *compiles* is still a step in the right direction. The fact that this is strictly an improvement on what was happening before is why I opted to submit this in this state. The way Windows build tools are set up is... Funky. MSVC intends you to pretty much always build through Visual Studio, and if you install just the build tools you're expected to run through a shortcut it provides which links MSVC DLLs and sets a ton of env vars for you. This is definitely suboptimal, and I couldn't get the crate to build at all when run from the MSVC console. The CC crate seems to handle most of this for us. I still ran into a few issues figuring everything out (Ruby builds in 32 bit by default even when compiled on a 64 bit system for... some reason), and I couldn't some extensions (enc and bigdecimal, possibly others) on 64-bit. But I don't think that's because of the build environment. The other changes were pretty straight forward. There was one instance where we assumed `long` was 64 bits (which isn't true), and the jemallocator crate doesn't build on Windows at the moment, so I've disabled it. The winapi dependency is basically just to get the appropriate `rustc-link-libs` that are needed for Ruby. I'm not sure if the `everything` feature actually slows anything down at runtime or not. It should probably get pared down at some point, but after realizing it's at least 6 or 7 dylibs I punted this for now Fixes fables-tales#301. Related to fables-tales#304. Related to fables-tales#310.
Merged
fables-tales
pushed a commit
that referenced
this issue
Jun 5, 2021
This commit attempts to clean up the code responsible for compiling Ruby in the build script, as well as get the crate compiling on Windows. For Unix, the main change here is that I've removed any attempts at skipping steps beyond those built into `make`. This should still skip the most expensive parts of the build, but be much more resilient. It is possible (or even likely) that this is going to be slightly slower than what was there before. If this is undesirable, we should move the C build out to a separate crate as outlined in #310. On Windows, this goes from the crate not compiling to the crate compiling. This commit does *not* mean that the build on Windows is working, yet. On my machine, it builds but immediately segfaults when run. This is most likely because I have configured Ruby to skip all extensions except ripper. The most likely candidate is `enc` (which unfortunately is also the one I was struggling to get building), but I haven't dug in further. Even though Windows builds segfault, I don't think that should block this PR being merged. I've spent several hours banging my head against this, and need to walk away from it for a while. The fact that it *compiles* is still a step in the right direction. The fact that this is strictly an improvement on what was happening before is why I opted to submit this in this state. The way Windows build tools are set up is... Funky. MSVC intends you to pretty much always build through Visual Studio, and if you install just the build tools you're expected to run through a shortcut it provides which links MSVC DLLs and sets a ton of env vars for you. This is definitely suboptimal, and I couldn't get the crate to build at all when run from the MSVC console. The CC crate seems to handle most of this for us. I still ran into a few issues figuring everything out (Ruby builds in 32 bit by default even when compiled on a 64 bit system for... some reason), and I couldn't some extensions (enc and bigdecimal, possibly others) on 64-bit. But I don't think that's because of the build environment. The other changes were pretty straight forward. There was one instance where we assumed `long` was 64 bits (which isn't true), and the jemallocator crate doesn't build on Windows at the moment, so I've disabled it. The winapi dependency is basically just to get the appropriate `rustc-link-libs` that are needed for Ruby. I'm not sure if the `everything` feature actually slows anything down at runtime or not. It should probably get pared down at some point, but after realizing it's at least 6 or 7 dylibs I punted this for now Fixes #301. Related to #304. Related to #310.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The build script as it is written makes an attempt to avoid rerunning make commands once Ruby has been built. This is brittle (see #301), and is something I'm probably going to remove as I go through the build script to do cleanup and fix #304).
makeshould handle not rebuilding everything all the time automatically, though this may be slower than the checks we're doing right now. A much more resilient and simple caching option would be to move the C build to a separate crateThe text was updated successfully, but these errors were encountered: