Adding Ruby Support into OSS-Fuzz via Ruzzy #12034

AdvenamTacet · 2024-06-06T19:00:39Z

This is a follow-up to the discussions held during our Monthly Fuzzing Collaboration meetings and directly relates to issue #11967.

This Pull Request integrates Ruzzy support for Ruby fuzzing into OSS-Fuzz. Ruzzy is a coverage-guided fuzzer for pure Ruby code and Ruby C extensions, developed by Matt (@mschwager) at Trail of Bits. More information on Ruzzy can be found in the blog post titled "Introducing Ruzzy, a coverage-guided Ruby fuzzer".

The first commit of this PR integrates Ruby support into the OSS-Fuzz project via Ruzzy, while the second one includes the Ox project as an example of its usage.

The first commit introduces changes in the infra directory, most notably by adding the base-builder-ruby docker and the ruby install script.

Two scripts, ruzzy-build and ruzzy, have been added to base-builder-ruby and base-runner respectively. The former creates scripts that start harnesses with the latter one, and the latter is simply a wrapper for ruby with LD_PRELOAD.
In order to prevent the duplication of many gigabytes of data, we use separate installation directories for RubyGem. Technically, Ruzzy can be installed in the default directory without any performance disadvantage, but having a separate directory may facilitate troubleshooting.

This implementation was arrived at through testing a few ideas. If you have suggestions for further improvements, please let me know. I am currently addressing my concerns in the related issue.

Using the provided scripts isn't necessary but it does simplify the process. Installation directories are set using environment variables in the Dockerfiles, making it transparent for users.

The second commit simply adds a project to illustrate how straightforward the integration process is. You can test it using the standard helper commands.

Fixes: #11967

Co-authored-by: mschwager [email protected]

google-cla · 2024-06-06T19:00:43Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

github-actions · 2024-06-06T19:01:47Z

AdvenamTacet is integrating a new project:
- Main repo: https://github.com/ohler55/ox
- Criticality score: 0.61870

jonathanmetzman

This looks like it is almost there!

infra/base-images/base-builder-ruby/Dockerfile

jonathanmetzman · 2024-06-11T21:50:34Z

infra/base-images/base-builder-ruby/Dockerfile

+
+# The MAKE variable allows overwriting the make command at runtime. This forces the
+# Ruby C extension to respect ENV variables when compiling, like CC, CFLAGS, etc.
+ENV MAKE="make --environment-overrides V=1"


I'm not sure if we're using some default CFLAGS and CXXFLAGS from OSS-Fuzz at this point so you might want to set them to nothing.

I believe those variables are set.

https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-clang/Dockerfile#L65-L67

If I should not depend on those, I should set them myself.

My concern isn't that depending on them is bad, my concern is that these are "optimized' for fuzzing, but are not optimal for code that is being used to fuzz. E.g. "-O1".

We can change it, but it also works as is and is tested that way. I will look into it.

I don't see any reason to not use -O3, initial tests didn't show any problem. If I confirm Tomorrow that new flags work same with all projects I have on disc, I will update that file.

jonathanmetzman · 2024-06-11T21:51:22Z

infra/base-images/base-builder-ruby/ruzzy-build

+#!/usr/bin/env bash
+
+fuzz_target=$(basename "$1")


Can you add "-e" ?

I'm testing it, I believe so.

jonathanmetzman · 2024-06-11T21:51:38Z

infra/base-images/base-builder/install_ruby.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+# Copyright 2021 Google LLC


jonathanmetzman · 2024-06-11T21:52:37Z

infra/base-images/base-builder-ruby/ruzzy-build

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash


nit: this needs the copyright and license header.

projects/ox-ruby/Dockerfile

jonathanmetzman · 2024-06-11T21:54:15Z

projects/ox-ruby/build.sh

@@ -0,0 +1,27 @@
+#!/bin/bash -eu
+# Copyright 2016 Google Inc.


nit: 2024 Google LLC

jonathanmetzman · 2024-06-11T21:54:45Z

projects/ox-ruby/project.yaml

@@ -0,0 +1,12 @@
+homepage: "https://github.com/ohler55/ox"
+language: ruby
+primary_contact: "[email protected]"


Why not add your own contact info?

Neither @mschwager nor I are contributors to the project. We used it as a demonstration of Ruzzy's easiness of use and PoC, not with a goal to upstream it. That said, having an example project might be beneficial, so removing it from the Pull Request may not be ideal. However, managing those reports isn't something we aim to undertake. What's the best course of action here?

No worries!

projects/ox-ruby/fuzz_parse.rb

jonathanmetzman · 2024-06-11T21:56:09Z

Can both of you sign the CLA?

jonathanmetzman · 2024-06-11T22:06:18Z

/gcbrun trial_build.py ruby --fuzzing-engines libfuzzer --sanitizers coverage

jonathanmetzman · 2024-06-17T21:20:30Z

I fixed an issue with the testing infra. Let's try agian.

jonathanmetzman · 2024-06-17T21:20:36Z

/gcbrun trial_build.py ruby --fuzzing-engines libfuzzer --sanitizers coverage

AdvenamTacet

I implemented some issues, the rest I should do now. I will also resolve CLA issue.

infra/base-images/base-builder-ruby/Dockerfile

AdvenamTacet · 2024-06-21T03:48:45Z

infra/base-images/base-builder-ruby/Dockerfile

+
+# The MAKE variable allows overwriting the make command at runtime. This forces the
+# Ruby C extension to respect ENV variables when compiling, like CC, CFLAGS, etc.
+ENV MAKE="make --environment-overrides V=1"


I believe those variables are set.

https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-clang/Dockerfile#L65-L67

If I should not depend on those, I should set them myself.

AdvenamTacet · 2024-06-21T03:49:15Z

infra/base-images/base-builder-ruby/ruzzy-build

+#!/usr/bin/env bash
+
+fuzz_target=$(basename "$1")


I'm testing it, I believe so.

projects/ox-ruby/Dockerfile

AdvenamTacet · 2024-06-21T05:16:49Z

infra/base-images/base-runner/Dockerfile

@@ -87,6 +88,15 @@ RUN wget https://repo1.maven.org/maven2/org/jacoco/org.jacoco.cli/0.8.7/org.jaco
 COPY install_javascript.sh /
 RUN /install_javascript.sh && rm /install_javascript.sh

+# Copy built ruby and ruzzy from builder


139M /usr/local/rvm 140M /install/ruzzy

If it's important, I can try to reduce it (but without promises).

infra/base-images/base-runner/ruzzy

projects/ox-ruby/Dockerfile

jonathanmetzman · 2024-06-21T15:19:01Z

Can you add ruby to LANGUAGES in infra/constants.py please?

jonathanmetzman · 2024-06-27T17:26:34Z

Can you add ruby to LANGUAGES in infra/constants.py please?

Can we do this? Really excited to get this landed.

AdvenamTacet

Thank you for your patience, I will sit to it today and clear remaining requests. Thank you for your feedback and help!

AdvenamTacet · 2024-06-27T18:22:38Z

infra/base-images/base-builder-ruby/Dockerfile

+
+# The MAKE variable allows overwriting the make command at runtime. This forces the
+# Ruby C extension to respect ENV variables when compiling, like CC, CFLAGS, etc.
+ENV MAKE="make --environment-overrides V=1"


We can change it, but it also works as is and is tested that way. I will look into it.

AdvenamTacet · 2024-06-27T18:38:01Z

projects/ox-ruby/project.yaml

@@ -0,0 +1,12 @@
+homepage: "https://github.com/ohler55/ox"
+language: ruby
+primary_contact: "[email protected]"


Neither @mschwager nor I are contributors to the project. We used it as a demonstration of Ruzzy's easiness of use and PoC, not with a goal to upstream it. That said, having an example project might be beneficial, so removing it from the Pull Request may not be ideal. However, managing those reports isn't something we aim to undertake. What's the best course of action here?

This commit brings Ruzzy, a coverage-guided fuzzer developed for pure Ruby code and Ruby C extensions, into the OSS-Fuzz project. This addition effectively integrates Ruby fuzzing support into OSS-Fuzz. Similar to Atheris, a Python fuzzer, Ruzzy uses libFuzzer for coverage instrumentation and the fuzzing engine. It offers support for AddressSanitizer and UndefinedBehaviorSanitizer, particularly when fuzzing C extensions. While the current code is functional, additional refinement and bug fixes may be required to ensure reliability. You can find further discussion about this development in issue: google#11967 Co-authored-by: mschwager <[email protected]>

This commit adds the 'Ox' project into the repository as a demonstration of how projects written in Ruby can be integrated with Ruzzy within the OSS-Fuzz framework. The Ox project acts as a straightforward illustrative example, demonstrating the steps to set up, configure, and utilize Ruzzy for fuzzing Ruby code. The harnesses within it are distinctly simplistic, solely to exemplify the interaction between Ruzzy and OSS-Fuzz. The primary aim is to provide a clear comprehension of how to employ Ruzzy with individual Ruby projects in the OSS-Fuzz environment. 'Ox' repository: https://github.com/ohler55/ox

AdvenamTacet · 2024-06-28T03:46:30Z

I'm happy to report that I solved CLA failure, but I had to force-push to accomplish that (squashed all changed while doing that).

I will test installing rvm from apt ppa:rael-gc/rvm repo Tomorrow

sudo apt-add-repository -y ppa:rael-gc/rvm
sudo apt-get update
sudo apt-get install rvm

and if it works, I will push that change.

I don't see anything else I should do before merge, let me know if I missed something. One more time, thank you for your help!

jonathanmetzman · 2024-07-01T16:50:09Z

/gcbrun trial_build.py ruby --fuzzing-engines libfuzzer --sanitizers coverage

jonathanmetzman · 2024-07-01T16:55:40Z

infra/base-images/base-runner/Dockerfile

@@ -87,6 +88,15 @@ RUN wget https://repo1.maven.org/maven2/org/jacoco/org.jacoco.cli/0.8.7/org.jaco
 COPY install_javascript.sh /
 RUN /install_javascript.sh && rm /install_javascript.sh

+# Copy built ruby and ruzzy from builder


eh this is a little big. We can have it for now, but if you can do any cheap fixes it would help.
(size makes things harder for CIFuzz which needs to pull images quickly and has limited disk space.

jonathanmetzman · 2024-07-01T16:56:46Z

infra/constants.py

 ]
 LANGUAGES_WITH_COVERAGE_SUPPORT = [
-    'c', 'c++', 'go', 'jvm', 'python', 'rust', 'swift', 'javascript'
+    'c', 'c++', 'go', 'jvm', 'python', 'rust', 'swift', 'javascript', 'ruby'


Do we actually have coverage support for Ruby?

Yes. It's not perfect at the moment for pure Ruby, but it works and soon should be fixed (to work well). Works well with C extensions.

Just to add a bit more context here: Ruby coverage support is currently okay, but not ideal. But we're also working with the upstream Ruby language maintainers to improve their stdlib coverage functionality: https://bugs.ruby-lang.org/issues/20448. This will improve Ruzzy's coverage gathering support significantly 👍

jonathanmetzman · 2024-07-01T16:56:59Z

projects/ox-ruby/project.yaml

@@ -0,0 +1,12 @@
+homepage: "https://github.com/ohler55/ox"
+language: ruby
+primary_contact: "[email protected]"


No worries!

jonathanmetzman · 2024-07-02T12:32:02Z

I think the integration is still broken. I'm going to try to fix it when i gt to the office. I'm going to use my own email for the ruby project for now so no one gets spurious emails.

jonathanmetzman reviewed Jun 11, 2024

View reviewed changes

AdvenamTacet commented Jun 21, 2024

View reviewed changes

jonathanmetzman reviewed Jun 21, 2024

View reviewed changes

infra/base-images/base-runner/ruzzy Show resolved Hide resolved

projects/ox-ruby/Dockerfile Show resolved Hide resolved

AdvenamTacet commented Jun 27, 2024

View reviewed changes

Advenam Tacet and others added 2 commits June 28, 2024 05:21

AdvenamTacet force-pushed the ruzzy branch from fa81900 to 0ef97f9 Compare June 28, 2024 03:21

Resolve TODOs

3f18090

AdvenamTacet force-pushed the ruzzy branch from 70d8e6d to 3f18090 Compare June 28, 2024 03:55

jonathanmetzman reviewed Jul 1, 2024

View reviewed changes

jonathanmetzman merged commit fa84440 into google:master Jul 2, 2024
16 of 19 checks passed

		#!/usr/bin/env bash

		fuzz_target=$(basename "$1")

		#!/usr/bin/env bash

		fuzz_target=$(basename "$1")

Adding Ruby Support into OSS-Fuzz via Ruzzy #12034

Adding Ruby Support into OSS-Fuzz via Ruzzy #12034

Conversation

AdvenamTacet commented Jun 6, 2024 • edited Loading

google-cla bot commented Jun 6, 2024

github-actions bot commented Jun 6, 2024

jonathanmetzman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathanmetzman commented Jun 11, 2024

jonathanmetzman commented Jun 11, 2024

jonathanmetzman commented Jun 17, 2024

jonathanmetzman commented Jun 17, 2024

AdvenamTacet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathanmetzman commented Jun 21, 2024

jonathanmetzman commented Jun 27, 2024

AdvenamTacet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AdvenamTacet commented Jun 28, 2024 • edited Loading

jonathanmetzman commented Jul 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AdvenamTacet Jul 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathanmetzman commented Jul 2, 2024

AdvenamTacet commented Jun 6, 2024 •

edited

Loading

AdvenamTacet commented Jun 28, 2024 •

edited

Loading

AdvenamTacet Jul 2, 2024 •

edited

Loading