Skip to content

Where Contributors Have Problems

Jump to bottom Edit New page
andychu edited this page Jun 13, 2022 · 51 revisions

Back to Contributing

This page documents where Oil contributors typically have problems. As of May 2022, the project has existed for 6 years, and Github says there have been 51 contributors. I would say at least 30 of them contributed code (as opposed to fixing typos, which is welcome). Thank you for all the help!

The goal of this page is to attract the right kind of contributor, by being up front about what to expect.

1. Trying to use OS X to develop Oil.

I've noticed that many contributors try to work on OS X. I tried this at first too, but now I use a Linux VM under VirtualBox when I'm on OS X.

A primary reason is that Spec Tests run against many shells, some of which aren't built for OS X. A secondary reason is that it's not clear what version of bash and Python are installed on OS X (although this is also an issue on Linux).

Follow-ups from comments on this:

2. We Have a Shell-Based Workflow and Development Environment

I use vim, grep, and tmux to navigate the code. Everything is highly automated with shell scripts.

However I've noticed that some people don't feel productive with this shell-based workflow. There is a lot of code generation done by custom tools, which means that it is somewhat fundamental. (That is, IDEs often don't work well with code generation.)

However:

  • There were times when the build scripts got messy. And I fixed them in response to contributor feedback. If you're having problems please complain about it! :)
  • I'm not a zealot about this: I also use JetBrains CLion to debug C++, because I need a GUI to debug productively!
    • In a distant utopia, there will be a GUI that is complementary to shell ... I just don't want to lose language-oriented composition and low latency with that evolution.

I wrote Oil Dev Cheat Sheet to help with this problem. Let me know if you are confused by it!

Hint: The Code Mostly Has a 2-Level Structure

When I need to find something, I simply use grep commands like this:

$ grep 'oil:basic' */*.py

$ grep CommandParser */*.sh

ctags and vim also work well on the codebase, although I often forget to set it up! devtools/ctags.sh has some shell functions. And I use grep -n pattern */*.py > out.txt, then open up out.txt in Vim and jump directly to locations. (Ctrl-])

3. A Very Test-Driven Workflow

The shell workflow is also test-driven. The shell is very intricate, so it's useful to isolate a specific piece of code with a test, and then make it go green.

We have many types of tests which run continuously (in Soil), like Spec Tests. They are published with every release on the quality.html page

I find that this type of workflow makes collaboration easier and progress very objective. However, I have noticed that not every contributor is comfortable with this style. Though I will add that I think it's a great thing to learn for any programmer! It's slower in the short term, but faster in the long run.

4. C and C++

There are no "guard rails" for oil-native! I had a lot of problems with the garbage collector and mysterious seg faults.

I spent a lot of time in the debugger.

I learned not to ignore certain compiler warnings! (e.g. issue 1128: -Wreturn-type)

A garbage collector is inherently unsafe code. The point of writing it is so that the rest of our code can be safe Python!

  • BUT for the initial stage of the NLNet grant project, I suggest we work on pushing the 1496 test/spec-cpp number up to the 1775 test/spec.sh number, without garbage collection (as I've been doing). This will move the project forward, and prevent us from getting stuck in the mud.

5. Abstraction and Metalanguages

This is a little fuzzy, but I've noticed that it can be a stretch for some programmers to view Python syntax as something other than Python.

We are thinking of it differently. It's a subset of statically typed Python, used as a metalanguage for Oil. This is so it can be translated to fast C++.

Oil Is Being Implemented "Middle Out" (March 2022)

  • However, if you make something cool work in Oil, in Python, you don't have to translate it to C++! (It often works with no effort, though this is not guaranteed.) You should send it to us first, and then we'll talk about the translation. I feel like not enough contributors have taken advantage of this nice property!

6. The Git Repo and Release Tarball Are Different

Several contributors had issues with this difference. We have both a "dev build" and "release build", which is explained on Contributing.

  • The dev build is very easy to make. You should be able to get bin/osh -c 'echo hi' running on a Linux machine in less than a minute.
  • Making the release build involves a big build process. It transforms the git repo into the release tarball. It's mostly automated in the Soil continuous build.

Conclusion

If none of that fazes you, then you might be a great person to work on Oil! I'm still engaged with this project after many years because it's allowed for an extremely high pace of learning. I have become a much better programmer by working on it.

And of course the reason I started the project is that I became a much better programmer by automating everything with shell! I'm able to tackle many subprojects at once because everything is automated and written down.

When I stop working, my mind is able to forget the details of what I did that day. When I start working, my shell scripts show me where I left off. They tell me what to do next.

Related:

Add a custom footer
Add a custom sidebar
Clone this wiki locally