“If you have wings, why not fly?” – Nymphomaniac Vol. 1

Introduction

This is a rewrite of the Go code (2021) in Nim. In turn, the Go code was a rewrite of the C++ work by Tomas Öhberg (2017) which itself was a rewrite/reimplementation of the research by Balázs Tóth and Tamás Umenhoffer (EUROGRAPHICS 2009). Some day, when and if WebGPU becomes reasonably usable (around the year 2040), I will rewrite this code again.

Volumetric Lighting

Dependencies and Compilation

nimble install nimgl opengl glm flatty
nim c -r --hints:off -d:release main.nim 

Nim Setup

• Install Nim via Choosenim:

  curl https://nim-lang.org/choosenim/init.sh -sSf | sh
  ...
  tokyo@tokyo-Z87-DS3H:~$ nim -v
  Nim Compiler Version 1.6.8 [Linux: amd64]
  Compiled at 2022-09-27
  Copyright (c) 2006-2021 by Andreas Rumpf
  
  git hash: c9f46ca8c9eeca8b5f68591b1abe14b962f80a4c
  active boot switches: -d:release

  Set the path in ".bashrc" as indicated in the command prompt.

  Consider a more specialized [text editor] which can at least highlight the Nim code. My choice is NeoVim as I prefer something simple snappy lightweight. Its Nim plugin is newer than that of Vim.

• Install NeoVim:

  sudo apt install neovim -y
  mkdir $HOME/.config/nvim

• Install Plug:

  sh -c 'curl -fLo "${XDG_DATA_HOME:-$HOME/.local/share}"/nvim/site/autoload/plug.vim --create-dirs \
       https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim'
  cd $HOME/.config/nvim
  gedit init.vim     

• Install this NeoVim plugin

  Copy-paste and save this into init.vim:

  call plug#begin('~/.vim/plugged')
  Plug 'alaviss/nim.nvim'
  call plug#end()  
  
  set nofoldenable
  

  Add this line to ~/.vim/plugged/nim.nvim/syntax/nim.vim:

  highlight link nimSugUnknown NONE
  

  in order to remove red highlights for unknown symbols, clf. this issue.

  Run nvim, press Esc and :PlugInstall, :q, restart nvim. Use gd and ctrl+o to jump/get back into type/function definitions.

• Compile and run gltfviewer to test it all:

  git clone https://github.com/guzba/gltfviewer.git $HOME/gltfviewer
  cd $HOME/gltfviewer
  nimble install
  nim c -r ./src/gltfviewer.nim

Random Notes Taken While Programming

• Go is better at passing user data into the GLFW callbacks. There are three ways in Go: (i) global/static variables, (ii) glfwgetwindowuserpointer, and (iii) lambda functions. The third option is remarkably simple. Set mydata.f(...) instead of the usual f(...) as a callback. f then sees all the variables in mydata when called, meeting all the original signature requirements of f(...) as if mydata did not even exist.

  Nim allows one to change the scope of the functions with pragmas, IIAR. However, the callback functions are already defined with the "{.cdecl.}" pragma in the GLFW bindings which would not let the callbacks be turned into lambdas with "{.closure.}". So I went the global/static variable way in Nim.

• Nim's GLTF viewer is a lot slower than this surprisingly fast Go library, if compiled with the default flags. Part of the problem here is that the Nim code reads all the images into a big intermediate Nim image sequence before uploading them into the GPU buffers. I only made this even slower by pre-extracting the mesh data on the CPU as well. In addition, there is always some "ref object" in Nim waiting to be replaced with "object". Remarkably, this problem disappears when compiling with "d:release" or "d:danger" flags. Without them, it takes about 25s. to load Sponza, with them it is as instantaneous as in the Go code.

• GLTF spec allows 1-byte or 2-byte numbers in the GLTF buffers. In this code, everything is converted to four-byte floats and integers before uploading them to the GPU even if initially data can be of different sizes, e.g. see the function "read_vert_indices". I assumed every number is four-bytes in GLTF at first, and later fixed the bug only with Renderdoc, test cube, and the correct working example in Go.

• I missed "glGenerateMipmap(GL_TEXTURE_2D)" in the Nim code, and without it nothing seemed to work, unlike in the Go code. Debugging such OpenGL texture issues is a lot harder than debugging mesh geometry.

• This line bypassed the Go compiler, but was caught in Nim:

  gl.TexParameterf(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE)

  The third argument expects a float, but I am passing in an OpenGL (integer) constant here. This demanded a change to "gl.TexParameteri" in Nim. Such an error had no consequence in Go at the runtime though.

• Nim's "distinct type" adds some friction with GLenum and GLint casting. The const/let/var mutability system often produces "cannot take an address of an expression" errors. Uploading constant data to the GPU demands sending pointers/addresses and the compiler does not allow the data to be immutable.

• A tricky case of "Mat4[system.float32]" vs. "Mat4f" occurs when printing an array value in "main.nim" with "import glm" or without it. Without importing the library, the system treats the variable "WINDOW_STATE.cam.view" as type "Mat4f" which does not use the pretty printing operator $ overloaded in the package "glm". After importing "glm", the type becomes "Mat4[system.float32]" which picks up the pretty printing.

• Nim's operator overloading, generics and templates make vector/matrix math even more compact than Matlab, look at this pretty vector swizzling in glm/vec.nim:

  proc cross*[T](v1,v2:Vec[3,T]): Vec[3,T] =
    v1.yzx * v2.zxy - v1.zxy * v2.yzx  

  However, this compile time substitution layer is a bit scary if one recalls the C++ template errors. Consider this Go function:

  func Sqrt(v float32) float32 {
      return float32(math.Sqrt(float64(v)))
  } 

  It won't impress a type theorist, but do we really need that whole layer of problems here? If you get into "go generate" and templates this way with the big lib mentality, then perhaps yes. Since Go version 1.18 one can use generic types, but I would not bother.

• There is a pointless split between "vmath" and "glm". I went with the "glm" library as this is almost a 3D vector math standard. I did not have to worry about any row-major vs column-major issues at all, though somebody did, before me...

• There are quite a few GLFW binding choices, despite a tiny community. Consider these GLFW function signatures:

  GLFWAPI GLFWmonitor** glfwGetMonitors(int* count)
  GLFWAPI GLFWmonitor* glfwGetPrimaryMonitor(void)

  Here "GLFWmonitor" is some opaque C struct hidden under platform specific layers, the "GLFWAPI" macro can be ignored.

  Input: C semantics with struct** and struct*.

  What do these output types become in Go and Nim bindings?

  Go: go-gl/glfw/v3.3: []*struct and *struct.

  Nim: treeform/staticglfw: ptr pointer and pointer.

  Nim: nimgl/glfw: ptr UncheckedArray[ptr object] and ptr object. Notice the missing pointer reported in this issue which then got fixed.

  jyapayne/nim-glfw: ptr ptr object, ptr object and pragma.

  gcr/turbo-mush: ptr ptr cint, ptr cint.

  They are all fine, most likely. I chose "nim/glfw" as it looked to be the most consolidating and future-proof.

• Let's examine the OpenGL bindings w.r.t. the OpenGL function

  void glShaderSource(GLuint shader,
  GLsizei count,
  const GLchar **string,
  const GLint *length);

  In particular, let's focus on the third argument, i.e. **string which in reality is just a shader code, some ASCII text.

  In Go with go-gl bindings, the type becomes **uint8 and the conversion is achieved with a special function gl.Strs, clf. the code by Nicholas Blaskey. One needs to append Go strings with "null termination", i.e. "\x00".

  For the record, a similar function in Ada, Zig: 1, 2, Rust: 1, 2, 3... Many of these Zig/Rust codes seem to ignore deallocation, but Zig-Game-Engine is an exception. This is all rather bureaucratic.

  In Nim, there are two main cases revolving around the packages "opengl" and "nimgl/opengl".

  1. cstringArray in the package "opengl": gltfviewer uses cstringArray with allocCStringArray and dealloc. Jack Mott does the same, but with deallocCStringArray, see also Samulus-2017. pseudo-random and treeform skip deallocations. Jason Beetham gets by with casting. Arne Döring does the same with self-hosted bindings which have the same "glShaderSource" signature.

  2. ptr cstring in the package "nimgl/opengl": Elliot Waite simply casts Nim's string to cstring and takes addr, without deallocations. anon767 does the same.

  Having made the choice of "nimgl/glfw" previously one would be inclined to go with "nimgl/opengl", but the "opengl" case looks cleaner so you will find the latter in this code. OpenGL is initialized with "glInit()" in "nim/opengl", but it is the function loadExtensions() that does it in "opengl".

  Notice that allocCStringArray and deallocCStringArray are in the standard lib/system.nim module, while the correpsonding Go and Ada solutions do not exist at the language/standard lib level and are only found in the custom user-made OpenGL bindings.

• What is the Go/Nim answer to the type void*? Consider this OpenGL function:

  void glVertexAttribPointer(	
  GLuint index,
  GLint size,
  GLenum type,
  GLboolean normalized,
  GLsizei stride,
  const void * pointer);

  Go with go-gl bindings: The type becomes unsafe.Pointer, clf. this file. The auxiliary "PtrOffset" function turns an integer into a required pointer with the unsafe.Pointer(uintptr(offset) expression. The Go code sets everywhere PtrOffset(0) as an argument to glVertexAttribPointer.

  Nim: The type is pointer, clf. this file. gltfviewer uses only nil value, but the case with non-zero offsets can be found in easygl, e.g. here which boils down to the expressions such as

  cast[pointer](3*float32.sizeof()). 

  Another example (with the "nim/opengl" package instead of "opengl") emphasizes the ByteAddress type instead of "int" before casting to Nim's "pointer", somewhat resembling Go's "uintptr".

• Nim's Case/Style Insensitivity. Check this out:

  GLFWCursorSpecial* = 0x00033001 ## Originally GLFW_CURSOR but conflicts with GLFWCursor type

  In the original GLFW C interface we have the GLFW_CURSOR constant and the GLFWCursor structure. In Nim these two become the same due its style rules.

  Here is another "ouch" situation in the "opengl" Nim package. Assume a perfectly normal-looking OpenGL function call somewhere in the user code:

  glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA16F.GLint, width.GLint, height.GLint, 0, GL_RGBA.GLenum, GL_FLOAT, nil)

  It does not compile however. The problem is that GL_FLOAT constant maps to "GLfloat* = float32" in opengl/private/types.nim. A fix is to set the 8th argument to

  glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA16F.GLint, width.GLint, height.GLint, 0, GL_RGBA.GLenum, cGL_FLOAT, nil)

  It maps to the correct "cGL_FLOAT* = 0x1406.GLenum" constant in [opengl/private/constants.nim].

  This is not as bad as Go's variable capitalization though. None of this is critical.

• Multiple hopeless attempts to make OpenGL easier: stisa-2017, AlxHnr-2017, floooh-2019, jackmott-2019, krux02-2020, liquidev-2021, treeform-2022...

• An ldd check on the final Ubuntu compiled binaries in Go and Nim:

  Go:

  tokyo@tokyo-Z87-DS3H:~/twinpeekz$ ldd twinpeekz
  linux-vdso.so.1 (0x00007ffc9cd9c000)
  libGL.so.1 => /lib/x86_64-linux-gnu/libGL.so.1 (0x00007f8ea74a3000)
  libX11.so.6 => /lib/x86_64-linux-gnu/libX11.so.6 (0x00007f8ea7363000)
  libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8ea727c000)
  libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8ea7054000)
  libGLdispatch.so.0 => /lib/x86_64-linux-gnu/libGLdispatch.so.0 (0x00007f8ea6f9c000)
  libGLX.so.0 => /lib/x86_64-linux-gnu/libGLX.so.0 (0x00007f8ea6f66000)
  libxcb.so.1 => /lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f8ea6f3c000)
  /lib64/ld-linux-x86-64.so.2 (0x00007f8ea7543000)
  libXau.so.6 => /lib/x86_64-linux-gnu/libXau.so.6 (0x00007f8ea6f36000)
  libXdmcp.so.6 => /lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f8ea6f2e000)
  libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007f8ea6f16000)
  libmd.so.0 => /lib/x86_64-linux-gnu/libmd.so.0 (0x00007f8ea6f07000)

  Nim:

  tokyo@tokyo-Z87-DS3H:~/twinpeekz2$ ldd main
  linux-vdso.so.1 (0x00007fff13588000)
  libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5e74d60000)
  libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5e74b38000)
  /lib64/ld-linux-x86-64.so.2 (0x00007f5e7505d000)

  Where has my libGL gone? The sizes of the binaries: 4.7MB (Go: default), 2.0MB (Nim: default), 1.1MB (Nim: d:release), 977KB (Nim: d:danger).

  It turns out that there are calls to C function "dlopen" at the runtime by Nim and the bindings. The libs loaded by "dlopen" are not known to ldd, lddtree, objdump, readelf which catch only what gets loaded at the pre-start of the program. Reading "/proc/PID/maps" does show the additional dependencies. Here is a more compact output of lsof command (strace did not work, but this SO might shed some light):

  tokyo@tokyo-Z87-DS3H:~/twinpeekz2$ pidof main
  15466
  tokyo@tokyo-Z87-DS3H:~/twinpeekz2$ lsof -p 15466|grep mem
  lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
        Output information may be incomplete.
  main    15466 tokyo  DEL    REG                0,1             7232 /memfd:/.glXXXXXX
  main    15466 tokyo  mem    CHR            195,255              939 /dev/nvidiactl
  main    15466 tokyo  mem    REG                8,3 32099568 6819477 /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.470.141.03
  main    15466 tokyo  mem    REG                8,3    18456 6819487 /usr/lib/x86_64-linux-gnu/libnvidia-tls.so.470.141.03
  main    15466 tokyo  DEL    REG                0,1             1025 /memfd:/.nvidia_drv.XXXXXX
  main    15466 tokyo  mem    REG                8,3   639848 6819479 /usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.470.141.03
  main    15466 tokyo  mem    REG                8,3   112856 6823495 /usr/lib/x86_64-linux-gnu/libxcb-glx.so.0.0.0
  main    15466 tokyo  mem    REG                8,3  1289616 6819471 /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.470.141.03
  main    15466 tokyo  mem    REG                8,3    84584 6822362 /usr/lib/x86_64-linux-gnu/libdrm.so.2.4.0
  main    15466 tokyo  mem    REG                8,3    14664 6823186 /usr/lib/x86_64-linux-gnu/librt.so.1
  main    15466 tokyo  mem    REG                8,3    21448 6823130 /usr/lib/x86_64-linux-gnu/libpthread.so.0
  main    15466 tokyo  mem    REG                8,3    14432 6822352 /usr/lib/x86_64-linux-gnu/libdl.so.2
  main    15466 tokyo  mem    REG                8,3    14048 6821909 /usr/lib/x86_64-linux-gnu/libX11-xcb.so.1.0.0
  main    15466 tokyo  mem    REG                8,3    18736 6821938 /usr/lib/x86_64-linux-gnu/libXinerama.so.1.0.0
  main    15466 tokyo  mem    REG                8,3    30912 6821930 /usr/lib/x86_64-linux-gnu/libXfixes.so.3.1.0
  main    15466 tokyo  mem    REG                8,3    43488 6821922 /usr/lib/x86_64-linux-gnu/libXcursor.so.1.0.2
  main    15466 tokyo  mem    REG                8,3    47728 6821948 /usr/lib/x86_64-linux-gnu/libXrender.so.1.3.0
  main    15466 tokyo  mem    REG                8,3    47504 6821946 /usr/lib/x86_64-linux-gnu/libXrandr.so.2.2.0
  main    15466 tokyo  mem    REG                8,3    76320 6821936 /usr/lib/x86_64-linux-gnu/libXi.so.6.1.0
  main    15466 tokyo  mem    REG                8,3    81640 6821928 /usr/lib/x86_64-linux-gnu/libXext.so.6.4.0
  main    15466 tokyo  mem    REG                8,3    22872 6821964 /usr/lib/x86_64-linux-gnu/libXxf86vm.so.1.0.0
  main    15466 tokyo  mem    REG                8,3 17167584 6821170 /usr/lib/locale/locale-archive
  main    15466 tokyo  mem    REG                8,3    47472 6822872 /usr/lib/x86_64-linux-gnu/libmd.so.0.0.5
  main    15466 tokyo  mem    REG                8,3    89096 6822206 /usr/lib/x86_64-linux-gnu/libbsd.so.0.11.5
  main    15466 tokyo  mem    REG                8,3    26800 6821926 /usr/lib/x86_64-linux-gnu/libXdmcp.so.6.0.0
  main    15466 tokyo  mem    REG                8,3    18720 6821915 /usr/lib/x86_64-linux-gnu/libXau.so.6.0.0
  main    15466 tokyo  mem    REG                8,3   166504 6823527 /usr/lib/x86_64-linux-gnu/libxcb.so.1.1.0
  main    15466 tokyo  mem    REG                8,3  1306280 6821911 /usr/lib/x86_64-linux-gnu/libX11.so.6.4.0
  main    15466 tokyo  mem    REG                8,3   141896 6821888 /usr/lib/x86_64-linux-gnu/libGLX.so.0.0.0
  main    15466 tokyo  mem    REG                8,3   715200 6821893 /usr/lib/x86_64-linux-gnu/libGLdispatch.so.0.0.0
  main    15466 tokyo  mem    REG                8,3   543056 6821882 /usr/lib/x86_64-linux-gnu/libGL.so.1.7.0
  main    15466 tokyo  mem    REG                8,3  2216304 6822210 /usr/lib/x86_64-linux-gnu/libc.so.6
  main    15466 tokyo  mem    REG                8,3   940560 6822861 /usr/lib/x86_64-linux-gnu/libm.so.6
  main    15466 tokyo  mem    CHR              195,0              940 /dev/nvidia0
  main    15466 tokyo  mem    REG                8,3   240936 6821873 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2

  A small binary does not mean much here as there are a lot of dynamic system dependencies. A few more useful links: the command "ldconfig -p" and linker vs runtime loader.

• White Space. Nim/Python white spaces make the code fragile in double loops where one needs to be extra careful not to push the last lines of the inner loop into the outter space, esp. when the "tabs" are only two-spaced, when the loops are long, when editing/rewriting takes place later. "gofmt" with "vim-go" is faster to type and more reliable.

• Naked imports are not a problem at all with Nim, paradoxically. You get into definitions with the right tools instantly (I use alaviss/nim.nvim), and the code becomes readable and terse without those package namespaces.

• Nim's "include" makes the compiler barf about duplication while "import" is demanding w.r.t. the manual markings of visibility. Function definition order within a file matters. Go made me think less about these matters.

• Go saved a lot of time as the GLTF library to load meshes both to CPU and GPU already pre-existed, but I would no longer push Go in 3D. Go is a new Erlang.

• cloc and clocrt:

  Language	files	blank	comment	code
  Nim	8	468	157	1368
  GLSL	7	107	89	261
  Markdown	1	95	0	235
  SUM:	16	670	246	1864

• Programming desktop 3D revolves around some big libs which become rather language-agnostic: GLFW/SDL, GLTF/Assimp, MGL vector math, stb_image, ImGui, OpenGL...

• Nim is a surprisingly productive language that one would hardly expect in a static non-GC space. The productivity is on par with Go or even better if we use only value types and scope-based "life time management". No need to clutter code with pointers.

• A punch line is still missing.