LuaRocks dependencies missing type information

[PedroHase]

Hey all,

as someone coming from Typescript I instantly fell in love with Teal. However, I am now struggling with typing LuaRocks dependencies.

I want to use luasocket in my project's main.tl and tried using the defined types in Teal-Types, but after copying the types folder with luasocket into my project, I am still getting the error that there is no type information for the required module.

I noticed inside the copied declaration files the same error occurs, e.g. in socket.d.tl which requires ltn12.d.tl directly through require("ltn12"). When changing the require path to the relative path with require("types.luasocket.ltn12"), however, the type error disappears. Same goes for the import in main.tl, but this will lead to lua issues, since this would be the wrong import path.

Is this maybe caused by a configuration issue on my side? How should dependencies originating from LuaRocks should be typed in general / how should a project be set up?

My tlconfig.lua looks like this:

return {
    include_dir={"src"},
    source_dir="src",
    build_dir = "dist/",
    global_env_def = "globals"
}

My project structure looks like this:

tlconfig.lua
src/
├─ types/
│  ├─ luasocket/
│  │  ├─ socket.d.tl
│  │  ├─ ltn12.d.tl
│  │  ├─ ...
│  ├─ globals.d.tl
├─ server/
│  ├─ main.tl
dist/

[lenscas]

put the .d.tl files next to main.tl. If the require has a path separator (in lua a . in the require also counts) then mimic that path with folders. So, if you have local a = require"some.path.to.file then make sure you have a file.d.tl file inside /some/path/to/ with the some folder living in the root of your project.

It is known that this is not ideal but people haven't gotten the time yet to properly think of and implement a solution for this yet. Ideally it is something that is fixed within cyan to make it smart enough to tell tl.tl (probably through just updating the package.path) where to look for .d.tl files (and ideally, even an easier way of getting those files)

[PedroHase]

Hey @lenscas thanks for your swift reply! Suppose I want to require luasocket in my main.tl like this:

-- using luasocket rock
const http = require("socket.http");

Where should I place the type declarations from Teal-Type (since luaRocks installs dependencies by default in /usr/share/lua/<lua-version>)? Do I have to it similarly to teal-language/teal-types#21 (comment) and install the dependencies in a local tree (e.g. luaModules)? Or is there a way to keep using global luaRocks dependencies while having teal declarations?

Also, would I have to manually update the require paths in the type declarations from Teal-Types? E.g. this declaration does not work, but must be changed to types.luasocket.ltn12 on my side

[lenscas]

A require like socket.http requires you to make a folder named socket next to your main.tl file. In this folder you place the http.d.tl file.

[PedroHase]

I got it to work by moving the contents of the luasocket folder from Teal-Type into src as @lenscas suggested. However, when using many dependencies, the src folder will be cluttered with declaration files, unfortunately. Is there maybe a way to separate the declaration files (like I tried by moving them into a types folder), or is this not possible yet?

[lenscas]

Second part of my first reply goes over that

It is known that this is not ideal but people haven't gotten the time yet to properly think of and implement a solution for this yet. Ideally it is something that is fixed within cyan to make it smart enough to tell tl.tl (probably through just updating the package.path) where to look for .d.tl files (and ideally, even an easier way of getting those files)

You could make a thin wrapper around tl.tl that changes the paths it looks for to include your types folder. Ideally though if you go that route you make a change that can be merged upstream so that everyone benefits :)

[Aire-One]

I have solved this problem in my teal project with the following structure:

.
├── build                # Where cyan will put the generated .lua files
├── justfile
├── lua_modules          # Luarocks local tree
├── set_paths.lua        # Generated from set_paths.tl to run the program from builded Lua files
├── set_paths.tl         # Teal version of set_paths.lua from https://leafo.net/guides/customizing-the-luarocks-tree.html
├── src
│   └── <project name>   # Source files
├── tlconfig.lua
└── types                # Teal type definitions

The config files are:

-- tlconfig.lua
return {
   build_dir = "build",
   source_dir = "src",
   include_dir = {
      "src/<project name>",
      "types",
   },
}

-- set_paths.tl
local version = _VERSION:match "%d+%.%d+"

local function lua_module_paths(module_base_path: string): string
   local paths = (module_base_path .. "/?.lua;")
      .. (module_base_path .. "/?/init.lua;")

   return paths
end

package.path = lua_module_paths("lua_modules/share/lua/" .. version)
   .. lua_module_paths "types"
   .. lua_module_paths "src/<project name>"
   .. package.path

package.cpath = "lua_modules/lib/lua/" .. version .. "/?.so;" .. package.cpath

# justfile
lua_version := "5.4"
lua := "lua" + " -l set_paths"

tl := "tl run" + " -l set_paths"

rocks_tree := "lua_modules"
rockspec_file := "<project name>-dev-1.rockspec"

lua_targets := `find src/ -type f -iname '*.lua' | xargs` + " " + `ls *.lua | xargs`
teal_targets := `find src/ types/ -type f -iname '*.tl' | xargs`

install:
  luarocks \
    --lua-version {{ lua_version }} \
    install \
    --tree {{ rocks_tree }} \
    --only-deps \
    {{ rockspec_file }}

uninstall:
  rm -rf {{ rocks_tree }}

check-lua *FLAGS:
  luacheck {{ lua_targets }} {{ FLAGS }}

check-teal:
  cyan check {{ teal_targets }}

check-format:
  stylua --check {{ lua_targets }}

check: check-lua check-format check-teal

build:
  cyan build

clean:
  rm -rf build

run:
  {{ tl }} src/<project name>/init.tl

run-lua:
  {{ lua }} build/<project name>/init.lua

gen-lua-set_path:
  cyan gen --output set_paths.lua --gen-compat "off" set_paths.tl
  sed -i 's/"src\/<project name>"/"build\/<project name>"/g' set_paths.lua
  stylua set_paths.lua

Replace all the <project name> occurrences by your actual project name.

I could move the types directory to the src/types location. But from what I have seen on Typescript projects, they also prefer to put this directory at the root. I have the idea that one day, Luarocks will ship the .d.tl files, and we would only need to remove this types directory.

I run the program only with tl. It is still possible to run the generated Lua files with the just run-lua command, and optionally the just gen-lua-set_path to make sure the set_paths.lua file is correct.

[PedroHase]

Hey @Aire-One,

After tinkering a bit I noticed that your solution requires a init.tl file & calling lua directly to execute the file. Unfortunately, I do not run the code directly, but rather it is called by an embedded Lua from another app (I basically start a server which consumes the lua files). This also means I do not have a clear entry file to go by.

The structure of my src folder (where the source code lies) looks like this:

src/
├─ Server/
│  ├─ MyPluginName/
│  │  ├─ main.tl
│  │  ├─ websocket.tl

(each folder in Server equals a server plugin)

When building, the files need to go into a Resources folder, the Lua inside is then consumed by the embedded Lua. The build itself as well as Teal typings work without issues, but when I want to use dependencies from LuaRocks, the embedded Lua VM cannot find the dependency.

E.g. when I require rapidjson in websocket.tl like

local json = require("rapidjson")

I get following error:

[LUA ERROR] Failed: "Resources/Server/TestPlugin/websocket.lua": [string "Resources/Server/TestPlugin/websocket.lua"]:1: module 'rapidjson' not found:
        no field package.preload['rapidjson']
        no file '/usr/local/share/lua/5.3/rapidjson.lua'
        no file '/usr/local/share/lua/5.3/rapidjson/init.lua'
        no file '/usr/local/lib/lua/5.3/rapidjson.lua'
        no file '/usr/local/lib/lua/5.3/rapidjson/init.lua'
        no file '/usr/share/lua/5.3/rapidjson.lua'
        no file '/usr/share/lua/5.3/rapidjson/init.lua'
        no file './rapidjson.lua'
        no file './rapidjson/init.lua'
        no file 'Resources/Server/TestPlugin/rapidjson.lua'
        no file 'Resources/Server/TestPlugin/lua/rapidjson.lua'
        no file '/usr/local/lib/lua/5.3/rapidjson.so'
        no file '/usr/lib/x86_64-linux-gnu/lua/5.3/rapidjson.so'
        no file '/usr/lib/lua/5.3/rapidjson.so'
        no file '/usr/local/lib/lua/5.3/loadall.so'
        no file './rapidjson.so'
        no file 'Resources/Server/TestPlugin/rapidjson.so'
        no file 'Resources/Server/TestPlugin/lib/rapidjson.so'

What would be a good solution to this issue? Should I manually copy the dependencies from lua_modules into the Resources build folder (and if so, what files would I need to copy)? Or is there another way to do that? I could alternatively install the LuaRocks dependencies globally, but I'd like to not do that if possible (in order to allow easier distribution).

[lenscas]

you can probably do something with the package.path to get lua (and teal) to look at the lua_modules folder

[Aire-One]

Oh yes, I see. The Server application is not aware of Luarocks and even less of the local tree installation. I'm not sure whether there is currently a correct solution to this issue.

A few points that can be explored:

• (As mentioned by @lenscas) Manipulate the package.path. This is the "require("set_paths") is a bad idea" scenario from Leafo blog post.
• Support custom paths in the Server application. So you could invoke the Server with an equivalent of set_paths. (Still not idea for distributing your project)
• Make the Server application Luarocks aware with require "luarocks.loader" from https://github.com/luarocks/luarocks/wiki/Using-LuaRocks#multiple-versions-using-the-luarocks-package-loader. I'm not sure how it will play with the local tree.
• Add some sort of "package system" to bundle the dependencies with your code into the Resources directory. It looks really cool to me, but I'm not aware of any implementation of such systems.
• more alternatives I don't think of :/

[PedroHase]

Thanks for the reply @Aire-One ! I "fixed" the issue by implementing a dumbed down "package system" script. It basically goes through the rock_manifest files for each used LuaRocks dependency and copies the lib and lua files/folders into my Resources build folder. The embedded Lua VM can then access the dependencies there.

[Aire-One]

That's so cool! Can you share your script?

I haven't played with that yes, but I suspect it can be run by cyan as a "post compilation script" https://github.com/teal-language/cyan/blob/main/docs/tlconfig.md#scripts.

[Aire-One]

I think your include_dir should be

include_dir = {
   "src/server",
   "src/types",
},

and global_env_def should be global_env_def = "src/types/globals"