Next version!

[hishamhm]

Checklist for next version release:

• usability milestones
  • LuaRocks codebase converted to Teal
  • Teal codebase itself uses interfaces
• ensure that everyone who's on the current version has a clear path to migration
  • ensure APIs from master are compatible (vscode-teal, Cyan, Defold, etc.)
    • +1 from vscode-teal
    • +1 from Cyan
    • +1 from teal-language-server
  • pragma integration to avoid source-level compatibility issues (feat-arity)
  • ideally no known regressions (no open bugs that are next-only)

[github-actions]

Teal Playground URL: https://732--teal-playground-preview.netlify.app

[bjornbm]

I'm excited for this new release, but struggle to grasp the interfaces and probably other changes. Basically I don't see much updates/explanation in tutorial.md, neither of interfaces not the move to nominal types (#711, #708).

[euclidianAce]

From a quick glance I like the api changes and interfaces are a huge boon. Writing up some quick and dirty luv definitions (which is full of inheritance/interfaces) seems to be working well. I see interface where clauses are implemented as macros, which are also exposed via a new local macroexp foo(...) statement which after giving some thought is probably the best/lowest-amount-of-tying-users-of-teal-to-one-object-model way for a language like teal to add that sort of feature. Also nominal types are always a 👍 from me.

I do need to get back into the swing of working on teal related stuff, but I definitely fell into the got-a-real-job-and-now-have-less-time-to-work-on-hobby-projects trap that happens so often with open source.

All in all, looks great to me!

[catwell]

Same as @euclidianAce, real life (work + new kid) means I have little time for OSS but the changes I see in this branch are a lot of things I thought would be great for Teal.

Once it is a bit more stable I'll try it on my existing code bases in Teal.

[hishamhm]

@bjornbm tests and documentation are still pretty much missing, because I wanted to iterate as quickly as possible to experiment with the usability of these features... but once the dust settles we definitely need them!

[hishamhm]

@euclidianAce @catwell Thanks a lot for the feedback!! And, yeah, I totally get that sentiment — over the years I've accepted that my work on FOSS projects happens in bursts. When I find that I get a window of availability and motivation to work on this stuff, I try to get the most out of it. :)

For Teal, having a quiet time for the project since the last burst was a good thing as it allowed me to set priorities: seeing what are the gaps that people most often try to solve, etc. I think it's clear by now that the top two things missing in current Teal are interfaces and better nil handling. I don't think we'll get non-nilable types just yet, but it's the next thing on my mind. With --feat flags it should be easier to make gradual changes to the language in the future. This writeup has made the rounds recently and makes a good case for gradual language evolution via per-compilation-unit feature flags.

[hishamhm]

Also, speaking of writeups, it's funny that CI is failing because the tl.type_check function is exceeding the maximum number of upvalues per function in Lua 5.1. Looks like old Lua wasn't a fan of Carmack's long function style; I'll have to do some refactoring!

[hishamhm]

Update: it took me a big reorganization in d885f90 to get CI green. This is because the size of tl.tl started hitting size limits in both Lua 5.1 (number of upvalues) and Lua 5.4 (number of locals). The goal of the changes in this latest commit is to be able to finally break tl.tl into multiple files, but I wanted to do the reorg first to produce a sensible diff (which would let me more easily track the changes). The commit description includes more details on the internal changes.

[hishamhm]

Regarding "making the type system more nominal", here's one interesting example:

https://toot.kottman.xyz/@michal/111860362351389256

One thing I'm missing (or haven't yet found in docs) is "distinct type" (borrowing term from Nim) or integer based enum - e.g. local type LibraryErrorCode = distinct integer; function work():LibraryErrorCode

https://mastodon.social/@hisham_hm/111861339267098398

if you create a new nominal type for a base type like integer, it will be distinct across nominal types, but still be compatible with plain integer (so you can use the math. library etc)

This checks without errors in the master branch, but will check for Code against Age here in next:

local type Code = integer

local function get_code(): Code
end

local type Age = integer

local function get_age(): Age
end

local x: integer = get_code()
x = get_age()

local c: Code = get_code()
c = get_age()                         -- ERROR: in assignment: Age is not a Code

local a: Age = get_age()
a = get_code()                        -- ERROR: in assignment: Code is not a Age

[catwell]

I played a bit with the where clause on the current next branch, and I noticed that it doesn't work with generics. I mean that this works:

local type Success = record
    where self.error == false
    error: boolean
    value: integer
end

but not this:

local type Success = record<T>
    where self.error == false
    error: boolean
    value: T
end

It results in:

example.tl:2:11: missing type arguments in record<T>
example.tl:2:16: cannot index key 'error' in invalid 'self' of type self

Is that expected or do you intend to support it?

(Slightly related: can you release your FOSDEM slides? I know the video is probably broken, which is too bad, but the slides at least would be useful.)

[hishamhm]

@catwell thanks for the test case! I haven't really worked on the relationship between generics and interfaces/where yet, so that's a good starting point.

Is that expected or do you intend to support it?

I intended where to work alongside is, as in record __ is __ where, but having where on its own might be workable for unions. As for the generic there, I intend to support this test case, but I expect I'll need to place some limitations in the interactions between is ___ subtyping and generics. I'll probably over-restrict to be on the safe side first, then see what can be relaxed.

[catwell]

(Slightly related: can you release your FOSDEM slides? I know the video is probably broken, which is too bad, but the slides at least would be useful.)

The video was not broken \o/

[uramer]

How are where and __is intended to work for type declarations?

[hishamhm]

I think we're in a good shape to merge this into master!

• I've gotten positive feedback from @euclidianAce in [wip] test next-compat-api branch for tl cyan#46 — I guess that counts as a +1?
• @pdesaulniers vscode-teal support looks good for next as well, right?
• @svermeulen could you check if this works with teal-language-server? (existing Teal code might need --feat-arity=off to work out of the box but otherwise it should!)

Last thing before the new release will be merging this, writing some docs about interfaces,then we're good to go!

[svermeulen]

@hishamhm will do. I'll also start migrating my codebase to next branch and report any issues I find there

[catwell]

@hishamhm Trying to test with my Advent of Code code base but I have arity errors despite passing --feat-arity off, do you know why?

(I'd investigate but I really have 0 time to more than run the checks for now.)

[euclidianAce]

I've just pushed my tl-next branch of cyan that I've been working with locally and things are working great. So this is a 👍 from me

[hishamhm]

@hishamhm Trying to test with my Advent of Code code base but I have arity errors despite passing --feat-arity off, do you know why?

Oh! Some of the standard library functions ended up having slightly different definitions when they were ported to Teal in this branch. I pushed adjustments that makes your existing code pass.

There is one remaining tricky error triggering in the day12 for which I already found a workaround (but which would merit a nicer solution since this uncovered the fact that my empty-table inference does not backpropagate correctly in the case of map[c1][c2] = true). I'll try to code a solution quickly, otherwise I'll document the edge case in the testsuite and go with the workaround.

Edit: ccd2417 is the workaround — now the whole adventofcode repo typechecks on next with --feat-arity off!

Thank you @catwell !

[svermeulen]

I can confirm tls working well with tl next with just some minor fixes.

Would it be possible to get preview version of tl published to luarocks? Then I can do the same with tls. Or alternatively I can just wait until it merges and use an official release.

Btw - Really appreciate the arity and interface features. After upgrading it's clear these will be very useful in our codebase. Upgrading was pretty straightforward for us - even caught some potential runtime bugs after enabling the arity check

[hishamhm]

Would it be possible to get preview version of tl published to luarocks?

You can do luarocks install tl --dev --branch next, is that sufficient?

Or alternatively I can just wait until it merges and use an official release.

I think that should be fine. It should be a matter of days!

Btw - Really appreciate the arity and interface features. After upgrading it's clear these will be very useful in our codebase. Upgrading was pretty straightforward for us - even caught some potential runtime bugs after enabling the arity check

That is great to hear! Thank you so much for the positive feedback, it means a lot!!

[hishamhm]

@pdesaulniers I'm assuming it's a +1 from vscode-teal as well, since you've added the extra syntax highlighting in 0.10.0, and @V1K1NGbg has used vscode-teal 0.10.0 happily with tl next in his LuaRocks porting project for quite a while!

[pdesaulniers]

@hishamhm Autocompletion behaves a bit strangely right now, but I haven't had the time to debug this further... (Also, I'm not sure if these issues are actually specific to the next branch!)

I can at least reproduce this specific case:

-- test.tl

local function hello(): number
    return 1
end

if 1 == 1 then
    local abc = hello()
    local def = abc
    def = abc
    




end

$ ./tl types -p 12:1 test.tl
{"hello":9}

It seems like the two local variables abc and def are missing in the output. I'm running tl at commit 58ebd4b.

[catwell]

now the whole adventofcode repo typechecks on next with --feat-arity off!

2021 does, there are still a few errors on 2020 :)

In particular it does not like when I call this with a single argument (e.g in day13 part1):

function M.parse_integers(s: string, i0: integer) : {integer}
    local t, p = {}, i0 or 1
    local function f(x: string)
        t[p] = math.tointeger(x)
        p = p + 1
    end
    s:gsub("[-%d]+", f)
    return t
end

In day20 there is a different error probably related to the "empty table" issue:

local function count_sea_monsters(image: {string}): integer, integer
    local c, m, n = 0, {{}, {}}, 0
    for row = 1, #image - 2 do
        m[row + 2] = {}
        for col = 1, #image[1] - 19 do
            if has_sea_monster(image, row, col) then
                c = c + 1
                mark_sea_monster(m, row, col)
            end
        end
    end
    if c then
        for row = 1, #image do
            for col = 1, #image[1] do
                if image[row]:sub(col, col) == "#" and (not m[row][col]) then
                    n = n + 1
                end
            end
        end
    end
    return c, n
end

./common.tl:238:22: in assignment: got {}, expected {} | {}
./common.tl:249:68: cannot index object of type {} | {} with integer

Line 238 is m[row + 2] = {}, line 249 is if image[row]:sub(col, col) == "#" and (not m[row][col]) then.

[hishamhm]

2021 does, there are still a few errors on 2020 :)

I guess my for-loop in the shell missed a few entries then! Thanks for the test cases, I will take a look later.

[hishamhm]

@pdesaulniers Interesting. This works:

-- test.tl

local function hello(): number
    return 1
end

if 1 == 1 then
    local abc = hello()
    local def = abc
    




    def = abc
end

...so it looks like it is ending the scope "too early"? Might be a simple fix. BTW, looks like it also happens on master, so I probably won't treat it as a blocker!

[hishamhm]

@catwell

for f in `find . -path './day*/p*tl'`; do ( cd $(dirname "$f"); pwd; echo $f; TL_BIN_PATH=$HOME/github.com/teal-language/tl/tl ../tl.sh run $(basename $f); echo ); done

😬

All of 2020 and 2021 should be passing with --feat-arity off now!

[hishamhm]

@pdesaulniers That test case should be now fixed! It seemed to be specific to if-blocks

[pdesaulniers]

@hishamhm I think I found a tl types issue that is specific to the next branch. It seems like record 'methods' are missing in the output.

point.tl:

local record Point
    x: number
    y: number
end

function Point:init(x: number, y: number)
    self.x = x
    self.y = y
end

$ tl types point.tl > point.json

Before (master branch):

        "4": {
            "fields": {
                "init": 2,
                "x": 1,
                "y": 1
            },
            "file": "point.tl",
            "str": "type Point",
            "t": 131080,
            "x": 1,
            "y": 1
        },
        "5": {
            "fields": {
                "init": 2,
                "x": 1,
                "y": 1
            },
            "file": "point.tl",
            "str": "Point",
            "t": 131080,
            "x": 1,
            "y": 1
        },

After (next branch):

        "9": {
            "fields": {
                "x": 5,
                "y": 5
            },
            "file": "point.tl",
            "str": "record",
            "t": 131080,
            "x": 1,
            "y": 1
        },
        "10": {
            "fields": {
                "x": 5,
                "y": 5
            },
            "file": "point.tl",
            "str": "type record",
            "t": 131080,
            "x": 1,
            "y": 1
        },

'init' is absent in the fields list.

[hishamhm]

@pdesaulniers it should be fixed now! the construction of the type report now happens as part of the type checking pass. Records with functions are (IIRC!) the only types that "grow" over time as one reads through a file, and their type definitions were not being updated in the report (because the typeinfo entries are memoized). 6117c8c should fix this, testcase included.

[pdesaulniers]

Great, seems OK now!

I think I noticed another tl types issue. It seems like the 'ref' field is missing for record fields in some cases. It appears to depend on the order in which the record fields are declared.

local record Operator
    operator: string
end

local record Node
    node1: Node
    operator: Operator
end

local function node_is_require_call(n: Node): string
    if n.operator and n.operator.operator == "." then
       return node_is_require_call(n.node1)
    end
end

$ tl types test.tl > test.json

In the master branch, there was a "ref" field which would lead to the record declaration (I simplified the output to make it a bit easier to read):

    "by_pos": {
        "test.tl": {
            "11": {
                "9": 5,
                "24": 5,
            }
        }
    },
    "types": {
        "5": {
            "file": "test.tl",
            "ref": 6, /* HERE */
            "str": "Operator",
            "t": 268435456,
            "x": 15,
            "y": 7
        },
        "6": {
            "fields": {
                "operator": 1
            },
            "file": "test.tl",
            "str": "type Operator",
            "t": 131080,
            "x": 1,
            "y": 1
        },
    }

In the next branch, this 'ref' field is missing (see type 12):

    "by_pos": {
        "test.tl": {
            "11": {
                "9": 12,
                "24": 12,
            },
        }
    },
    "types": {
        "9": {
            "fields": {
                "operator": 7
            },
            "file": "test.tl",
            "str": "Operator",
            "t": 131080,
            "x": 1,
            "y": 1
        },
        "12": { /* MISSING REF HERE */
            "file": "test.tl",
            "str": "Operator",
            "t": 268435456,
            "x": 15,
            "y": 7
        }
    }

However, in the next branch, if I reverse the order of the fields in the Node record so that 'operator' is in the first position, the 'ref' field reappears in the output:

local record Node
    operator: Operator
    node1: Node
end

        "9": {
            "fields": {
                "operator": 7
            },
            "file": "test.tl",
            "str": "Operator",
            "t": 131080,
            "x": 1,
            "y": 1
        },
        "10": {
            "file": "test.tl",
            "ref": 9, /* HERE */
            "str": "Operator",
            "t": 268435456,
            "x": 15,
            "y": 6
        },