Vector literals?

[petrihakkinen]

Currently literal vectors always seem to generate fast-call to the vector constructor. I think it would make sense (perf. wise) to support vector literals by storing them in the constant table. Are there any plans for this?

[zeux]

I don't think we've discussed this before. We can do analysis on our internal code base; does this come up often in your experience?

Something we have discussed is that it would be nice to make the integration of vectors more first class somehow, e.g. have some ways to work with vectors that doesn't require a lot of external scaffolding. Maybe that can then imply that we can do more optimizations around them...

Separately, a concrete issue around specifically vector literals is that we'd need to somehow support 3-vector and 4-vector forms from the same bytecode. I don't think we should bifurcate the bytecode format, and the compiler doesn't even know right now if it's targeting 3-vector or 4-vector environment. This could be easier if for example we only support literals with a fixed 4-th component (e.g. say it's always 0 or 1) as that would make the 2-variant encoding more straighforward.

[petrihakkinen]

Yes, I think so. I searched all vector literals from our codebase and here are some raw numbers:

vector(n, n)          273 matches
vector(n, n, n)       7853 matches
vector(n, n, n, n)    3262 matches

The large number of 3d and 4d vectors come from some generated VFX data scripts. These scripts are not executed in inner loops, so performance is not critical for them (although storing the literals in the constant table would make the bytecode smaller). Filtering the data scripts out leaves the following:

vector(n, n)          273 matches
vector(n, n, n)       992 matches
vector(n, n, n, n)    235 matches

As expected 3d vectors are most common, but 2d and 4d vectors still have significant numbers. Typical usage for 3d vectors are constant offsets to positions, fixed positions in local/world space and various parameter values (colors etc.).

Our vector constructor accepts 0-4 arguments and the omitted arguments are filled with zeroes. I think this makes most sense and allows using the same constructor for 2d, 3d and 4d vectors. We don't use homogeneous coordinates because we don't expose matrices in the Lua bindings -- quaternions are used for orientations -- so setting the fourth component to one would have felt too much of a special case and does not mesh well with 2d vectors.

That brings up the question how should the compiler handle argument counts != 3. One solution is to pad omitted arguments with zero, but is it too much to assume everyone uses this convention? Another solution is to only support vector literals with 3 components (and generate the fast-call as usual for other cases). Of course we'd be leaving some performance on the table with this option but it would be better than nothing...

I don't personally see a big problem if bytecode would be dependent on LUA_VECTOR_SIZE though. An application uses either LUA_VECTOR_SIZE 3 or 4, never both, and I can't think of a case where the bytecode has to be portable between applications. Perhaps I'm missing some scenario? But it could certainly add confusion and at least LUA_VECTOR_SIZE should be added to the header to detect incompatible bytecode.

Another option is to always store 4 components in the constant table, even for LUA_VECTOR_SIZE = 3 (and pad the fourth component to zero?). This of course would bloat the constant table by 25% with the default settings...

All in all, perhaps the simplest way is to only support vector literals with exactly three components? That sidesteps all the complications (padding, bytecode versioning, etc.) I think. We can always re-evaluate other vector sizes in the future.

[zeux]

One issue is that the compiler doesn't even depend on anything that would contain LUA_VECTOR_SIZE definition, and I think it's better that it stays that way as it enforces separation of build environments (Compiler doesn't depend on VM, intentionally so as it makes it easier to ensure cross compilation works -- Roblox doesn't compile bytecode on the clients for example). We could pass that via compile options as well as other vector related bits of course, but then the bytecode format would start depending on compile options... I think all of this is technically feasible but just suggests this is the wrong route to take because it increases divergence between build environments and configurations.

Just supporting 3 arguments seems like a reasonable compromise. Roblox Vector3.new also defaults to 0 for 1-2 but I'm not sure that it's a reasonable assumption, because there's other reasonable interfaces and due to the lack of full formalized specification I mentioned it's unclear what the behavior should be.

[petrihakkinen]

I realized we cannot escape the padding issue completely even if we supported only 3 arguments. When LUA_VECTOR_SIZE = 4, we still have to decide if we pad the fourth component with 0 or 1 in luau_load. I suggest padding always with zero (see my other message for rationale).

And once we do that, there's no reason to not support 2 arguments really. It's only logical and consistent to pad the third and fourth component with zeroes.

With the above, the compiler could support 2d and 3d vector literals.

For the 4d case, I also realized we don't have to configure vector size for the compiler and have different bytecode formats if we did the following:

• emit LBC_CONSTANT_VECTOR3 for 2d and 3d literals
• emit LBC_CONSTANT_VECTOR4 for 4d literals

(Side note: We could also consider adding LBC_CONSTANT_VECTOR2 to save a few bytes from bytecode dumps)

Trying to load bytecode with LBC_CONSTANT_VECTOR4 when VM is configured with LUA_VECTOR_SIZE = 3 would cause an error in luau_load. Note that technically calling vector(x, y, z, w) can be considered an error when LUA_VECTOR_SIZE = 3, so this solution would just move the error from runtime to load time.

[zeux]

I don't think I agree with this. Not every embedder might pad with zeroes for 2-vector case, note that 4-vector case is not the default. When using 3-vectors, using 4 arguments may not be an error - indeed, it's not in Roblox, and it's unusual to error when more arguments is provided. So both padding 2-vectors with zero and erroring on 4-vectors when VM is compiled with 3-vectors will have a potential to introduce regressions and deviations from current behavior. Handling 3-vectors is provably safe but everything else isn't.

[petrihakkinen]

Ok, fair enough. Let's stick with 3 arguments for now. I can try to implement a proof of concept soon.

You're right that we really should come up with a more formal spec for vectors. Another symptom of this is that the REPL and compiler executables do not do the necessary initialization for the vector type so it's not possible to test Lua scripts with vectors without modifying the code first, which is a bit annoying.

[petrihakkinen]

Something we have discussed is that it would be nice to make the integration of vectors more first class somehow, e.g. have some ways to work with vectors that doesn't require a lot of external scaffolding. Maybe that can then imply that we can do more optimizations around them...

I don't mind having to implement our own vector operations personally, but having more first-class operations would certainly speed things up. For example, length, normalization, dot product, etc. are fairly common and could be implemented as fast calls if they were part of the VM.

However, a major problem I see is that there are many kinds of conventions for these. For example, Roblox API takes an OOP approach, while our Lua bindings are completely procedural and vector functions are global functions resembling vector ops in shader languages. Perhaps a more generic solution is to allow implementing new fast-calls in user code or come up with another mechanism for doing faster native function calls (you mentioned this earlier)? We have some other hotspot application specific functions that would benefit from user specified fast-calls too.

[zeux]

I don't mean anything nearly as large & complex as Roblox Vector3 API certainly. It's more about:

• There's no way to create a vector object in code that is portable across embedders
• There's some ambiguity about dynamic access (is it okay to do v[comp] where comp is dynamically a string "X"?)
• There's no way to refer to vector type via a type annotation in a way that is portable across embedders (more narrowly relevant)

Having a more formalized specification with an absolutely minimal interface could improve this, even if we end up adding just one vector constructor and blessing one particular way to spell the type, even if others are valid. I think in this case we would definitely diverge from how Roblox spells this out, which would introduce more than one way to do some of these basic operations in Roblox but that seems possibly fine because we'd get slightly more portable vector type out of it maybe.

[petrihakkinen]

Right, I see. A formal spec for the vector constructor would indeed make this feature easier to implement.

This is how it works in our engine (some of this I already mentioned in previous messages):

• The constructor is a global function simply called vector. I think this is a good name because it reflects the type name (internally but also what is returned by type). Vector size is not part of the name (e.g. "vector4"), because there's really just one vector type which can be used for 1d, 2d, 3d and 4d math.
• The constructors accepts 0-4 to components and omitted arguments are filled with zero. The idea here is that this allows using the same constructor for 2d, 3d and 4d vectors. Technically only the last two components would need to be optional, but we made all of them optional for consistency. Casting a single number to vector with vector(x) can also be useful sometimes.
• Padding the fourth component to one was considered but it felt too much of a special case. For example, why should vector(x, y) return (x, y, 0, 1)? Doesn't seem sensible. For the rare cases where fourth component needs to be one, it's better to be explicit and provide all four components.
• We use lower case x for indexing. Lowercase is used elsewhere in our engine for vector indexing and also in shader languages so there's less friction when switching languages. Also Lua built-ins are in lowercase.

Maybe this could be a good starting point for the spec?

[petrihakkinen]

Oh, I just noticed luauF_vector already pads the fourth component with zero, so in that sense it's already decided :)

[petrihakkinen]

There's some ambiguity about dynamic access (is it okay to do v[comp] where comp is dynamically a string "X"?)

FWIW, I don't think I've ever used dynamic vector component indexing. If dropping dynamic indexing removes the need to implement __index for vectors in user code, that would make embedding Luau a bit more pleasant for sure (I think forgetting to implement __index for vectors is a common pitfall for embedders). But I think there are other code paths that trigger __index too...?

[petrihakkinen]

Implementation seems to be fairly straightforward. Compiler and VM seems to work fine, haven't looked at codegen yet. I'll analyze performance next.

Work in progress:
https://github.com/petrihakkinen/luau/tree/petrih-vector-literals

Note: this relies on built-in folding which is only enabled on -O2.

I'll extend this to a proper PR later...

[HaroldCindy]

@petrihakkinen Semi-related, but do you have any more detail about how you're using 4 component vectors?

I'm also interested in having a mix of Vector3s and Quaternions (all stored as 4-component vectors,) but there's no way to associate a metatable with a Vector, so I can't see any way to have different math operations for Vector3 vs Vector4 vs Quaternion without using utility fastcalls instead of operators. Is that what you do?

[petrihakkinen]

Vector is just a primitive data type for us, similar to how it is in shading languages and, for example, how SIMD vectors work on the CPU. All operations such as dot product, computing the length, transforming a vector with a quaternion, etc. are implemented as functions.

[HaroldCindy]

That makes sense, thanks!