[move-compiler-v2] add parser code for lambda types

[brmataptos]

Description

Extend syntax

• lambda expression: move |args| body with copy+store. Note that:
  • we don't use has copy, store because commas will confuse parsing in many situation
  • we don't use has copy+store because that leads to nonuniform syntax with has
  • at some point Wolfgang suggested with, so that's what we're using.
• function types: |T1, T2| T3 with copy+store
• curry construction:
  • from lambda: |a, b| f(x, 3, a, b)
  • to simplify review, EarlyBind expression now takes a function-valued expression and a set of argument expressions, which correspond to a prefix of the function parameters.
• arbitrary function calls:
  • (|x, y| x + y)(2, 3)
• suppress printing common abilities (all functions have drop)

Modify exp_builder and lambda_lifter to generate function values.

Modify model to track "used" functions in addition to "called" functions to be able to catch all dependencies when function values are created but not called.

Attaches an AbilitySet to Type::Fun and Exp::Lambda based on source. Adds a new ExpCall operation in parser/expansion ASTs to be able to carry more generalized function calls through to move-model, which already can support this through Invoke, which previously was underutilized. Added basic type checking for function abilities.

Current Gaps

I was targeting the test cases return_func.move and doable_func.move, but this PR is still missing

• inference of ability store for function values built by curry with only storable parameters
• allowing storable function types as type parameters and function returns

I will try to patch those in while this is being reviewed, or add them to a later PR.

How Has This Been Tested?

Added more lambda tests under move-compiler-v2/tests/lambda/ which are run "with" and "without" lambda features enabled. Currently, many things pass through to hit "not yet implemented" errors in bytecode gen, etc.

Ran and carefully checked all tests under third_party and aptos.

Key Areas to Review

Key features are illustrated in test move-compiler-v2/tests/lambda/storable/doable_func.move and the two corresponding output files, along with the test return_func.move in the same directory. Most other test outputs should be largely unchanged, although error messages related to lambda use in default configuration have changed.

Tricky features are:

• parser is modified for lambda syntax extensions.
• In some cases, abilities may not be inferred well yet, so examples may need to be over-annotated with abilities.
• in ty.rs: unification checks Fun abilities as well as other things.
• in lambda_lifter.rs, we (1) reject lambdas without move free-var handling, (2) try to reduce lambda to curry, by checking for a simple function call with simple args, the last of which are identical to the lambda parameters.

Type of Change

• New feature
• Bug fix
• Breaking change
• Performance improvement
• Refactoring
• Dependency update
• Documentation update
• Tests

Which Components or Systems Does This Change Impact?

• Validator Node
• Full Node (API, Indexer, etc.)
• Move/Aptos Virtual Machine
• Aptos Framework
• Aptos CLI/SDK
• Developer Infrastructure
• Move Compiler
• Other (specify)

Checklist

• I have read and followed the CONTRIBUTING doc
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I identified and added all stakeholders and component owners affected by this change as reviewers
• I tested both happy and unhappy path of the functionality
• I have made corresponding changes to the documentation

[trunk-io]

⏱️ 2h 37m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
rust-move-unit-coverage	17m	🟩
rust-cargo-deny	15m	🟩 🟩 🟩 🟩 🟩 (+4 more)
rust-move-tests	13m	🟥
general-lints	12m	🟩 🟩 🟩 🟩 🟩 (+4 more)
rust-move-unit-coverage	12m	🟩
rust-move-unit-coverage	11m	🟩
rust-move-tests	10m	🟩
rust-move-tests	9m	🟩
check-dynamic-deps	8m	🟩 🟩 🟩 🟩 🟩 (+4 more)
rust-move-tests	8m	🟥
rust-move-unit-coverage	5m	🟥
rust-move-unit-coverage	4m	🟥
rust-move-unit-coverage	4m	🟥
rust-move-unit-coverage	4m	🟥
rust-move-unit-coverage	3m	🟥

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[brmataptos]

• Lambda support in VM: Wolfgang's PR plus some simplifications (removing mask, clarifying ops) #15387
• [move-compiler-v2] clean up a few remaining issues in lambda parser/front-end code #15365
• [move-compiler-v2] add parser code for lambda types #14792 👈 (View in Graphite)
• [compiler-v2] add test cases for lambda #14791
• [move-compiler-v2] fix #14633: fix error messages for lambdas, identify change points for more complete lambda implementation #14742
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codecov]

Codecov Report

Attention: Patch coverage is 79.67290% with 87 lines in your changes missing coverage. Please review.

Please upload report for BASE (09-27-extend_parser_for_lambda_types@243ea0f). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
third_party/move/move-compiler/src/parser/ast.rs	24.2%	25 Missing ⚠️
...hird_party/move/move-compiler/src/expansion/ast.rs	25.0%	24 Missing ⚠️
third_party/move/move-model/src/exp_rewriter.rs	55.8%	15 Missing ⚠️
third_party/move/move-model/src/ty.rs	56.5%	10 Missing ⚠️
third_party/move/move-model/src/ast.rs	50.0%	7 Missing ⚠️
...arty/move/move-compiler/src/expansion/translate.rs	96.2%	4 Missing ⚠️
...piler-v2/src/env_pipeline/unused_params_checker.rs	92.3%	1 Missing ⚠️
...d_party/move/move-model/src/builder/exp_builder.rs	98.4%	1 Missing ⚠️

Additional details and impacted files

@@                           Coverage Diff                           @@
##             09-27-extend_parser_for_lambda_types   #14792   +/-   ##
=======================================================================
  Coverage                                        ?    59.9%           
=======================================================================
  Files                                           ?      852           
  Lines                                           ?   207826           
  Branches                                        ?        0           
=======================================================================
  Hits                                            ?   124528           
  Misses                                          ?    83298           
  Partials                                        ?        0           

Flag	Coverage Δ
	59.9% <79.6%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

✅ Forge suite realistic_env_max_load success on 3fc195646e9b5078a1c1c1c867e5c617f65259b2

two traffics test: inner traffic : committed: 14106.85 txn/s, latency: 2817.49 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 7000 ms), latency samples: 5363720
two traffics test : committed: 99.98 txn/s, latency: 2241.65 ms, (p50: 1900 ms, p70: 2100, p90: 2200 ms, p99: 15500 ms), latency samples: 1760
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.970, avg: 1.208", "ConsensusProposalToOrdered: max: 0.331, avg: 0.293", "ConsensusOrderedToCommit: max: 0.378, avg: 0.361", "ConsensusProposalToCommit: max: 0.665, avg: 0.655"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.01s no progress at version 2310585 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 15.77s no progress at version 2310583 (avg 15.77s) [limit 16].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on 010570d3b7aa20889fb5ad0e5b23800aa33f5634 ==> 3fc195646e9b5078a1c1c1c867e5c617f65259b2

Compatibility test results for 010570d3b7aa20889fb5ad0e5b23800aa33f5634 ==> 3fc195646e9b5078a1c1c1c867e5c617f65259b2 (PR)
Upgrade the nodes to version: 3fc195646e9b5078a1c1c1c867e5c617f65259b2
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1322.89 txn/s, submitted: 1325.43 txn/s, failed submission: 2.54 txn/s, expired: 2.54 txn/s, latency: 2344.73 ms, (p50: 2000 ms, p70: 2400, p90: 4200 ms, p99: 5500 ms), latency samples: 114360
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1408.00 txn/s, submitted: 1411.14 txn/s, failed submission: 3.14 txn/s, expired: 3.14 txn/s, latency: 2079.53 ms, (p50: 1800 ms, p70: 2100, p90: 3100 ms, p99: 4800 ms), latency samples: 125660
5. check swarm health
Compatibility test for 010570d3b7aa20889fb5ad0e5b23800aa33f5634 ==> 3fc195646e9b5078a1c1c1c867e5c617f65259b2 passed
Upgrade the remaining nodes to version: 3fc195646e9b5078a1c1c1c867e5c617f65259b2
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1386.05 txn/s, submitted: 1390.02 txn/s, failed submission: 3.97 txn/s, expired: 3.97 txn/s, latency: 2254.21 ms, (p50: 2100 ms, p70: 2400, p90: 3900 ms, p99: 5700 ms), latency samples: 125720
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on 010570d3b7aa20889fb5ad0e5b23800aa33f5634 ==> 3fc195646e9b5078a1c1c1c867e5c617f65259b2

Compatibility test results for 010570d3b7aa20889fb5ad0e5b23800aa33f5634 ==> 3fc195646e9b5078a1c1c1c867e5c617f65259b2 (PR)
1. Check liveness of validators at old version: 010570d3b7aa20889fb5ad0e5b23800aa33f5634
compatibility::simple-validator-upgrade::liveness-check : committed: 13450.69 txn/s, latency: 2153.98 ms, (p50: 1800 ms, p70: 1900, p90: 2200 ms, p99: 16700 ms), latency samples: 526800
2. Upgrading first Validator to new version: 3fc195646e9b5078a1c1c1c867e5c617f65259b2
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6646.00 txn/s, latency: 4309.68 ms, (p50: 4700 ms, p70: 4900, p90: 5500 ms, p99: 5900 ms), latency samples: 127760
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6876.06 txn/s, latency: 4736.47 ms, (p50: 5100 ms, p70: 5200, p90: 6300 ms, p99: 6900 ms), latency samples: 231120
3. Upgrading rest of first batch to new version: 3fc195646e9b5078a1c1c1c867e5c617f65259b2
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 5262.60 txn/s, latency: 5408.15 ms, (p50: 5800 ms, p70: 6400, p90: 7200 ms, p99: 7500 ms), latency samples: 99780
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 5372.62 txn/s, latency: 6034.97 ms, (p50: 6500 ms, p70: 6800, p90: 7100 ms, p99: 8100 ms), latency samples: 181720
4. upgrading second batch to new version: 3fc195646e9b5078a1c1c1c867e5c617f65259b2
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 10597.66 txn/s, latency: 2580.65 ms, (p50: 2500 ms, p70: 3100, p90: 3800 ms, p99: 4100 ms), latency samples: 184560
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10262.87 txn/s, latency: 3082.23 ms, (p50: 2800 ms, p70: 3400, p90: 5500 ms, p99: 6200 ms), latency samples: 334100
5. check swarm health
Compatibility test for 010570d3b7aa20889fb5ad0e5b23800aa33f5634 ==> 3fc195646e9b5078a1c1c1c867e5c617f65259b2 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking