Lambda Parsing Bug Fix

Found a black inspired reformat that tickled a bug in the func_adl parser. Updated.

What's Changed

• Infinite loop if this lambda is parsed in Jupyter by @gordonwatts in #111

Full Changelog: 3.1.1...3.1.2

Lambda Parsing Upgrade

Correctly parse a lambda formatting that black will often impose:

    my_obj().do_it(
        lambda e: e.Jets("AntiKt4EMTopoJets").do_it(
            lambda j: j.Jets("AntiKt4EMTopoJets").do_it(
                lambda j1: j1.pt() / 1000.0
            )
        )
    )

Lambda Parsing Upgrade

This point release is focused on correctly parsing lambda's - increasing the range of styles of code we can correctly detect inside a Select, SelectMany and Where statement.

For example, the following two statements now work (and didn't properly work before):

    my_obj().Select(lambda x: x
                   + 1
                   + 2
                   + 20
                   )

and

    my_obj().do_it(lambda event: event + 1
                   ).do_it(lambda event: event)

Since this is using heuristics, likely there are cases we've missed. Please do not hesitate to submit a bug report (or PR).

What's Changed

• Support multi-line lambda functions which are valid before complete by @gordonwatts in #103
• Test to make sure multi-line lambda with funny breaks works by @gordonwatts in #105
• Add as_awkward and as_pandas, etc by @gordonwatts in #106
• Python 3.11 support by @gordonwatts in #107

Full Changelog: 3.0...3.1

AST Parsing and Python 3.11

What's Changed

• Support multi-line lambda functions which are valid before complete by @gordonwatts in #103
• Test to make sure multi-line lambda with funny breaks works by @gordonwatts in #105
• Add as_awkward and as_pandas, etc by @gordonwatts in #106
• Python 3.11 support by @gordonwatts in #107

Full Changelog: 3.0...3.1b1

Types & Metadata

This is a major release. Old queries against old backends should work without any changes. However, you'll need a new backend to take advantage of most of the new features.

Features marked with a ** require the new backend.

New Features in the language

• C++ Code injection can be done via a decorator
• List comprehension and Generator expressions are now supported, automatically turned into func_adl statements. This is particularly helpful when doing 2-object matching.
• Local and global variables are properly captured in lambda's.

New Features in the infrastructure

• Support for Query Metadata. This metadata is sent to the backend without being touched **
• Support for in-query Metadata. This metadata is not sent to the backend, but can be extracted from the query. The in-query metadata can be overridden by setting a value more than once. **
• A Fully typed ObjectStream is supported via Generics. Various editors that type-follow will be able to correctly predict methods, etc., that are available. You'll need a complete object model for this to work (see the func_adl_servicex_xaodr21 package) **
• Python 10 is supported.

The PR's

• Add a function to allow MetaData over the wire by @gordonwatts in #78
• Added typing into ObjectStream by @gordonwatts in #80
• Support Python 10 by @gordonwatts in #86
• Extensability via the type system by @gordonwatts in #82
• Pythonic Decorators for Code Injection by @gordonwatts in #87
• Support Lambda Capture by @gordonwatts in #88
• Turning on black as a formatter by @gordonwatts in #89
• Parameterized Method Calls by @gordonwatts in #90
• Pr_query_metadata by @gordonwatts in #91
• Support ListComp and Generator expressions by @gordonwatts in #92
• Fixes to get a version that runs in production by @gordonwatts in #94
• More robust parsing of ast's from callables by @gordonwatts in #97
• Regression (untested) for lambdas continuing accross lines by @gordonwatts in #98
• Calls to AsXXX with column names as single items or single arrays should be the same by @gordonwatts in #99
• Bug Fix: argument names should override captured variables by @gordonwatts in #100

Full Changelog: 2.3...3.0

Fix multi-line ast parsing regression

Regression (untested) for lambdas continuing accross lines (#98)

* Fix up multi-line function parsing

Add better ast parsing

• More robust in finding and parsing source code in methods
• Ability to provide the finder hints

Allow IfExpr to see Any as a float

Minor bug when any's get used in IfExpr. I get the feeling for untyped stuff, we'll be finding a lot of things like this.

Production Running

Can run the complete notebook that demos xaod_usage

• Do not require clients to install the func-adl.ast package.
• Fix bug in how QMetaData updates items in its dict (overriden items were ignored!)
• Add a warning if there is a lambda that isn't scanned for type forwarding
• Fix bug where function names were being captured during lambda resolution
• Move to a single package rather than two (no more .ast). This is certianly a breaking change!

Bug Fixes

• incorrect __init__ file - made meta data clearing for the unreleased version of func_adl_servicex fail.
• QMetaData was not dealing with updated items properly.