Rewriter to Visitor Transform Guide

Rewriter Upgrade → PIG VisitorTransform

General Procedure

(migrating FooRewriter → FooVisitorTransform)

1. Make a copy of FooRewriter named FooVisitorTransform
2. Similarly for the rewriter tests (if they exist) i.e. FooRewriterTests → FooVisitorTransformTests
3. Make the FooVisitorTransform have a base class of PartiqlAst.VisitorTransform rather than AstRewriter
4. Decide which code will require a upgrade to PIG
   1. Most commonly: anything involving ExprNode and ast.kt’s MetaContainer
5. For any call to override function rewriteXXX, find the equivalent transformXXX method in partiql-domains.kt
   1. First you need to find the ExprNode type’s equivalent PartiqlAst type. You can look at how the ExprNode is converted to a PartiqlAst in ExprNodeToStatement.kt. Usually, this is a pretty simple 1:1 mapping, but there can be some ambiguities and differences between ExprNode and PartiqlAst. Some cases can be found in the “Common Conversions” section.
   2. After finding the type to convert, there should be a corresponding transform function in partiql-domains.kt.
   3. You may need to add extension functions to do the same as an ExprNode but using PartiqlAst. Add these to lang/domains/util.kt
   4. Sometimes there are some trickier conversions. These cases are found in the sections below:
      1. Inner rewrites
         1. innerRewriteSelect
         2. innerRewriteDataManipulation
      2. PartiqlAst.Expr copying data
6. Write/translate tests, making sure they have a base class VisitorTransformTestBase
7. Change the usages of FooRewriter to use FooVisitorTransform
8. Once tests are passing and meet the desired standard, ensure existing code isn’t broken by the changes (run ./gradlew clean build)
9. Remove all parts of the old rewriter code:
   1. Delete unused imports
   2. Find all usages of the old rewriter (e.g. code, comments, kdoc) and replace them with the visitor transform version
   3. Delete the migrated rewriter (and its tests)
   4. (Optional but recommended) within FooVisitorTransform and its tests change usages of the term rewriter/rewritten to transformer/transformed
10. Rerun ./gradlew clean build to make sure everything works

Common Conversions

ExprNode	PartiqlAst	ExprNode Rewriter	PartiqlAst VisitorTransform	Notes
VariableReference	Expr.Id	rewriteVariableReference	transformExprId
PathComponentExpr	PathStep.PathExpr	rewritePathComponent	transformPathStepPathExpr
Literal	Expr.Lit	rewriteLiteral	transformExprLit
NAryOp.CALL	Expr.Call	rewriteNAry, then case on NAryOp.CALL	transformExprCall	First arg in ExprNode is the function name, so removal of this isn't necessary for PartiqlAst
FromSourceJoin	FromSource.Join	rewriteFromSourceJoin	transformFromSourceJoin
SelectProjectionPivot	Projection.ProjectPivot	rewriteSelectProjectionPivot	transformProjectionProjectPivot
SelectProjectionValue	Projection.ProjectValue	rewriteSelectProjectionValue	transformProjectionProjectValue
SelectProjectionList	Projection.ProjectList	rewriteSelectProjectionList	transformProjectionProjectList
FromSourceLet	FromSource.Scan and FromSource.Unpivot	rewriteFromSourceLet	transformFromSourceScan and transformFromSourceUnpivot	PartiqlAst does not have the supertype w/ Scan and Unpivot
FromSourceExpr	FromSource.Scan	rewriteFromSourceExpr	transformFromSourceScan
ScopeQualifier.LEXICAL	ScopeQualifier.LocalsFirst			ScopeQualifier.UNQUALIFIED is the same

Converting Metas

The constructor for metas is slightly different between ast’s MetaContainer and IonElement’s MetaContainer. Here is an example conversion:

metaContainerOf(UniqueNameMeta("foo"))

becomes

 metaContainerOf(UniqueNameMeta.TAG to UniqueNameMeta("foo"))

Generally, you do not want to keep ast’s MetaContainer and have to use the conversion function ( toPartiQlMetaContainer)

To add a new meta to an IonElement MetaContainer, you can use the withMeta(metaKey: String, metaValue: Any) function.

To add multiple metas, you will need to add two MetaContainers together:

metas = oldMetaContainer + metaContainerOf(newMeta)

Inner Rewrite Query

In the rewriter code, you may encounter innerRewriteSelect. This function calls the rewriter function for each clause of the SFW query. The direct equivalent in PIG is transformExprSelect. In most cases you should be able to just call transformExprSelect directly or the base class’s transform, super.transformExprSelect.

However, currently there are some slight differences between the order of transforms in innerRewriteSelect and transformExprSelect. innerRewriteSelect follows the traversal order (SQL semantic order) that is

FROM → (FROM LET) → (WHERE) → (GROUP BY) → (HAVING) → PROJECTION → (OFFSET) → (LIMIT)

transformExprSelect transforms the clauses in the order they are written for a PartiQL query, that is

PROJECTION → FROM → (FROM LET) → (WHERE) → (GROUP BY) → (HAVING) → (LIMIT) → (OFFSET)

This slight difference can lead to some different behaviors when translating innerRewriteSelect (e.g. StaticTypeRewriter). So to completely solve this issue, you can have your VisitorTransform implement VisitorTransformBase and call transformExprSelectEvaluationOrder rather than transformExprSelect to get the same behavior.

It's also worth noting that innerRewriteSelect and the visitor transform equivalent, transformExprSelectEvaluationOrder, can be used to avoid infinite recursion in the case of nested rewriter/transform instances.

fun transformExprSelectEvaluationOrder(node: PartiqlAst.Expr.Select): PartiqlAst.Expr {
    val from = transformExprSelect_from(node)
    val fromLet = transformExprSelect_fromLet(node)
    val where = transformExprSelect_where(node)
    val group = transformExprSelect_group(node)
    val having = transformExprSelect_having(node)
    val setq = transformExprSelect_setq(node)
    val project = transformExprSelect_project(node)
    val offset = transformExprSelect_offset(node)
    val limit = transformExprSelect_limit(node)
    val metas = transformExprSelect_metas(node)
    return PartiqlAst.build {
        PartiqlAst.Expr.Select(
            setq = setq,
            project = project,
            from = from,
            fromLet = fromLet,
            where = where,
            group = group,
            having = having,
            limit = limit,
            offset = offset,
            metas = metas)
    }
}

This also applies to innerRewriteDataManipulation, which has a slight different order than transformStatementDml.

(FROM) → (WHERE) → DML Operation vs. DML Operation → (FROM) → (WHERE)

To achieve the same behavior as the rewriter form, you can have your VisitorTransform implement VisitorTransformBase and call transformDataManipulationEvaluationOrder rather than transformStatementDml to get the same behavior.

It's also worth noting that innerRewriteDataManipulation and the visitor transform equivalent, transformDataManipulationEvaluationOrder, can be used to avoid infinite recursion in the case of nested rewriter/transform instances.

fun transformDataManipulationEvaluationOrder(node: PartiqlAst.Statement.Dml): PartiqlAst.Statement {
    val from = node.from?.let { transformFromSource(it) }
    val where = node.where?.let { transformStatementDml_where(node) }
    val dmlOperation = transformDmlOp(node.operation)
    val metas = transformMetas(node.metas)

    return PartiqlAst.build {
        dml(dmlOperation, from, where, metas)
    }
}

Copying Data from PartiqlAst.Expr

The PartiqlAst.Expr and all the subtypes don’t yet have a .copy function yet (will be added once copy() is released by PIG). This adds slightly more work when trying to create an instance of PartiqlAst.Expr from another. A workaround is to create a helper function as follows:

/**
  * Copies all parts of [PartiqlAst.Expr.Select] except [newProjection] for [PartiqlAst.Projection].
  */
private fun copyProjectionToSelect(node: PartiqlAst.Expr.Select, newProjection: PartiqlAst.Projection): PartiqlAst.Expr {
    return PartiqlAst.build {
        select(
            setq = node.setq,
            project = newProjection,
            from = node.from,
            fromLet = node.fromLet,
            where = node.where,
            group = node.group,
            having = node.having,
            limit = node.limit,
            offset = node.offset,
            metas = node.metas)
    }
}

Another annoyance you may run into is changing/copying metas for the generic PartiqlAst.Expr. This will be trivial once copyMetas() is released by PIG. A current workaround is mentioned here https://github.com/partiql/partiql-lang-kotlin/issues/311. You can create an inner class that overrides transformMetas.

inner class MetaVisitorTransform(private val newMetas: MetaContainer) : PartiqlAst.VisitorTransform() {
    override fun transformMetas(metas: MetaContainer): MetaContainer = newMetas
}

So in the ExprNode rewriters, rather than calling

node.copy(newMetaContainer)

For PartiqlAst transformers, you would call

MetaVisitorTransform(newMetaContainer).transformExpr(node)

Additional Notes:

• Some checks that are part of ExprNode rewrites easier using PIG VisitorTransforms. These (non-exhaustively) include checks that select star projection isn’t included with other projection items and clause arity checks.
• General NAry operations in ExprNode are easier to rewrite (like in PartiqlEvaluationRewriter.kt) than transforming in the PIG domain. This is because currently in PIG, NAry operations are directly PartiqlAst.Expr types. Once these operations are better modeled as NAry operations (https://github.com/partiql/partiql-lang-kotlin/issues/241), it will become easier to reuse code.
• After making FooVisitorTransform (copied from FooVisitorRewriter), delete the old imports, so you can be sure previous ExprNode types aren’t used. This is helpful when ExprNode types and PIG domain have the same type names
• There may be still be bugs in the conversion process between ExprNode and PartiqlAst (in ExprNodeToStatement.kt and StatementToExprNode.kt) that cause errors, especially when copying over the metas.

Sample Rewriter to VisitorTransform Conversions

• SelectListItemAliasRewriter
• FromSourceAliasRewriter
• AggregateRewriterSupport
• GroupByItemAliasRewriter
• GroupByPathExpressionRewriter and SubstitutionRewriter
• SelectStarRewriter
• PartialEvaluation example
• PipelinedRewriter
• StaticTypeRewriter