fix(redshift-driver): resolves issue where redshift column order can …

…come back in a different order, causing pre-aggregations across partitions to encounter a union all error (cube-js#6965) Thanks @jskarda829 @magno32 !

* fix(redshift-driver): resolves issue where redshift column order can come back in a different order, causing preaggregations across partitions to encounter a union all error

* feat: enable ordinal column position from env flag

This adds an environment variable, `CUBEJS_DB_FETCH_COLUMNS_BY_ORDINAL_POSITION` that optionally toggles the
ordinal column position during pre-aggregation.  Currently defaults to `false`.

---------

Co-authored-by: Jeremy Skarda <[email protected]>
Co-authored-by: Justin Smith <[email protected]>

docs/content/Reference/Configuration/Environment-Variables-Reference.mdx

@@ -447,6 +447,23 @@ The timeout value for any queries made to the database by Cube.
 | ---------------------------------------- | ---------------------- | --------------------- |
 | A number in seconds or a duration string | `10m`                  | `10m`                 |
 
+## `CUBEJS_DB_FETCH_COLUMNS_BY_ORDINAL_POSITION`
+
+Force fetching of columns by ordinal positions.  Certain data-providers (Redshift) do not guarantee columns in the
+same order on each request. (e.g. `SELECT * FROM foo`).  This flag ensures columns will be fetched in proper order
+for pre-aggregation generation.
+
+<InfoBox>
+
+    This flag currently defaults to `false`, as changing the value to `true` can cause breaking changes for existing
+    pre-aggregations.  This will eventually default to `true`.  For new deployments, consider setting this to `true`.
+
+</InfoBox>
+
+| Possible Values                                    | Default in Development | Default in Production |
+| -------------------------------------------------- | ---------------------- | --------------------- |
+| Whether to force fetch columns in ordinal position | `false`                | `false`               |
+
 ## `CUBEJS_DB_SNOWFLAKE_ACCOUNT`
 
 The Snowflake account identifier to use when connecting to the database.
@@ -1303,4 +1320,4 @@ The port for a Cube deployment to listen to API connections on.
 [snowflake-docs-warehouse]:
   https://docs.snowflake.com/en/user-guide/warehouses.html
 [wiki-tz-database]: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
-[ref-sql-api]: /backend/sql
+[ref-sql-api]: /backend/sql

packages/cubejs-backend-shared/src/env.ts

@@ -588,6 +588,16 @@ const variables: Record<string, (...args: any) => any> = {
     .default('true')
     .asBoolStrict(),
 
+  /**
+   * Fetch Columns by Ordinal Position
+   *
+   * Currently defaults to 'false' as changing this in a live deployment could break existing pre-aggregations.
+   * This will eventually default to true.
+   */
+  fetchColumnsByOrdinalPosition: (): boolean => get('CUBEJS_DB_FETCH_COLUMNS_BY_ORDINAL_POSITION')
+    .default('false')
+    .asBoolStrict(),
+
   /** ****************************************************************
    * JDBC options                                                    *
    ***************************************************************** */

packages/cubejs-base-driver/src/BaseDriver.ts

@@ -370,7 +370,8 @@ export abstract class BaseDriver implements DriverInterface {
              columns.table_schema as ${this.quoteIdentifier('table_schema')},
              columns.data_type  as ${this.quoteIdentifier('data_type')}
       FROM information_schema.columns
-      WHERE table_name = ${this.param(0)} AND table_schema = ${this.param(1)}`,
+      WHERE table_name = ${this.param(0)} AND table_schema = ${this.param(1)}
+      ${getEnv('fetchColumnsByOrdinalPosition') ? 'ORDER BY columns.ordinal_position' : ''}`,
       [name, schema]
     );