Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Provide spark catalog, dsv2 and use parquet for copy/unload #120

Closed
wants to merge 175 commits into from
Closed

[WIP] Provide spark catalog, dsv2 and use parquet for copy/unload #120

wants to merge 175 commits into from

Commits on Nov 8, 2017

  1. Notes about inlining this in Databricks Runtime.

    @rxin
    rxin authored Nov 8, 2017
    Configuration menu
    Copy the full SHA
    717a4ad View commit details
    Browse the repository at this point in the history

  2. Make the note more obvious.

    @rxin
    rxin authored Nov 8, 2017
    Configuration menu
    Copy the full SHA
    184b442 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2018

  1. Remove itests. Fix jdbc url. Update Redshift jdbc driver

    @dichiarafrancesco
    dichiarafrancesco committed May 9, 2018
    Configuration menu
    Copy the full SHA
    a3a39a2 View commit details
    Browse the repository at this point in the history

  2. Merge pull request spark-redshift-community#1 from Yelp/fdc_DATALAKE-… 

    …563_remove-itests-from-public

Remove itests. Fix jdbc url. Update Redshift jdbc driver
    @dichiarafrancesco
    dichiarafrancesco authored May 9, 2018
    Configuration menu
    Copy the full SHA
    cafa05f View commit details
    Browse the repository at this point in the history

  3. Fix double type to float and cleanup

    @dichiarafrancesco
    dichiarafrancesco committed May 9, 2018
    Configuration menu
    Copy the full SHA
    ab8124a View commit details
    Browse the repository at this point in the history

  4. Avoid logging creds. log sql query statement only

    @dichiarafrancesco
    dichiarafrancesco committed May 9, 2018
    Configuration menu
    Copy the full SHA
    3230aaa View commit details
    Browse the repository at this point in the history

  5. Add bit and default types

    @dichiarafrancesco
    dichiarafrancesco committed May 9, 2018
    Configuration menu
    Copy the full SHA
    3384333 View commit details
    Browse the repository at this point in the history

  6. Fix test

    @dichiarafrancesco
    dichiarafrancesco committed May 9, 2018
    Configuration menu
    Copy the full SHA
    58fb829 View commit details
    Browse the repository at this point in the history

Commits on May 10, 2018

  1. Merge pull request spark-redshift-community#2 from Yelp/fdc_DATALAKE-… 

    …488_cleanup-fix-double-to-float

Fix double type to float and cleanup
    @dichiarafrancesco
    dichiarafrancesco authored May 10, 2018
    Configuration menu
    Copy the full SHA
    040b4a9 View commit details
    Browse the repository at this point in the history

  2. Merge pull request spark-redshift-community#3 from Yelp/fdc_DATALAKE-… 

    …486_avoid-log-creds

datalake-486 avoid log creds
    @dichiarafrancesco
    dichiarafrancesco authored May 10, 2018
    Configuration menu
    Copy the full SHA
    967dddb View commit details
    Browse the repository at this point in the history

Commits on May 11, 2018

  1. Fix Empty string is converted to null

    @dichiarafrancesco
    dichiarafrancesco committed May 11, 2018
    Configuration menu
    Copy the full SHA
    3ae6a9b View commit details
    Browse the repository at this point in the history

Commits on May 12, 2018

  1. Fix convertion bit and test

    @dichiarafrancesco
    dichiarafrancesco committed May 12, 2018
    Configuration menu
    Copy the full SHA
    475e7a1 View commit details
    Browse the repository at this point in the history

  2. Fix indentation

    @dichiarafrancesco
    dichiarafrancesco committed May 12, 2018
    Configuration menu
    Copy the full SHA
    d16317e View commit details
    Browse the repository at this point in the history

Commits on May 14, 2018

  1. Fix parenthesis

    @dichiarafrancesco
    dichiarafrancesco committed May 14, 2018
    Configuration menu
    Copy the full SHA
    e15ccb5 View commit details
    Browse the repository at this point in the history

  2. Fix scalastyle

    @dichiarafrancesco
    dichiarafrancesco committed May 14, 2018
    Configuration menu
    Copy the full SHA
    d06fe3b View commit details
    Browse the repository at this point in the history

  3. Fix File line length exceeds 100 characters

    @dichiarafrancesco
    dichiarafrancesco committed May 14, 2018
    Configuration menu
    Copy the full SHA
    689635c View commit details
    Browse the repository at this point in the history

  4. Merge pull request spark-redshift-community#4 from Yelp/fdc_DATALAKE-… 

    …4899_empty-string-to-null

Empty string is converted to null
    @dichiarafrancesco
    dichiarafrancesco authored May 14, 2018
    Configuration menu
    Copy the full SHA
    0d2a130 View commit details
    Browse the repository at this point in the history

  5. First Yelp release

    @dichiarafrancesco
    dichiarafrancesco committed May 14, 2018
    Configuration menu
    Copy the full SHA
    fbb58b3 View commit details
    Browse the repository at this point in the history

  6. Merge pull request spark-redshift-community#5 from Yelp/fdc_first-ver… 

    …sion

3.0.0 release
    @dichiarafrancesco
    dichiarafrancesco authored May 14, 2018
    Configuration menu
    Copy the full SHA
    50dfd98 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2019

  1. Fixed NewFilter - including hadoop-aws - s3n test is failing

    @lucagiovagnoli
    lucagiovagnoli committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    90581a8 View commit details
    Browse the repository at this point in the history

  2. Upgraded jackson by excluding it in aws

    @lucagiovagnoli
    lucagiovagnoli committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    834f0d6 View commit details
    Browse the repository at this point in the history

  3. force spark.avro - hadoop 2.7.7 and awsjavasdk downgraded

    @lucagiovagnoli
    lucagiovagnoli committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    ea5da29 View commit details
    Browse the repository at this point in the history

  4. Compiles with spark 2.4.0 - amazon unmarshal error

    @lucagiovagnoli
    lucagiovagnoli committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    0fe37d2 View commit details
    Browse the repository at this point in the history

  5. Compiling - managed to run tests but they mostly fail

    @lucagiovagnoli
    lucagiovagnoli committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    da10897 View commit details
    Browse the repository at this point in the history

  6. Removing conn.commit() everywhere - got 88% of integration tests to r… 

    …un - fix for STS token aws access in progress
    @lucagiovagnoli
    lucagiovagnoli committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    95cdf94 View commit details
    Browse the repository at this point in the history

  7. Ignoring a bunch of tests as did snowflake - close to have a green bu… 

    …ild to try out
    @lucagiovagnoli
    lucagiovagnoli committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    b1fa3f6 View commit details
    Browse the repository at this point in the history

  8. sbt assembly the package into a fat jar - found the perfect coordinat… 

    …ion between different libraries versions! Tests pass and can compile spark-on-paasta and spark successfullygit add src/ project/
    @lucagiovagnoli
    lucagiovagnoli committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    f3bbdb7 View commit details
    Browse the repository at this point in the history

  9. aws_variables.env gitignored

    @lucagiovagnoli
    lucagiovagnoli committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    0666bc6 View commit details
    Browse the repository at this point in the history

  10. remove in Memory FileSystem class and clean up comments in the sbt bu… 

    …ild file
    @lucagiovagnoli
    lucagiovagnoli committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    094cc15 View commit details
    Browse the repository at this point in the history

  11. Moving to external github issues - rename spName to spark-redshift-co… 

    …mmunity
    @lucagiovagnoli
    lucagiovagnoli committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    866d4fd View commit details
    Browse the repository at this point in the history

  12. Revert sbt scripts to an older version

    @lucagiovagnoli
    lucagiovagnoli committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    25acded View commit details
    Browse the repository at this point in the history

  13. Merge pull request spark-redshift-community#6 from spark-redshift-com… 

    …munity/luca-spark-2.4

spark-redshift on spark 2.4 [baseline - snapshot]
    @lucagiovagnoli
    lucagiovagnoli authored Jun 26, 2019
    Configuration menu
    Copy the full SHA
    5b0f949 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2019

  1. Moving package from com.databricks.spark.redshift to com.spark.redshi… 

    …ft.community and README and CHANGELOG
    @lucagiovagnoli
    lucagiovagnoli committed Jun 28, 2019
    Configuration menu
    Copy the full SHA
    7746c51 View commit details
    Browse the repository at this point in the history

  2. Better CHANGELOG - modernize SparkRedshiftBuild to build.sbt

    @lucagiovagnoli
    lucagiovagnoli committed Jun 28, 2019
    Configuration menu
    Copy the full SHA
    bef9893 View commit details
    Browse the repository at this point in the history

  3. Fix all broken databricks spark-redshift substitutions to the communi… 

    …ty package
    @lucagiovagnoli
    lucagiovagnoli committed Jun 28, 2019
    Configuration menu
    Copy the full SHA
    94449d0 View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2019

  1. Remove merge_pr utility - minor README update

    @lucagiovagnoli
    lucagiovagnoli committed Jul 1, 2019
    Configuration menu
    Copy the full SHA
    235468b View commit details
    Browse the repository at this point in the history

  2. Renaming package to com.spark_redshift_community.spark.redshift

    @lucagiovagnoli
    lucagiovagnoli committed Jul 1, 2019
    Configuration menu
    Copy the full SHA
    492b1ca View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2019

  1. Merge pull request spark-redshift-community#12 from spark-redshift-co… 

    …mmunity/luca-COREML-867-refactor-and-readme

    Moving package from com.databricks.spark.redshift to com.spark.redshift.community and README and CHANGELOG
    @lucagiovagnoli
    lucagiovagnoli authored Jul 2, 2019
    Configuration menu
    Copy the full SHA
    4a210a9 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2019

  1. Removing AWSCredentialsInUriIntegrationSuite test - credentials in th… 

    …e URI are a potentially unsafe practice
    @lucagiovagnoli
    lucagiovagnoli committed Jul 10, 2019
    Configuration menu
    Copy the full SHA
    5fd10e3 View commit details
    Browse the repository at this point in the history

  2. Update CHANGELOG and version

    @lucagiovagnoli
    lucagiovagnoli committed Jul 10, 2019
    Configuration menu
    Copy the full SHA
    b2dc8ff View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2019

  1. Merge pull request spark-redshift-community#15 from spark-redshift-co… 

    …mmunity/luca-COREML-889-AWSCredentialsInUriIntegrationSuite-failing-test

Remove AWSCredentialsInUriIntegrationSuite test
    @lucagiovagnoli
    lucagiovagnoli authored Jul 12, 2019
    Configuration menu
    Copy the full SHA
    9574e44 View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2019

  1. Move to previewDATE rather than SNAPSHOT releases before 4.0.0

    @lucagiovagnoli
    lucagiovagnoli committed Jul 16, 2019
    Configuration menu
    Copy the full SHA
    d2690f8 View commit details
    Browse the repository at this point in the history

  2. Refactor package and organization to be io.github.spark_redshift_comm… 

    …unity
    @lucagiovagnoli
    lucagiovagnoli committed Jul 16, 2019
    Configuration menu
    Copy the full SHA
    0b8f706 View commit details
    Browse the repository at this point in the history

  3. Merge pull request spark-redshift-community#18 from spark-redshift-co… 

    …mmunity/luca-move-to-preview-releases

Move to previewDATE rather than SNAPSHOT releases - rename to package io.github
    @lucagiovagnoli
    lucagiovagnoli authored Jul 16, 2019
    Configuration menu
    Copy the full SHA
    9fb479a View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2019

  1. Travis CI using hadoop 2.7.7, spark 2.4.3

    @lucagiovagnoli
    lucagiovagnoli committed Jul 23, 2019
    Configuration menu
    Copy the full SHA
    c53c055 View commit details
    Browse the repository at this point in the history

  2. Removing assembly plugin (unused) - add icons for build and coverage … 

    …status
    @lucagiovagnoli
    lucagiovagnoli committed Jul 23, 2019
    Configuration menu
    Copy the full SHA
    92d6d56 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2019

  1. Merge pull request spark-redshift-community#21 from spark-redshift-co… 

    …mmunity/luca-set-up-travis-ci

Travis CI using hadoop 2.7.7 and spark 2.4.3
    @lucagiovagnoli
    lucagiovagnoli authored Jul 24, 2019
    Configuration menu
    Copy the full SHA
    034fd5a View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2019

  1. Add a simple how to build from source tutorial

    Steven Moy authored Aug 2, 2019
    Configuration menu
    Copy the full SHA
    2515795 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2019

  1. Re-enable RedshiftSourceSuite using a new InMemoryS3AFileSystem

    @lucagiovagnoli
    lucagiovagnoli committed Aug 5, 2019
    Configuration menu
    Copy the full SHA
    e31778c View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2019

  1. Merge pull request spark-redshift-community#24 from lucagiovagnoli/lu… 

    …ca-re-enable-RedshiftSourceSuite

Re-enable RedshiftSourceSuite using a new InMemoryS3AFileSystem
    @lucagiovagnoli
    lucagiovagnoli authored Aug 6, 2019
    Configuration menu
    Copy the full SHA
    c0af333 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2019

  1. Address review comments on using repo's sbt 

    Address review comments
    Steven Moy authored Aug 7, 2019
    Configuration menu
    Copy the full SHA
    2c37142 View commit details
    Browse the repository at this point in the history

  2. Merge pull request spark-redshift-community#23 from smoy/how_to_build 

    Add a simple how to build from source tutorial
    @lucagiovagnoli
    lucagiovagnoli authored Aug 7, 2019
    Configuration menu
    Copy the full SHA
    1b6fbf7 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2019

  1. Modernize datetime parsing - use DateTimeFormatter from Java 8

    @lucagiovagnoli
    lucagiovagnoli committed Aug 14, 2019
    Configuration menu
    Copy the full SHA
    c1bec40 View commit details
    Browse the repository at this point in the history

  2. Fix line too long and update version

    @lucagiovagnoli
    lucagiovagnoli committed Aug 14, 2019
    Configuration menu
    Copy the full SHA
    5f83d87 View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2019

  1. Fix DateTimeFormatter String - timezone in Redshift in the form of +00

    @lucagiovagnoli
    lucagiovagnoli committed Aug 15, 2019
    Configuration menu
    Copy the full SHA
    5a3c56e View commit details
    Browse the repository at this point in the history

  2. Fix ZonedDateTime conversion to timestamp

    @lucagiovagnoli
    lucagiovagnoli committed Aug 15, 2019
    Configuration menu
    Copy the full SHA
    2987ba7 View commit details
    Browse the repository at this point in the history

  3. Fix timezone test to run correctly on machines in any timezone

    @lucagiovagnoli
    lucagiovagnoli committed Aug 15, 2019
    Configuration menu
    Copy the full SHA
    890be17 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2019

  1. Merge pull request spark-redshift-community#26 from lucagiovagnoli/lu… 

    …ca-COREML-952-parse-timestamptz

Modernize datetime parsing - use DateTimeFormatter from Java 8
    @lucagiovagnoli
    lucagiovagnoli authored Aug 16, 2019
    Configuration menu
    Copy the full SHA
    0c8ad55 View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2019

  1. Expected value in timstamptz test was wrong

    @lucagiovagnoli
    lucagiovagnoli committed Aug 21, 2019
    Configuration menu
    Copy the full SHA
    87cceaa View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2019

  1. Merge pull request spark-redshift-community#27 from lucagiovagnoli/lu… 

    …ca-COREML-timestamptz-fix-bug-in-ittests

Expected value in timestamptz test was wrong
    @lucagiovagnoli
    lucagiovagnoli authored Aug 22, 2019
    Configuration menu
    Copy the full SHA
    768130c View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2019

  1. Change groupId to use hyphens

    @lucagiovagnoli
    lucagiovagnoli committed Sep 4, 2019
    Configuration menu
    Copy the full SHA
    c199b2e View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2019

  1. successfully publishLocalSigned

    @lucagiovagnoli
    lucagiovagnoli committed Sep 10, 2019
    Configuration menu
    Copy the full SHA
    4359ec0 View commit details
    Browse the repository at this point in the history

  2. correct checkThatBucketHasObjectLifecycleConfiguration when s3 lifecy… 

    …cle has no prefix
    @julienbachmann
    julienbachmann committed Sep 10, 2019
    Configuration menu
    Copy the full SHA
    fcb3641 View commit details
    Browse the repository at this point in the history

  3. Merge pull request spark-redshift-community#1 from nagra-insight/fix/… 

    …checkThatBucketHasObjectLifecycleConfiguration

correct checkThatBucketHasObjectLifecycleConfiguration
    @julienbachmann
    julienbachmann authored Sep 10, 2019
    Configuration menu
    Copy the full SHA
    5c2a400 View commit details
    Browse the repository at this point in the history

  4. Add changelog and snapshot version to test publishing to Maven Central

    @lucagiovagnoli
    lucagiovagnoli committed Sep 10, 2019
    Configuration menu
    Copy the full SHA
    8311244 View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2019

  1. remove extra space

    @lucagiovagnoli
    lucagiovagnoli committed Sep 16, 2019
    Configuration menu
    Copy the full SHA
    aec0fdf View commit details
    Browse the repository at this point in the history

  2. Merge pull request spark-redshift-community#28 from lucagiovagnoli/lu… 

    …ca-issue19-publish-maven-central

Prepare for Maven Publishing
    @lucagiovagnoli
    lucagiovagnoli authored Sep 16, 2019
    Configuration menu
    Copy the full SHA
    147c843 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2019

  1. Stable 4.0.0 release to publish to maven central

    @lucagiovagnoli
    lucagiovagnoli committed Sep 17, 2019
    Configuration menu
    Copy the full SHA
    58ec6f6 View commit details
    Browse the repository at this point in the history

  2. Merge pull request spark-redshift-community#29 from lucagiovagnoli/lu… 

    …ca-publish-maven-central-full-release

Stable 4.0.0 release to publish to maven central
    @lucagiovagnoli
    lucagiovagnoli authored Sep 17, 2019
    Configuration menu
    Copy the full SHA
    4df21f0 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2019

  1. test all 4 branches in the `checkThatBucketHasObjectLifecycleConfigur… 

    …ation` method
    @julienbachmann
    julienbachmann committed Sep 18, 2019
    Configuration menu
    Copy the full SHA
    bd350e2 View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2019

  1. correct missing dot

    @julienbachmann
    julienbachmann committed Sep 22, 2019
    Configuration menu
    Copy the full SHA
    9ebb338 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2019

  1. Fixed group ids as per issue spark-redshift-community#33 

    spark-redshift-community#33
    @grmontpetit
    grmontpetit committed Oct 24, 2019
    Configuration menu
    Copy the full SHA
    c354a8f View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2019

  1. Addressed PR's comments.

    @grmontpetit
    grmontpetit committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    da4b715 View commit details
    Browse the repository at this point in the history

  2. Merge pull request spark-redshift-community#38 from sniggel/fix/group… 

    …-ids

Fixed group ids as per issue spark-redshift-community#33
    @lucagiovagnoli
    lucagiovagnoli authored Oct 25, 2019
    Configuration menu
    Copy the full SHA
    ae6280c View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2019

  1. README improvements 

    - add installation instructions for use without build tool
- clean up installation instructions for SBT and Maven
- remove installation instructions for Databricks
- TOC improvements
    @eeshugerman
    eeshugerman authored and Elliott Shugerman committed Nov 6, 2019
    Configuration menu
    Copy the full SHA
    51271a0 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2019

  1. Merge pull request spark-redshift-community#49 from eeshugerman/master 

    README improvements
    @lucagiovagnoli
    lucagiovagnoli authored Nov 8, 2019
    Configuration menu
    Copy the full SHA
    ccab1aa View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2019

  1. Handle microseconds from redshift

    @lucagiovagnoli
    lucagiovagnoli committed Nov 14, 2019
    Configuration menu
    Copy the full SHA
    a26007b View commit details
    Browse the repository at this point in the history

  2. Bump version and changelog

    @lucagiovagnoli
    lucagiovagnoli committed Nov 14, 2019
    Configuration menu
    Copy the full SHA
    1ed1fc1 View commit details
    Browse the repository at this point in the history

  3. Handle 4 and 5 digits after the comma

    @lucagiovagnoli
    lucagiovagnoli committed Nov 14, 2019
    Configuration menu
    Copy the full SHA
    dec70a6 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2019

  1. Merge pull request spark-redshift-community#51 from lucagiovagnoli/lu… 

    …ca-handle-microseconds

Handle microseconds from redshift
    @lucagiovagnoli
    lucagiovagnoli authored Nov 21, 2019
    Configuration menu
    Copy the full SHA
    e9d4653 View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2019

  1. ISSUE-56 | Trimming preactions and postactions before splitting to av… 

    …oid executing empty sql
    @chandanatalef
    chandanatalef committed Dec 7, 2019
    Configuration menu
    Copy the full SHA
    657c2e8 View commit details
    Browse the repository at this point in the history

  2. ISSUE-56 | Reformatting

    @meetchandan
    meetchandan committed Dec 7, 2019
    Configuration menu
    Copy the full SHA
    1c1b421 View commit details
    Browse the repository at this point in the history

  3. ISSUE-56 | Added tests

    @meetchandan
    meetchandan committed Dec 7, 2019
    Configuration menu
    Copy the full SHA
    1d38a5f View commit details
    Browse the repository at this point in the history

  4. ISSUE-56 | fixing tests

    @meetchandan
    meetchandan committed Dec 7, 2019
    Configuration menu
    Copy the full SHA
    c9b6d9c View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2019

  1. Merge pull request spark-redshift-community#57 from meetchandan/ISSUE-56 

    ISSUE-56 | Trimming preactions and postactions before splitting to av…
    @lucagiovagnoli
    lucagiovagnoli authored Dec 13, 2019
    Configuration menu
    Copy the full SHA
    78465a8 View commit details
    Browse the repository at this point in the history

  2. Bump version and changelog to 4.0.2 - bug fix sql text trimming

    @lucagiovagnoli
    lucagiovagnoli committed Dec 13, 2019
    Configuration menu
    Copy the full SHA
    c28e985 View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2019

  1. add 'include_column_list' parameter 

    Co-authored-by: kyrozetera <[email protected]>
    @kyrozetera
    Elliott Shugerman and kyrozetera committed Dec 14, 2019
    Configuration menu
    Copy the full SHA
    849dd82 View commit details
    Browse the repository at this point in the history

  2. fix scalastyle check

    Elliott Shugerman committed Dec 14, 2019
    Configuration menu
    Copy the full SHA
    edebea4 View commit details
    Browse the repository at this point in the history

  3. code review: don't use null for schema in test 

    also some minor formatting tweaks for the same test
    Elliott Shugerman committed Dec 14, 2019
    Configuration menu
    Copy the full SHA
    bb252e3 View commit details
    Browse the repository at this point in the history

  4. code review: add test for include_column_list=false

    Elliott Shugerman committed Dec 14, 2019
    Configuration menu
    Copy the full SHA
    5d8f9d2 View commit details
    Browse the repository at this point in the history

  5. code review: changelog, bump version

    Elliott Shugerman committed Dec 14, 2019
    Configuration menu
    Copy the full SHA
    bc113fa View commit details
    Browse the repository at this point in the history

Commits on Dec 16, 2019

  1. Merge pull request spark-redshift-community#58 from eeshugerman/colum… 

    …n-list

Add 'include_column_list' parameter
    @lucagiovagnoli
    lucagiovagnoli authored Dec 16, 2019
    Configuration menu
    Copy the full SHA
    d4396f8 View commit details
    Browse the repository at this point in the history

  2. Update "How to help" section 

    @talalryz or @lydian  for review
    @lucagiovagnoli
    lucagiovagnoli authored Dec 16, 2019
    Configuration menu
    Copy the full SHA
    9dd950b View commit details
    Browse the repository at this point in the history

Commits on Dec 17, 2019

  1. Merge pull request spark-redshift-community#59 from spark-redshift-co… 

    …mmunity/lucagiovagnoli-patch-1

Update "How to help" section
    @lucagiovagnoli
    lucagiovagnoli authored Dec 17, 2019
    Configuration menu
    Copy the full SHA
    21e3ec8 View commit details
    Browse the repository at this point in the history

Commits on Feb 25, 2020

  1. Update README.md 

    Fixes spark-redshift-community#67  and spark-redshift-community#39
    @lucagiovagnoli
    lucagiovagnoli authored Feb 25, 2020
    Configuration menu
    Copy the full SHA
    9af6b6c View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2020

  1. Merge pull request spark-redshift-community#68 from spark-redshift-co… 

    …mmunity/lucagiovagnoli-README-remove-jitpack

Remove jitpack from README.md
    @lucagiovagnoli
    lucagiovagnoli authored Feb 26, 2020
    Configuration menu
    Copy the full SHA
    84ebe9d View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2020

  1. fix

    @karuppayya
    karuppayya committed Apr 1, 2020
    Configuration menu
    Copy the full SHA
    42fa95e View commit details
    Browse the repository at this point in the history

Commits on Apr 17, 2020

  1. Refactor

    @karuppayya
    karuppayya committed Apr 17, 2020
    Configuration menu
    Copy the full SHA
    2e82f97 View commit details
    Browse the repository at this point in the history

  2. Fix build.sbt

    @karuppayya
    karuppayya committed Apr 17, 2020
    Configuration menu
    Copy the full SHA
    d0c5279 View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2020

  1. Refactoring, fixes

    @karuppayya
    karuppayya committed Apr 23, 2020
    Configuration menu
    Copy the full SHA
    6283202 View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2020

  1. cross publish to 2.12

    @joprice
    joprice committed Jun 15, 2020
    Configuration menu
    Copy the full SHA
    07da20b View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2020

  1. Fix compilation

    @karuppayya
    karuppayya committed Jul 9, 2020
    Configuration menu
    Copy the full SHA
    da88eb7 View commit details
    Browse the repository at this point in the history

  2. Rename class name

    @karuppayya
    karuppayya committed Jul 9, 2020
    Configuration menu
    Copy the full SHA
    55b63db View commit details
    Browse the repository at this point in the history

  3. Fix checkstyle

    @karuppayya
    karuppayya committed Jul 9, 2020
    Configuration menu
    Copy the full SHA
    c89ce05 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2020

  1. Temporarily unblock build

    @karuppayya
    karuppayya committed Jul 10, 2020
    Configuration menu
    Copy the full SHA
    633e1e4 View commit details
    Browse the repository at this point in the history

  2. Fix spark version

    @karuppayya
    karuppayya committed Jul 10, 2020
    Configuration menu
    Copy the full SHA
    45b3e1b View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2020

  1. Update README.md 

    removing the version number from the sbt instructions README.md since it seems to not resolve correctly.
    @wseaton
    wseaton authored Jul 14, 2020
    Configuration menu
    Copy the full SHA
    888b650 View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2020

  1. Merge pull request spark-redshift-community#73 from wseaton/patch-1 

    Update README.md
    @lucagiovagnoli
    lucagiovagnoli authored Sep 15, 2020
    Configuration menu
    Copy the full SHA
    e9e6a65 View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2020

  1. Merge pull request spark-redshift-community#72 from joprice/crossPublish 

    cross publish to 2.12
    @lucagiovagnoli
    lucagiovagnoli authored Sep 16, 2020
    Configuration menu
    Copy the full SHA
    e8b354a View commit details
    Browse the repository at this point in the history

  2. Bump to 4.1.1 to cross publish for scala 2.12

    @lucagiovagnoli
    lucagiovagnoli committed Sep 16, 2020
    Configuration menu
    Copy the full SHA
    d39ef0b View commit details
    Browse the repository at this point in the history

  3. Merge pull request spark-redshift-community#75 from spark-redshift-co… 

    …mmunity/luca-bump-to-4.1.1

Bump to 4.1.1 to cross publish for scala 2.12
    @lucagiovagnoli
    lucagiovagnoli authored Sep 16, 2020
    Configuration menu
    Copy the full SHA
    674d6d9 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2020

  1. spark-3.0.1 compatible

    vanhoale committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    d6f2281 View commit details
    Browse the repository at this point in the history

  2. spark-3.0.1 compatible - updated CI

    vanhoale committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    020051a View commit details
    Browse the repository at this point in the history

  3. spark-3.0.1 compatible - updated test suites

    vanhoale committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    9f68173 View commit details
    Browse the repository at this point in the history

  4. used enableHiveSupport for sqlContext

    vanhoale committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    f34026b View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2020

  1. address review comment

    Hoa Le authored Oct 6, 2020
    Configuration menu
    Copy the full SHA
    7dbf620 View commit details
    Browse the repository at this point in the history

  2. keep CI testing spark 2.x

    Hoa Le authored Oct 6, 2020
    Configuration menu
    Copy the full SHA
    ada001a View commit details
    Browse the repository at this point in the history

  3. we can't test the same code with 2 versions of sparks

    Hoa Le authored Oct 6, 2020
    Configuration menu
    Copy the full SHA
    8f5f700 View commit details
    Browse the repository at this point in the history

  4. bump minor version per review

    Hoa Le authored Oct 6, 2020
    Configuration menu
    Copy the full SHA
    636f8aa View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2020

  1. removed trailing space

    Hoa Le authored Oct 7, 2020
    Configuration menu
    Copy the full SHA
    6d3d3fe View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2020

  1. updated version

    Hoa Le authored Oct 8, 2020
    Configuration menu
    Copy the full SHA
    8e34bbf View commit details
    Browse the repository at this point in the history

  2. Merge pull request spark-redshift-community#77 from vanhoale/spark3.x… 

    …-compatible

spark-3.0.1 compatible
    @lucagiovagnoli
    lucagiovagnoli authored Oct 8, 2020
    Configuration menu
    Copy the full SHA
    cd1aea7 View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2020

  1. Merge branch 'master' of https://github.com/spark-redshift-community/… 

    …spark-redshift
    @julienbachmann
    julienbachmann committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    de59930 View commit details
    Browse the repository at this point in the history

  2. do not display stacktrace on error on getBucketLifecycle

    @julienbachmann
    julienbachmann committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    8230764 View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2020

  1. Fix compilation errors

    @88manpreet
    88manpreet committed Dec 14, 2020
    Configuration menu
    Copy the full SHA
    5288e24 View commit details
    Browse the repository at this point in the history

Commits on Dec 18, 2020

  1. Ignore remaining 6 failing unit tests and fix dependencies

    @88manpreet
    88manpreet committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    0b232cf View commit details
    Browse the repository at this point in the history

  2. Fix hadoop version for travis

    @88manpreet
    88manpreet committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    bda5ec1 View commit details
    Browse the repository at this point in the history

Commits on Jan 7, 2021

  1. Improve readability

    @88manpreet
    88manpreet committed Jan 7, 2021
    Configuration menu
    Copy the full SHA
    4349270 View commit details
    Browse the repository at this point in the history

Commits on Jan 13, 2021

  1. Bump version for hadoop 3 support in spark-redshift

    @88manpreet
    88manpreet committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    3ed0e5c View commit details
    Browse the repository at this point in the history

  2. Bump version to 5.0.0 Hadoop 3 support

    @88manpreet
    88manpreet committed Jan 13, 2021
    Configuration menu
    Copy the full SHA
    a9e2604 View commit details
    Browse the repository at this point in the history

  3. Merge pull request spark-redshift-community#79 from spark-redshift-co… 

    …mmunity/hadoop3_2_1_support

Hadoop 3.2.11 support for spark-redshift
    @88manpreet
    88manpreet authored Jan 13, 2021
    Configuration menu
    Copy the full SHA
    453e3d0 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2021

  1. Creating a serializer per row has a really low performance. 

    Instead of creating a new serializer per row, it is better to create only one per schema.
It is thread-safe because this object does not escape from the thread associated with some Spark task.
    @gumartinm
    gumartinm committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    cf58172 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2021

  1. Merge pull request spark-redshift-community#88 from gumartinm/spark3_… 

    …low_performance

Issue spark-redshift-community#87: Creating a serializer per row has a really low performance.
    @88manpreet
    88manpreet authored Apr 24, 2021
    Configuration menu
    Copy the full SHA
    1c24b8c View commit details
    Browse the repository at this point in the history

Commits on May 1, 2021

  1. Bump version to 5.0.1

    @88manpreet
    88manpreet committed May 1, 2021
    Configuration menu
    Copy the full SHA
    7726c10 View commit details
    Browse the repository at this point in the history

Commits on May 5, 2021

  1. add sse kms support

    munk committed May 5, 2021
    Configuration menu
    Copy the full SHA
    11494b1 View commit details
    Browse the repository at this point in the history

Commits on May 6, 2021

  1. Put comment on two lines to appease the linter

    munk committed May 6, 2021
    Configuration menu
    Copy the full SHA
    20827fb View commit details
    Browse the repository at this point in the history

  2. whitespace fix

    munk committed May 6, 2021
    Configuration menu
    Copy the full SHA
    160c82d View commit details
    Browse the repository at this point in the history

  3. Merge pull request spark-redshift-community#89 from munk/jd/add-sse-k… 

    …ms-support

Add sse kms support
    @88manpreet
    88manpreet authored May 6, 2021
    Configuration menu
    Copy the full SHA
    dfc2b8e View commit details
    Browse the repository at this point in the history

  4. Bump version to v5.0.2

    @88manpreet
    88manpreet committed May 6, 2021
    Configuration menu
    Copy the full SHA
    c9999a9 View commit details
    Browse the repository at this point in the history

Commits on May 7, 2021

  1. Remove reliance on sbt-spark-package.

    @jsleight
    jsleight committed May 7, 2021
    Configuration menu
    Copy the full SHA
    59234aa View commit details
    Browse the repository at this point in the history

  2. Merge pull request spark-redshift-community#90 from spark-redshift-co… 

    …mmunity/jsleight_remove_sbt_spark_package

Remove reliance on sbt-spark-package
    @jsleight
    jsleight authored May 7, 2021
    Configuration menu
    Copy the full SHA
    3cb8667 View commit details
    Browse the repository at this point in the history

Commits on May 10, 2021

  1. Bump version to v5.0.3

    @88manpreet
    88manpreet committed May 10, 2021
    Configuration menu
    Copy the full SHA
    02b2ff5 View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2021

  1. Update Spark version and aws java sdk version for consistency purposes

    @88manpreet
    88manpreet committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    bc040d5 View commit details
    Browse the repository at this point in the history

  2. Bump version to 5.0.4

    @88manpreet
    88manpreet committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    cb39659 View commit details
    Browse the repository at this point in the history

  3. Merge pull request spark-redshift-community#92 from spark-redshift-co… 

    …mmunity/d/manpreet/upgrade_spark_aws_java_sdk_version

Update Spark version and aws java sdk version for consistency
    @88manpreet
    88manpreet authored Jul 9, 2021
    Configuration menu
    Copy the full SHA
    846cc23 View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2021

  1. Merge pull request spark-redshift-community#30 from nagra-insight/master 

    Correct wrong warning when tmp bucket lifecycle policy has no prefix
    @88manpreet
    88manpreet authored Nov 9, 2021
    Configuration menu
    Copy the full SHA
    daf2ecf View commit details
    Browse the repository at this point in the history

  2. Bump version to 5.0.5 to avoid warning when bucket if configured for … 

    …lifecycle without prefix
    @88manpreet
    88manpreet committed Nov 9, 2021
    Configuration menu
    Copy the full SHA
    bfe823b View commit details
    Browse the repository at this point in the history

  3. Merge pull request spark-redshift-community#97 from 88manpreet/d/manp… 

    …reet/bump_version_bucket_lifecycle_policy_fix_bump

Bump version to 5.0.5 to avoid warning when bucket if configured
    @88manpreet
    88manpreet authored Nov 9, 2021
    Configuration menu
    Copy the full SHA
    557783f View commit details
    Browse the repository at this point in the history

Commits on Jan 13, 2022

  1. fix: log4j-api not compatibility with spark 3.2.0 

    upgrade slf4j-api to 1.7.32 can solve problem
    Nguyen Van Long committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    dfa47dc View commit details
    Browse the repository at this point in the history

  2. Merge pull request spark-redshift-community#100 from nvlong198/fix/lo… 

    …g4jcompatibility

fix: log4j-api not compatibility with spark 3.2.0
    @jsleight
    jsleight authored Jan 13, 2022
    Configuration menu
    Copy the full SHA
    1ede80a View commit details
    Browse the repository at this point in the history

Commits on Jan 27, 2022

  1. Upgrade to spark v3.2.0

    @jsleight
    jsleight committed Jan 27, 2022
    Configuration menu
    Copy the full SHA
    ab96178 View commit details
    Browse the repository at this point in the history

  2. Merge pull request spark-redshift-community#101 from jsleight/u/jslei… 

    …ght/spark3.2

Upgrade to spark v3.2.0
    @jsleight
    jsleight authored Jan 27, 2022
    Configuration menu
    Copy the full SHA
    1f12470 View commit details
    Browse the repository at this point in the history

Commits on Mar 29, 2022

  1. Add catalyst type mapping for LONGVARCHAR

    Matthew Jones committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    b8eb032 View commit details
    Browse the repository at this point in the history

Commits on Mar 30, 2022

  1. Merge pull request spark-redshift-community#105 from matthewrj/suppor… 

    …t-super

Support reading from SUPER columns
    @jsleight
    jsleight authored Mar 30, 2022
    Configuration menu
    Copy the full SHA
    a675a3c View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2022

  1. make manifest file path use s3a/n scheme

    @nanflasted
    nanflasted committed Sep 9, 2022
    Configuration menu
    Copy the full SHA
    f0d98f6 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2022

  1. Merge pull request spark-redshift-community#110 from nanflasted/ych_t… 

    …ries_fixing_manifest_s3a

make manifest file path use s3a/n scheme
    @jsleight
    jsleight authored Sep 21, 2022
    Configuration menu
    Copy the full SHA
    a51b440 View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2022

  1. Bump version to 5.1.0 to support Spark 3.2

    @88manpreet
    88manpreet committed Sep 22, 2022
    Configuration menu
    Copy the full SHA
    84870fa View commit details
    Browse the repository at this point in the history

  2. Merge pull request spark-redshift-community#111 from spark-redshift-c… 

    …ommunity/d/manpreet/bump_version_to_support_spark_3.2

Bump version to 5.1.0 to support Spark 3.2
    @88manpreet
    88manpreet authored Sep 22, 2022
    Configuration menu
    Copy the full SHA
    78792a9 View commit details
    Browse the repository at this point in the history

Commits on Dec 23, 2022

  1. Fix build

    @parisni
    parisni committed Dec 23, 2022
    Configuration menu
    Copy the full SHA
    88948bd View commit details
    Browse the repository at this point in the history

Commits on Dec 29, 2022

  1. Merge remote-tracking branch 'karu/redshift_dsv2' into dsv2 

    # Conflicts:
#	.travis.yml
#	build.sbt
#	src/it/scala/io/github/spark_redshift_community/spark/redshift/IntegrationSuiteBase.scala
#	src/main/scala/io/github/spark_redshift_community/spark/redshift/Conversions.scala
#	src/main/scala/io/github/spark_redshift_community/spark/redshift/Parameters.scala
#	src/main/scala/io/github/spark_redshift_community/spark/redshift/RedshiftRelation.scala
#	src/main/scala/io/github/spark_redshift_community/spark/redshift/RedshiftWriter.scala
#	src/test/scala/io/github/spark_redshift_community/spark/redshift/RedshiftSourceSuite.scala
    @parisni
    parisni committed Dec 29, 2022
    Configuration menu
    Copy the full SHA
    3479ac6 View commit details
    Browse the repository at this point in the history

  2. Fix dsv2 for spark3.2

    @parisni
    parisni committed Dec 29, 2022
    Configuration menu
    Copy the full SHA
    69b1445 View commit details
    Browse the repository at this point in the history

Commits on Dec 30, 2022

  1. Make sbt build on local 

    Merge branch 'lbc/build-catalog' into parisni-dsv2
    @parisni
    parisni committed Dec 30, 2022
    Configuration menu
    Copy the full SHA
    eca6ea8 View commit details
    Browse the repository at this point in the history

  2. Add redshift catalog

    @parisni
    parisni committed Dec 30, 2022
    Configuration menu
    Copy the full SHA
    a72bcd1 View commit details
    Browse the repository at this point in the history

  3. Implement pushdown agg

    @parisni
    parisni committed Dec 30, 2022
    Configuration menu
    Copy the full SHA
    7381aea View commit details
    Browse the repository at this point in the history

  4. Revert "Implement pushdown agg" 

    apparently this does not work. More importantly, agg pushdown put the
load on redshift, and this might not be a good idea.
    @parisni
    parisni committed Dec 30, 2022
    Configuration menu
    Copy the full SHA
    cf64c1d View commit details
    Browse the repository at this point in the history

  5. Handle redshift empty parquet files

    @parisni
    parisni committed Dec 30, 2022
    Configuration menu
    Copy the full SHA
    ee6ea20 View commit details
    Browse the repository at this point in the history

Commits on Jan 1, 2023

  1. Implement table ttl in minutes

    @parisni
    parisni committed Jan 1, 2023
    Configuration menu
    Copy the full SHA
    8fad123 View commit details
    Browse the repository at this point in the history

Commits on Jan 2, 2023

  1. Implement catalog write support

    @parisni
    parisni committed Jan 2, 2023
    Configuration menu
    Copy the full SHA
    b133fbb View commit details
    Browse the repository at this point in the history

  2. Implement copy in parquet format

    @parisni
    parisni committed Jan 2, 2023
    Configuration menu
    Copy the full SHA
    511a209 View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2023

  1. Document features

    @parisni
    parisni committed Jan 3, 2023
    Configuration menu
    Copy the full SHA
    89e436f View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2023

  1. Support drop, alter, rename table and database

    @parisni
    parisni committed Jan 4, 2023
    Configuration menu
    Copy the full SHA
    885f7b3 View commit details
    Browse the repository at this point in the history

  2. Support refresh table to invalidate cache

    @parisni
    parisni committed Jan 4, 2023
    Configuration menu
    Copy the full SHA
    6d17ad1 View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2023

  1. Support desc redshift comments

    @parisni
    parisni committed Jan 12, 2023
    Configuration menu
    Copy the full SHA
    e04356f View commit details
    Browse the repository at this point in the history

  2. Speedup redshift schema discovery

    @parisni
    parisni committed Jan 12, 2023
    Configuration menu
    Copy the full SHA
    869c49e View commit details
    Browse the repository at this point in the history

Commits on Jan 21, 2023

  1. Use cache manifest 

    This avoid s3 file listings to find the last cache candidate
Also use hadoop FS instead of aws low level client
    @parisni
    parisni committed Jan 21, 2023
    Configuration menu
    Copy the full SHA
    48dbdd3 View commit details
    Browse the repository at this point in the history