Code

Merge develop into cascading3#1762

Code

Merged

johnynek merged 104 commits intotwitter:cascading3twitter/scalding:cascading3from piyushnarang:cascading3piyushnarang/scalding:cascading3Copy head branch name to clipboardJan 25, 2018

+11,671-2,904

Commits

Commits on May 14, 2015

• Prepare for release of 0.14.0

  

  Joe Nievelt

  committed
  e90edc1

  Copy full SHA for e90edc1

Commits on May 15, 2015

• Merge branch 'release/0.14.0'

  

  Joe Nievelt

  committed
  ca905a2

  Copy full SHA for ca905a2

Commits on May 21, 2015

• Merge branch 'release/0.15.0'

  

  Katya Gonina

  committed
  afbde5a

  Copy full SHA for afbde5a

• Merge branch 'master' of github.com:twitter/scalding

  Show description for ece459a
  

  Katya Gonina

  committed
  ece459a

  Copy full SHA for ece459a

Commits on Feb 4, 2016

• Merge branch 'master' of github.com:twitter/scalding

  oscar-stripe

  committed
  0f9bdc9

  Copy full SHA for 0f9bdc9

• Merge branch 'oscar/0.16.0-RC2'

  oscar-stripe

  committed
  cba13d1

  Copy full SHA for cba13d1

Commits on Nov 15, 2016

• Moving the repl in wonderland to a dedicated md file (twitter#1614)

  Show description for c77b4de
  

  Piyush Narang

  authored and

  johnynek

  committed
  c77b4de

  Copy full SHA for c77b4de

Commits on Nov 17, 2016

• Revert FileSource support for empty directories

  isnotinvain

  committed
  462746e

  Copy full SHA for 462746e

• Clean up temporary files created by forceToDiskExecution (twitter#1621)

  Show description for d389c63
  

  ajohnson-stripe

  authored and

  johnynek

  committed
  d389c63

  Copy full SHA for d389c63

• Merge pull request twitter#1622 from isnotinvain/alexlevenson/revert-empty-dir

  Show description for 3c72dd7
  isnotinvain

  authored
  3c72dd7

  Copy full SHA for 3c72dd7

Commits on Nov 19, 2016

• Generate Scalding microsite via sbt-microsites (twitter#1623)

  sritchie

  authored
  1d42ada

  Copy full SHA for 1d42ada

Commits on Dec 5, 2016

• TypedText - Added versions of dailyPrefixSuffix for TSV and CSV sources. (twitter#1627)

  Show description for 845c33f
  

  cavorite

  authored and

  johnynek

  committed
  845c33f

  Copy full SHA for 845c33f

Commits on Dec 10, 2016

• Upgrade build to sbt 0.13.13 (twitter#1629)

  Show description for 6c199c6
  

  sritchie

  authored and

  johnynek

  committed
  6c199c6

  Copy full SHA for 6c199c6

• Allow supplying better error messages in InvaidSourceTap/Exception (twitter#1626)

  Show description for 712418e
  

  isnotinvain

  authored and

  johnynek

  committed
  712418e

  Copy full SHA for 712418e

Commits on Jan 13, 2017

• Fixes a bug in how we use this stack (twitter#1632)

  

  ianoc

  authored and

  johnynek

  committed
  aa143bb

  Copy full SHA for aa143bb

Commits on Jan 18, 2017

• Bump chill version to 0.8.3 (twitter#1634)

  Show description for b0ed1be
  ttim

  authored
  b0ed1be

  Copy full SHA for b0ed1be

Commits on Feb 6, 2017

• Added Execution API for work with distributed cache files

  dieu

  committed
  07d8b3c

  Copy full SHA for 07d8b3c

• Merge pull request twitter#1635 from twitter/apanasenko/distributed_cache_execution

  Show description for 72ef68b
  dieu

  authored
  72ef68b

  Copy full SHA for 72ef68b

Commits on Feb 13, 2017

• Pick up Algebird 0.13.0

  

  Piyush Narang

  committed
  bbc5bd6

  Copy full SHA for bbc5bd6

• Fix TypedPipeDiffTest compile error

  

  Piyush Narang

  committed
  5e50a5e

  Copy full SHA for 5e50a5e

Commits on Feb 14, 2017

• Merge pull request twitter#1640 from piyushnarang/bump-algebird

  Show description for 6f11ced
  

  Piyush Narang

  authored
  6f11ced

  Copy full SHA for 6f11ced

Commits on Feb 23, 2017

• Use JobConf input files list for input size computation used by ReducersEstimators

  dieu

  committed
  1a42b5a

  Copy full SHA for 1a42b5a

• Merge pull request twitter#1644 from dieu/apanasenko/reduce_estimator_vkvs

  Show description for 34cc46c
  dieu

  authored
  34cc46c

  Copy full SHA for 34cc46c

Commits on Feb 24, 2017

• Using MAP_OUTPUT_BYTES instead of FILE_BYTES_READ in RatioBasedEstimator

  dieu

  committed
  6e3ad96

  Copy full SHA for 6e3ad96

• Merge pull request twitter#1645 from dieu/apanasenko/1510

  Show description for 946e1bd
  dieu

  authored
  946e1bd

  Copy full SHA for 946e1bd

Commits on Feb 28, 2017

• Use Batched in Sketch production (twitter#1648)

  johnynek

  authored
  8bebb57

  Copy full SHA for 8bebb57

Commits on Mar 2, 2017

• Fixed reduce estimator for paths with a glob pattern

  dieu

  committed
  b892acf

  Copy full SHA for b892acf

• Merge pull request twitter#1650 from dieu/apanasenko/fix_reduce_estimator

  Show description for 98cf320
  dieu

  authored
  98cf320

  Copy full SHA for 98cf320

Commits on Mar 8, 2017

• Expand libjars globs to match the behavior of Tool.

  jbalogh

  authored
  f780fd8

  Copy full SHA for f780fd8

Commits on Mar 9, 2017

• Merge pull request twitter#1651 from jbalogh/repl-expand-libjars

  Show description for 0b07c4b
  ianoc

  authored
  0b07c4b

  Copy full SHA for 0b07c4b

Commits on Mar 22, 2017

• Improve Vertica support for scalding-db

  alambert

  committed
  793d051

  Copy full SHA for 793d051

• reset version

  alambert

  committed
  9731e0e

  Copy full SHA for 9731e0e

• remove unnecessary condition

  alambert

  committed
  96e9ff7

  Copy full SHA for 96e9ff7

Commits on Mar 23, 2017

• Scalding updates for Scala 2.12 (twitter#1646)

  Show description for 12aaa7e
  

  Piyush Narang

  authored
  12aaa7e

  Copy full SHA for 12aaa7e

Commits on Mar 24, 2017

• Add andThen to Mappable (twitter#1656)

  Show description for 968f381
  

  lclementi

  authored and

  johnynek

  committed
  968f381

  Copy full SHA for 968f381

• Merge pull request twitter#1655 from alambert/alambert/vertica

  Show description for 708b5e7
  benpence

  authored
  708b5e7

  Copy full SHA for 708b5e7

Commits on Mar 27, 2017

• Estimate input size only for Hfs and GlobHfs taps

  dieu

  committed
  f53b5c1

  Copy full SHA for f53b5c1

Commits on Mar 28, 2017

• Use CombinedWritableSequenceFile for VKVS and make separate Combined*SequenceFileScheme

  dieu

  committed
  d7c2708

  Copy full SHA for d7c2708

• Merge pull request twitter#1647 from dieu/apanasenko/vkvs_to_combine_input_seq_file

  Show description for 115fd1a
  dieu

  authored
  115fd1a

  Copy full SHA for 115fd1a

• Merge pull request twitter#1652 from dieu/apanasenko/fix_for_reducer_estimator

  Show description for 9cc390d
  dieu

  authored
  9cc390d

  Copy full SHA for 9cc390d

Commits on Apr 5, 2017

• add explicit return types to implicit methods and values (twitter#1660)

  

  fwbrasil

  authored and

  johnynek

  committed
  c5f3922

  Copy full SHA for c5f3922

• Simplify match statement and use collection.breakOut (twitter#1661)

  Show description for d46948f
  

  zaneli

  authored and

  johnynek

  committed
  d46948f

  Copy full SHA for d46948f

Commits on Apr 7, 2017

• Use reflection over Jobs to find serialized classes (twitter#1654)

  Show description for 68519c1
  

  jcdavis

  authored and

  johnynek

  committed
  68519c1

  Copy full SHA for 68519c1

• Make a new method private (twitter#1662)

  johnynek

  authored
  5d5dd54

  Copy full SHA for 5d5dd54

Commits on Apr 11, 2017

• Skip scala doc in 2.12 for java classes (twitter#1663)

  

  Piyush Narang

  authored
  73c2e69

  Copy full SHA for 73c2e69

Commits on Apr 12, 2017

• Request for Scalding release 0.17.0 (twitter#1641)

  Show description for b1b39dc
  

  Piyush Narang

  authored and

  johnynek

  committed
  b1b39dc

  Copy full SHA for b1b39dc

• Adds Execution tutorial

  benpence

  committed
  91e8e4d

  Copy full SHA for 91e8e4d

Commits on Apr 13, 2017

• Addresses some comments

  benpence

  committed
  6a010e7

  Copy full SHA for 6a010e7

Commits on Apr 14, 2017

• Estimator memory of container for mapper/reducer.

  

  Piyush Narang

  authored and

  dieu

  committed
  871218b

  Copy full SHA for 871218b

Commits on Apr 17, 2017

• make ordered serialization stable across compilations (twitter#1664)

  Show description for e9d5d33
  

  fwbrasil

  authored and

  Piyush Narang

  committed
  e9d5d33

  Copy full SHA for e9d5d33

Commits on Apr 20, 2017

• Remove unnecessary semicolon (twitter#1668)

  

  zaneli

  authored and

  johnynek

  committed
  64d6351

  Copy full SHA for 64d6351

Commits on Apr 21, 2017

• Refactor the cascading code in the typed API into a self contained backend (twitter#1666)

  Show description for 3983250
  johnynek

  authored
  3983250

  Copy full SHA for 3983250

Commits on Apr 22, 2017

• Add tailrec annotation (twitter#1671)

  

  zaneli

  authored and

  johnynek

  committed
  31f618e

  Copy full SHA for 31f618e

Commits on Apr 24, 2017

• Refactor the cascading planning code out of Grouped (twitter#1670)

  Show description for ee2d592
  johnynek

  authored
  ee2d592

  Copy full SHA for ee2d592

Commits on Apr 26, 2017

• Move hashJoin related cascading code into CascadingBackend (twitter#1672)

  johnynek

  authored
  93c4658

  Copy full SHA for 93c4658

• Be more paranoid about Kryo registration order (twitter#1673)

  johnynek

  authored
  b7bb6f5

  Copy full SHA for b7bb6f5

Commits on May 1, 2017

• Update sbt version to 0.13.15 (twitter#1677)

  

  zaneli

  authored and

  johnynek

  committed
  10fd056

  Copy full SHA for 10fd056

Commits on May 2, 2017

• Register all Boxed classes in Kryo (twitter#1678)

  Show description for 9b70d7d
  

  jcdavis

  authored and

  johnynek

  committed
  9b70d7d

  Copy full SHA for 9b70d7d

• Move CoGrouped cascading logic into CascadingBackend (twitter#1676)

  johnynek

  authored
  b137949

  Copy full SHA for b137949

Commits on May 3, 2017

• Remove cascading dependency from Grouped/CoGrouped, seal the traits (twitter#1679)

  johnynek

  authored
  cde9692

  Copy full SHA for cde9692

Commits on May 6, 2017

• Pull cascading flowDef thread into cascading_backend (twitter#1681)

  Show description for 53e2568
  johnynek

  authored
  53e2568

  Copy full SHA for 53e2568

Commits on May 8, 2017

• Remove CascadingPipe node from TypedPipe ADT (twitter#1683)

  Show description for 396498a
  johnynek

  authored
  396498a

  Copy full SHA for 396498a

Commits on May 10, 2017

• Move toIterableExecution and forceToDiskExecution into Execution (twitter#1682)

  Show description for 6166921
  johnynek

  authored
  6166921

  Copy full SHA for 6166921

Commits on May 15, 2017

• Fix serialization of KryoHadoop (twitter#1685)

  Show description for 808ecb2
  

  ttim

  authored and

  johnynek

  committed
  808ecb2

  Copy full SHA for 808ecb2

Commits on May 17, 2017

• Refactoring memory and reducer estimator

  dieu

  committed
  ce455b7

  Copy full SHA for ce455b7

Commits on May 22, 2017

• Fix stack overflow in typedPipeMonoid.zero

  pankajroark

  committed
  2ca5089

  Copy full SHA for 2ca5089

• Timur's comments.

  pankajroark

  committed
  2d66748

  Copy full SHA for 2d66748

• Merge pull request twitter#1687 from twitter/pg/fix_typed_pipe_monoid

  Show description for 639e7eb
  pankajroark

  authored
  639e7eb

  Copy full SHA for 639e7eb

• Update the sbt runner script

  Show description for a3873c1
  

  Jackson Davis

  committed
  a3873c1

  Copy full SHA for a3873c1

• Merge pull request twitter#1689 from jcdavis/sbt-script-update

  Show description for 09436d1
  johnynek

  authored
  09436d1

  Copy full SHA for 09436d1

Commits on May 23, 2017

• Fix missed lint changes

  

  Jackson Davis

  committed
  35c0874

  Copy full SHA for 35c0874

Commits on May 24, 2017

• Makes use of ??? consistent

  benpence

  committed
  ddd09f7

  Copy full SHA for ddd09f7

• Comments

  benpence

  committed
  522f30d

  Copy full SHA for 522f30d

• Removes final

  benpence

  committed
  43607d0

  Copy full SHA for 43607d0

Commits on May 25, 2017

• Merge pull request twitter#1665 from benpence/add-execution-tutorial

  Show description for 3d98017
  benpence

  authored
  3d98017

  Copy full SHA for 3d98017

• Fix Execution tutorial typos

  benpence

  authored
  ede884e

  Copy full SHA for ede884e

• Merge pull request twitter#1691 from twitter/benpence-patch-3

  Show description for 625bea7
  benpence

  authored
  625bea7

  Copy full SHA for 625bea7

• Update Execution.md

  benpence

  authored
  3b9a166

  Copy full SHA for 3b9a166

• Merge pull request twitter#1692 from twitter/benpence-patch-3

  Show description for 8a9ab40
  benpence

  authored
  8a9ab40

  Copy full SHA for 8a9ab40

Commits on May 26, 2017

• Lookup classses using the mirorr instead of by name

  Show description for 4842cf8
  

  Jackson Davis

  committed
  4842cf8

  Copy full SHA for 4842cf8

• Merge pull request twitter#1693 from jcdavis/classfinder-improvements

  Show description for fc43985
  johnynek

  authored
  fc43985

  Copy full SHA for fc43985

Commits on May 29, 2017

• Merge pull request twitter#1690 from jcdavis/lint-fixes

  Show description for 4aeb09f
  johnynek

  authored
  4aeb09f

  Copy full SHA for 4aeb09f

Commits on Jun 1, 2017

• Wrap findReferencedClasses in try catch (twitter#1694)

  Show description for dacb39c
  

  Piyush Narang

  authored
  dacb39c

  Copy full SHA for dacb39c

Commits on Jun 4, 2017

• Make an ADT for CoGrouped

  johnynek

  committed
  4c1b7c3

  Copy full SHA for 4c1b7c3

Commits on Jun 7, 2017

• Merge pull request twitter#1698 from twitter/oscar/cogroup-adt

  Show description for d4f3655
  johnynek

  authored
  d4f3655

  Copy full SHA for d4f3655

Commits on Jun 19, 2017

• Merge pull request twitter#1667 from dieu/apanasenko/memory_estimation

  Show description for 7615dbf
  dieu

  authored
  7615dbf

  Copy full SHA for 7615dbf

Commits on Jun 23, 2017

• First draft of pure-scalding memory backend (twitter#1697)

  Show description for 026ba62
  johnynek

  authored
  026ba62

  Copy full SHA for 026ba62

• Break out estimator tests into separate module (twitter#1701)

  

  Piyush Narang

  authored
  dc80bdd

  Copy full SHA for dc80bdd

Commits on Jul 15, 2017

• Cherry pick 0.17.2 release related changes into develop (twitter#1710)

  Show description for c1044af
  ttim

  authored
  c1044af

  Copy full SHA for c1044af

Commits on Aug 3, 2017

• Add summingbird graph library as a zero-dep module

  johnynek

  committed
  f4b77ce

  Copy full SHA for f4b77ce

Commits on Aug 9, 2017

• Merge pull request twitter#1711 from twitter/oscar/mv-summingbird-graph-lib

  Show description for cc6e71d
  johnynek

  authored
  cc6e71d

  Copy full SHA for cc6e71d

Commits on Dec 19, 2017

• Merge branch 'master' of https://github.com/twitter/scalding into develop

  piyushnarang

  committed
  2f99c39

  Copy full SHA for 2f99c39

• Merge remote-tracking branch 'origin/master' into cascading3

  piyushnarang

  committed
  12a5325

  Copy full SHA for 12a5325

Commits on Dec 27, 2017

• Merge branch 'develop' into cascading3

  piyushnarang

  committed
  c87c0b5

  Copy full SHA for c87c0b5

Commits on Jan 2, 2018

• Fix StepNum / StepID in HRavenHistory service test

  piyushnarang

  committed
  ee5f699

  Copy full SHA for ee5f699

• Fix VKVS test broken by merge

  piyushnarang

  committed
  41fbdb1

  Copy full SHA for 41fbdb1

Commits on Jan 3, 2018

• Fix Guava exclusion error

  piyushnarang

  committed
  398878f

  Copy full SHA for 398878f

• Switch to getOrdinal instead of stepNum in HRavenHistoryService

  piyushnarang

  committed
  c4723c1

  Copy full SHA for c4723c1

Commits on Jan 17, 2018

• Use getOrdinal

  piyushnarang

  committed
  5bc26fd

  Copy full SHA for 5bc26fd

Commits on Jan 18, 2018

• Include Cyrille's fix from twitter#1605

  piyushnarang

  committed
  4e64cb5

  Copy full SHA for 4e64cb5

• Scala formatting fixes

  piyushnarang

  committed
  cc7f249

  Copy full SHA for cc7f249

Commits on Jan 22, 2018

• Switch step count to 3 in MergeTwoSinksForceToDiskTypedJob test

  piyushnarang

  committed
  5f1987e

  Copy full SHA for 5f1987e

• Fix for a couple of memory estimator tests

  piyushnarang

  committed
  e24fd40

  Copy full SHA for e24fd40

Commits on Jan 24, 2018

• Fix Oscar's comment on MergedType pipe in CascadingBackend

  piyushnarang

  committed
  3c350b9

  Copy full SHA for 3c350b9