You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
caseclassHistogramTransformation(hist: IndexedSeq[String]) extendsTransformation {
overridedeftransform(value: Any):Double=???/** * This method should determine if the new data will cause the creation of a new revision. * * @paramnewTransformation * the new transformation created with statistics over the new data * @return * true if the domain of the newTransformation is not fully contained in this one.*/overridedefisSupersededBy(newTransformation: Transformation):Boolean=???/** * Merges two transformations. The domain of the resulting transformation is the union of this * * @paramother * @return * a new Transformation that contains both this and other.*/overridedefmerge(other: Transformation):Transformation=???
}
objectHistogramTransformation {
defapply(hist: IndexedSeq[String]):HistogramTransformation=newHistogramTransformation(hist)
}
We would take advantage of the first step of OTreeDataAnalyzer and compute an approximate histogram or quartiles of the columns specified.
/** * Analyze a specific group of columns of the dataframe and extract valuable statistics * @paramdata * the data to analyze * @paramcolumnTransformers * the columns to analyze * @return*/private[index] defgetDataFrameStats(
data: DataFrame,
columnTransformers: IISeq[Transformer]):Row= {
valcolumnStats= columnTransformers.map(_.stats)
valcolumnsExpr= columnStats.flatMap(_.statsSqlPredicates)
data.selectExpr(columnsExpr ++Seq("count(1) AS count"): _*).first()
}
The text was updated successfully, but these errors were encountered:
After analyzing the efficiency of distribution functions for indexing (see issue #336 ), we can start implementing the HistogramTransformation.
The idea is to build it as another type of transformation, and eventually turn it into the default.
The API can be something like:
And under the hood:
We would take advantage of the first step of
OTreeDataAnalyzer
and compute an approximate histogram or quartiles of the columns specified.The text was updated successfully, but these errors were encountered: