Home

Welcome to Hivemall, the scalable machine learning library for Hive.

Getting Started

• Installation
• Installing Hivemall UDFs as permanent functions
• Input formats for training (Please read this!)

Tips for Effective Hivemall

• Using explicit addBias() for better prediction
• Use rand_amplify() to better prediction results
• Building a Real-time Prediction by Integrating with RDBMS
• List of Supported Algorithms for each Hivemall Version

Advanced topics

• Ensemble learning for stable prediction
• Iterative training using distributed cache (#1 Logistic Regression on a9a dataset) (#2 Classification using Confidence Weight on news20 binary/multiclass dataset)
• Mixing models for a better prediction convergence (MIX server) [experimental]
• Run Hivemall on Amazon Elastic MapReduce
• Hivemall on Pig [experimental]
• Hivemall on Spark [experimental]

General Hive/Hadoop tips

• Adding rowid for each row
• Hadoop tuning for Hivemall
• Compressing a large training table
• Efficient Top-K query processing on Hive using Hivemall

Troubleshooting

• OutOfMemoryError in training
• SemanticException Generate Map Join Task Error: Cannot serialize object
• Asterisk argument for UDTF does not work
• The number of mappers is less than input splits in Hadoop 2.x
• Map-side Join causes ClassCastException (LazyBinaryArray cannot be cast to [Ljava.lang.Object) on Tez

---

Feature Engineering

• Feature Scaling/Feature Normalization
• Feature Hashing
• TF-IDF calculation

Feature Transformation

• Vectorize Features
• Quantify non-number features
• Polynomial Feature for Non-Linear Regression/Classification (a.k.a. Feature Pairing)

Evaluation

1. Statistical evaluation of a prediction model

Dataset generation

1. classification/logistic regression

Binary Classification

a9a binary classification

• Data preparation

1. Logistic Regression
2. Logistic Regression w/ Mini-batch Gradient Descent
3. Iterative training using distributed cache

news20 binary classification

• Data preparation

1. Perceptron, Passive Aggressive
2. CW, AROW, SCW
3. AdaGradRDA, AdaGrad, AdaDelta

KDD2010a/b binary classification

• Data preparation for KDD10a

1. PA/CW/AROW/SCW

• Data preparation for KDD10b

1. AROW

Webspam binary classification

• Data pareparation

1. PA1,AROW,SCW

Titanic survivor binary classification

1. RandomForest

Multiclass Classification

news20 multiclass classification

• Data preparation
• Data preparation for one vs the rest classifiers

1. PA
2. CW, AROW, SCW
3. Ensemble learning
4. one vs the rest classifier

Iris dataset multiclass classification

• Dataset preparation

1. SCW
2. RandomForest

Regression

E2006 tfidf regression

• Data preparation

1. Passive Aggressive, AROW

KDDCup 2012 track 2 CTR prediction

• Data preparation

1. Logistic Regression, Passive Aggressive
2. Logistic Regression with Amplifier
3. AdaGrad, AdaDelta

Recommendation

News20 multiclass related article recommendation

• Data preparation

1. LSH/Minhash

MovieLens movie recommendation

• Data preparation

1. Matrix Factorization
2. 10-fold Cross Validation (Matrix Factorization)
3. Factorization Machine

Nearest Neighbor

News20 multiclass similar article search

• Data preparation

1. LSH/Minhash and Brute-Force Search
2. kNN search using b-Bits Minhash

Anomaly Detection

1. Outlier Detection using Local Outlier Factor (LOF)

Natural Language Processing

1. English/Japanese Text Tokenizer