MNT make a dataset containing no missing values

[glemaitre]

Addresses point 1. of #361 (comment)

I will make a PR in gitlab regarding the simplification of the wrap-up quiz.

Here is the code to remove the missing values:

# %%
import pandas as pd

ames_housing = pd.read_csv("../datasets/house_prices.csv", na_values="?")
ames_housing = ames_housing.drop(columns="Id")

target_name = "SalePrice"
data, target = ames_housing.drop(columns=target_name), ames_housing[target_name]
target = (target > 200_000).astype(int)

# %% [markdown]
# Preprocessing used to remove missing values.

# %%
from sklearn.compose import make_column_transformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline

numerical_features = [
    "LotFrontage",
    "LotArea",
    "MasVnrArea",
    "BsmtFinSF1",
    "BsmtFinSF2",
    "BsmtUnfSF",
    "TotalBsmtSF",
    "1stFlrSF",
    "2ndFlrSF",
    "LowQualFinSF",
    "GrLivArea",
    "BedroomAbvGr",
    "KitchenAbvGr",
    "TotRmsAbvGrd",
    "Fireplaces",
    "GarageCars",
    "GarageArea",
    "WoodDeckSF",
    "OpenPorchSF",
    "EnclosedPorch",
    "3SsnPorch",
    "ScreenPorch",
    "PoolArea",
    "MiscVal",
    target_name,
]
categorical_features = data.columns.difference(numerical_features)

categorical_processor = SimpleImputer(strategy="most_frequent")
numerical_processor = SimpleImputer()

preprocessor = make_column_transformer(
    (categorical_processor, categorical_features),
    (numerical_processor, numerical_features),
)
ames_housing_preprocessed = pd.DataFrame(
    preprocessor.fit_transform(ames_housing),
    columns=categorical_features.tolist() + numerical_features,
)
ames_housing_preprocessed = ames_housing_preprocessed[ames_housing.columns]
ames_housing_preprocessed = ames_housing_preprocessed.astype(ames_housing.dtypes)
ames_housing_preprocessed.to_csv("../datasets/ames_housing_no_missing.csv", index=False)

TODO:

• Change predictive modeling pipeline wrap-up quiz
• Change linear models wrap-up quiz
• Change tree-based models wrap-up quiz
• Remove Q. 9 and Q. 12 of linear models wrap-up quiz
• Update python_scripts/datasets_ames_housing.py

[ogrisel]

LGTM but we should not merge this PR prior to the v1.0 debrief meeting (early September).

[lesteve]

Note to our future selves: probably the appendix section about ames_housing dataset would need to change as well.

[ArturoAmorQ]

Is it worth keeping Q9 in the linear models wrap-up quiz

Are there any missing values in the dataset contained in the variable data?

after having removed any discussion on missing values?

In any case sounds like a question more suitable for M1 Tabular data exploration

[glemaitre]

Nop we could remove it. I did not remove it yet because it would change the order of the question numbering but it is good to add it in the TODO if we merge this

[ArturoAmorQ]

Note to our future selves: probably the appendix section about ames_housing dataset would need to change as well.

As we are keeping both house_prices.csv and ames_housing_no_missing.csv, do you think we should create separate notebooks to analyze both versions? Having a notebook for each dataset would come handy if one day we create a lesson about Imputers.

We could also just add a message at the end of datasets_ames_housing.py saying/showing how we created ames_housing_no_missing.csv from house_prices.csv.

What do you think?

[ogrisel]

+1 for a single notebook with the message you suggest.

[ogrisel]

LGTM, just a few minor suggestions:

[ogrisel]

Merged!