favorita-forecasting-model.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"62e20c21-896d-41b6-8faa-bb1a96c65a44","_uuid":"5d2e57e2-aa13-4bde-af85-08a002eb45dd","trusted":true},"source":["# Favorita"]},{"cell_type":"markdown","metadata":{"_cell_guid":"d091e965-120e-495d-a157-46c9b19cc6d4","_uuid":"14ed1e94-644a-420a-9168-3f64cf88e73d","trusted":true},"source":["## The Goal\n","\n","The goal is to build a machine learning model that can predict the unit sales for various items sold at different Favorita stores with higher accuracy, using a training dataset that includes information about dates, stores, items, promotions, and unit sales."]},{"cell_type":"markdown","metadata":{"_cell_guid":"7db0585d-1628-4adb-a9d9-a008c350cfc9","_uuid":"8cc3004b-b2a4-4de9-8768-c30d7403a655","trusted":true},"source":["## Kaggle notebook setup"]},{"cell_type":"markdown","metadata":{},"source":["Let's start to import some library..."]},{"cell_type":"code","execution_count":1,"metadata":{"_cell_guid":"1dc75da9-a5e1-4435-abd3-5138d04cc194","_uuid":"7c296e9e-20bf-4403-aa2d-6624cc9f415f","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:02:48.359267Z","iopub.status.busy":"2023-02-20T18:02:48.358773Z","iopub.status.idle":"2023-02-20T18:02:51.232212Z","shell.execute_reply":"2023-02-20T18:02:51.231186Z","shell.execute_reply.started":"2023-02-20T18:02:48.359221Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["import zipfile\n","from IPython.display import display, Markdown, HTML\n","import numpy as np\n","import pandas as pd\n","from sklearn.linear_model import LinearRegression\n","from sklearn.metrics import mean_squared_error, r2_score\n","import category_encoders as ce\n","import os\n","from sklearn.model_selection import train_test_split\n","import xgboost as xgb\n","import lightgbm as lgb\n","from sklearn.ensemble import VotingRegressor\n","from sklearn.ensemble import RandomForestRegressor\n","from sklearn.metrics import make_scorer"]},{"cell_type":"markdown","metadata":{},"source":["... and verify that we have data in input folder."]},{"cell_type":"code","execution_count":2,"metadata":{"_cell_guid":"cd1b3b1a-a576-49e4-badb-0f633f572a2c","_uuid":"da70cfce-cfe5-4e64-9db3-ab2a295114d3","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:02:51.237957Z","iopub.status.busy":"2023-02-20T18:02:51.237416Z","iopub.status.idle":"2023-02-20T18:02:51.247517Z","shell.execute_reply":"2023-02-20T18:02:51.246057Z","shell.execute_reply.started":"2023-02-20T18:02:51.237913Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["for dirname, _, filenames in os.walk(\"/input\"):\n","    for filename in filenames:\n","        print(os.path.join(dirname, filename))"]},{"cell_type":"markdown","metadata":{"_cell_guid":"dfbee8e9-814c-498a-89d9-3be4b3cd70a6","_uuid":"37f3f3e1-d97a-4d0a-b2f7-91f8c99b6f9c","trusted":true},"source":["Scripts"]},{"cell_type":"markdown","metadata":{"_cell_guid":"27cbd858-2504-439d-9ae0-1a48acbdb5f5","_uuid":"13e3b559-8c14-474d-8593-8e47b20a246c","trusted":true},"source":["For sake of clarify, we put here several functions that that perform different tasks related to data preparation, regression modeling, and data analysis.\n","<ul>\n","        <li><b>getMemoryUsage(df)</b>: Given a pandas DataFrame df, this function returns the total memory usage of the DataFrame in bytes. </li>\n","        <li><b>makeEmptyDf(df)</b>: Given a pandas DataFrame df, this function returns an empty DataFrame with the same column names as df.</li>\n","        <li><b>showDistinctValues(colName, dtFrame)</b>: Given a column name colName and a pandas DataFrame dtFrame, this function returns the number of distinct values in the specified column.</li>\n","        <li><b>showMissingData(df)</b>: Given a pandas DataFrame df, this function displays the columns that contain missing data (i.e., NaN values), and for each such column, it calls the showPercentageNan(colName, dtFrame) function to display the percentage of missing values in that column.</li>\n","        <li><b>showMissingDataPerCol(df, colName)</b>: Given a pandas DataFrame df and a column name colName, this function displays the number of missing values in the specified column and the percentage of missing values in that column, using the showPercentageNan(colName, dtFrame) function.</li>\n","        <li><b>showPercentageNan(colName, dtFrame)</b>: Given a column name colName and a pandas DataFrame dtFrame, this function displays the percentage of missing values in the specified column.</li>\n","        <li><b>basicEDA(df, title)</b>: Given a pandas DataFrame df and a string title, this function displays some basic information about the DataFrame, including its shape, data types, and summary statistics. If the DataFrame contains any missing values, it calls the showMissingData(df) function to display information about the missing values.</li>\n","        <li><b>getDistinctValues(df, lstFeatures)</b>: Given a pandas DataFrame df and a list of column names lstFeatures, this function displays the distinct values in each specified column.</li>\n","        <li><b>delColumn(df, colName)</b>: Given a pandas DataFrame df and a column name colName, this function removes the specified column from the DataFrame.</li>\n","        <li><b>reduce_mem_usage(df)</b>: Given a pandas DataFrame df, this function attempts to reduce the memory usage of the DataFrame by downcasting numeric columns to smaller data types where possible. It then displays information about the memory usage before and after the optimization.</li>\n","        <li><b>handleWithLinearRegression(df, colNameWithNa)</b>: This function handles missing values in a specified column of the input dataframe df using linear regression. The function first creates a copy of the input dataframe and converts any datetime columns to integers. The function then separates the rows with missing values in the specified column and the rows without missing values, and prepares the data for regression. It fits a linear regression model on the training data, predicts the missing values, and fills the missing values with the predictions. Finally, the function computes and prints the R2 and RMSE metrics for evaluating the regression performance and returns the filled dataframe and the number of missing values in the specified column. </li>\n","        <li><b>BinEncCategory(df, lstCols)</b>: This function applies binary encoding to categorical columns specified in lstCols of the input dataframe df. </li>\n","        <li><b>MSE(y_true,y_pred)</b>: This function computes and prints the mean squared error (MSE) between the true and predicted values. </li>\n","        <li><b>R2(y_true,y_pred)</b>: This function computes and prints the R2 score between the true and predicted values. </li>\n","        <li><b>two_score(y_true,y_pred)</b>: This function calls the MSE and R2 functions and returns the R2 score. </li>\n","        <li><b>two_scorer()</b>: This function returns a scorer object for use in cross-validation, which uses the two_score function to compute the score. </li>\n","        <li><b>executeVoter(X_train, X_test, y_train, y_test, whatToPredict)</b>: This function fits a voting regressor model using the specified models and returns the trained model. It also prints the estimators used in the voting regressor. </li>\n","        <li><b>makeCrossValidation(model, X_train, y_train, scoring)</b>: This function performs cross-validation on the input model using the specified scoring method and returns the results. </li>\n","        <li><b>fillTypeHolidayColumn(df)</b>: This function fills missing values in the \"type_Holiday\" column of the input dataframe df. It iterates over each row of the dataframe, and for each row with a missing value in the \"type_Holiday\" column, it checks whether the \"date\" column corresponds to a weekend or workday and fills the missing value accordingly. </li>\n","</ul>"]},{"cell_type":"code","execution_count":3,"metadata":{"_cell_guid":"71b0806b-b954-4ac6-8ed7-4211a168e63d","_uuid":"eaffc944-0ac6-46ac-ade9-ee0a2d2a6485","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:02:51.249850Z","iopub.status.busy":"2023-02-20T18:02:51.249297Z","iopub.status.idle":"2023-02-20T18:02:51.285167Z","shell.execute_reply":"2023-02-20T18:02:51.284529Z","shell.execute_reply.started":"2023-02-20T18:02:51.249815Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["def getMemoryUsage(df):\n","    return df.memory_usage().sum() # in bytes\n","\n","def makeEmptyDf(df):\n","    return df[0:0] \n","\n","def showDistinctValues(colName, dtFrame):\n","    return dtFrame.groupby([colName]).size()\n","\n","def showMissingData(df):\n","    colNan = list(df[df.columns[df.isna().any()]].columns)\n","    display(f\"The columns with missing data are: {colNan}\")\n","\n","    for col in colNan:\n","        showPercentageNan(col, df)\n","    return\n","\n","def showMissingDataPerCol(df, colName):\n","    nrNa = df[colName].isna().sum()\n","    display(f\"The '{colName}' column has {nrNa} missing data\")\n","    showPercentageNan(colName, df)\n","\n","def showPercentageNan(colName, dtFrame):\n","    nr = dtFrame[colName].isnull().sum()\n","    perc = (nr / dtFrame.shape[0])\n","    print(f\"Percent of missing '{colName}' records is {round(perc * 100,3)} % ({nr} values on {dtFrame.shape[0]} total)\")\n","    return\n","\n","def basicEDA(df, title):\n","    display(Markdown(\"**Just first five rows**\"))\n","    display(df.head(3))\n","    display(f\"The {title} data set consists of {df.shape[1]} different features which for {df.shape[0]} samples.\")\n","    display(Markdown(\"**Info about the index dtype and columns, non-null values and memory usage.**\"))\n","    display(df.info())\n","     \n","    # isnull() is just an alias of the isna() method as shown in pandas source code.\n","    nrNa = df.isna().sum()\n","    display(Markdown(\"**Count na values**\"))\n","    display(nrNa)\n","\n","    if nrNa.any() > 0:\n","        showMissingData(df)\n","        \n","        #colNan = list(df[df.columns[df.isna().any()]].columns)\n","        #display(f\"The columns with missing data are: {colNan}\")\n","\n","        #for col in colNan:\n","        #    showPercentageNan(col, df)\n","\n","    display(Markdown(\"**Show the statistic report of the numeric features of the dataset**\"))\n","    display(df.describe(datetime_is_numeric=True).transpose())   \n","    return\n","\n","def getDistinctValues(df, lstFeatures):\n","    display(Markdown(f\"Distint values in:\"))\n","    for category in lstFeatures:\n","        dist = sorted(df[category].unique())\n","        display(Markdown(f\"{category} -> {dist}\"))\n","\n","\n","def delColumn(df, colName):\n","    for name in df.columns:\n","        if name == colName:\n","            df.drop(colName, axis=1, inplace=True)\n","    return\n","\n","# from https://www.mikulskibartosz.name/how-to-reduce-memory-usage-in-pandas/\n","# (adapted)\n","def reduce_mem_usage(df):\n","    start_mem = df.memory_usage().sum() / 1024**2\n","    print(df.info())\n","    print(\"----------------\")\n","    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n","\n","    for col in df.columns:\n","        col_type = df[col].dtype\n","        if col_type != object:\n","                c_min = df[col].min()\n","                c_max = df[col].max()\n","                if str(col_type)[:3] == 'int':\n","                    if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n","                        df[col] = df[col].astype(np.int8)\n","                    elif c_min > np.iinfo(np.uint8).min and c_max < np.iinfo(np.uint8).max:\n","                        df[col] = df[col].astype(np.uint8)\n","                    elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n","                        df[col] = df[col].astype(np.int16)\n","                    elif c_min > np.iinfo(np.uint16).min and c_max < np.iinfo(np.uint16).max:\n","                        df[col] = df[col].astype(np.uint16)\n","                    elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n","                        df[col] = df[col].astype(np.int32)\n","                    elif c_min > np.iinfo(np.uint32).min and c_max < np.iinfo(np.uint32).max:\n","                        df[col] = df[col].astype(np.uint32)                    \n","                    elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n","                        df[col] = df[col].astype(np.int64)\n","                    elif c_min > np.iinfo(np.uint64).min and c_max < np.iinfo(np.uint64).max:\n","                        df[col] = df[col].astype(np.uint64)\n","                elif str(col_type)[:5] == 'float':\n","                    if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n","                        df[col] = df[col].astype(np.float16)\n","                    elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n","                        df[col] = df[col].astype(np.float32)\n","                    else:\n","                        df[col] = df[col].astype(np.float64)\n","        elif col_type == object:\n","            df[col] = df[col].astype(\"category\")\n","\n","    end_mem = df.memory_usage().sum() / 1024**2\n","    print('\\nMemory usage AFTER optimization is: {:.2f} MB'.format(end_mem))\n","    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n","    return df\n","\n","def handleWithLinearRegression(df, colNameWithNa):\n","    dftmp = df.copy()\n","    cols = list(dftmp.columns)\n","    for col in cols:\n","        if \"datetime\" in str(dftmp.dtypes[col]):\n","            dftmp[col] = dftmp[col].view(int)\n","\n","    # Separating missing or nan values rows\n","    test = dftmp[dftmp[colNameWithNa].isna()] # Assume the Age column contains missing values.\n","    train = dftmp.dropna()\n","    \n","    makeEmptyDf(dftmp)\n","    del dftmp\n","    \n","    # Preparing df\n","    X_train = train.drop([colNameWithNa], axis=1)\n","    y_train = train[colNameWithNa]\n","    \n","    X_test = test.drop([colNameWithNa], axis=1)\n","    \n","    # Getting ready a model to predict missing values(Genre column)\n","    linear = LinearRegression()\n","    linear.fit(X_train, y_train)\n","    # Predictions\n","    y_pred = linear.predict(X_test)\n","    # filling the missing values with predictions\n","    df.loc[df[colNameWithNa].isnull(), colNameWithNa] = y_pred\n","    \n","    # Running Evaluation Metrics\n","    predictions = linear.predict(X_test)\n","    r2 = r2_score(y_pred, predictions)\n","    rmse = mean_squared_error(y_pred, predictions, squared=False)\n","   \n","    print('The r2 is: ', r2)\n","    print('The rmse is: ', rmse)\n","    return df, df[colNameWithNa].isna().sum()\n","\n","\n","def BinEncCategory(df, lstCols):\n","    encoder = ce.BinaryEncoder(cols = lstCols)\n","    return encoder.fit_transform(df)\n","\n","# see: https://stats.stackexchange.com/questions/110599/how-to-get-both-mse-and-r2-from-a-sklearn-gridsearchcv\n","def MSE(y_true,y_pred):\n","    mse = mean_squared_error(y_true, y_pred)\n","    print(\"MSE: %2.3f\" % mse)\n","    \n","    return mse\n","\n","def R2(y_true,y_pred):    \n","    r2 = r2_score(y_true, y_pred)\n","    print(\"R2: %2.3f\" % r2) \n","    return r2\n","    \n","def two_score(y_true,y_pred):    \n","    MSE(y_true,y_pred) # set score here and not below if using MSE in GridCV\n","    score = R2(y_true,y_pred)\n","    return score\n","\n","def two_scorer():\n","    return make_scorer(two_score, greater_is_better=True) # change for false if using MSE\n","\n","def executeVoter(X_train, X_test, y_train, y_test, whatToPredict):\n","    models = [\n","        ['lr', LinearRegression()],\n","        ['rfr', RandomForestRegressor()],\n","        ['lgb', lgb.LGBMRegressor()],\n","        ['xgb', xgb.XGBRegressor()]\n","    ]\n","\n","    voter_regr = VotingRegressor(models)\n","    voter_regr.fit(X_train, y_train)\n","    voter_regr.score(X_test, y_test)\n","\n","    voter_regr.predict(X_test[:whatToPredict])\n","    print(f\"This are the estimators:\\n {voter_regr.named_estimators_}\")\n","    \n","    return voter_regr\n","\n","def makeCrossValidation(model, X_train, y_train, scoring):\n","    # https://www.kaggle.com/code/cesarsupo/tps-09-votingregressor\n","    from sklearn.model_selection import cross_validate\n","\n","    return cross_validate(model, X_train, y_train, cv=2, n_jobs=-1,\n","                         scoring=scoring,\n","                         return_estimator = True\n","    )\n","\n","def fillTypeHolidayColumn(df):\n","    for index, row in df.iterrows():\n","        if pd.isnull(df.loc[index, \"type_Holiday\"]):\n","            test = pd.to_datetime(df.loc[index,\"date\"]).strftime(\"%A\")\n","            if test == \"Sunday\" or test == \"Saturday\":\n","                df.at[index,\"type_Holiday\"] = \"Weekend\"\n","            else:\n","                df.at[index,\"type_Holiday\"] = \"Work Day\"\n","    return df"]},{"cell_type":"markdown","metadata":{},"source":["Let's enable automatic garbage collection to free automatically memory that is no longer in use by the program."]},{"cell_type":"code","execution_count":4,"metadata":{"_cell_guid":"6b4b8e7d-a247-44f9-8e5d-f90333d6ccf5","_uuid":"fad1e38f-ae02-4fee-b6d2-e4cddcd50db2","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:02:51.288862Z","iopub.status.busy":"2023-02-20T18:02:51.287569Z","iopub.status.idle":"2023-02-20T18:02:51.301799Z","shell.execute_reply":"2023-02-20T18:02:51.300965Z","shell.execute_reply.started":"2023-02-20T18:02:51.288800Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["import gc\n","gc.enable()"]},{"cell_type":"markdown","metadata":{"_cell_guid":"74f5e5d4-2f2d-4451-9aa6-bd734385a865","_uuid":"d7f493fb-8611-4ac5-90b9-96a2e9131c23","trusted":true},"source":["#  Import data sets <a name=\"importdataset\"></a>"]},{"cell_type":"markdown","metadata":{"_cell_guid":"0dec65aa-8d07-4826-91d9-342ca479a646","_uuid":"71410d4a-9728-4cb6-86cf-fca9ab727927","trusted":true},"source":["We have available a variety of data sources, each of which provides valuable insights into different aspects of the problem at hand.\n","\n","The Holidays and Events dataset contains metadata that can be used to understand past sales trends and seasonal patterns, although it does require some additional manipulation to make it useful. \n","\n","Similarly, the Daily Oil Price data is particularly relevant to our analysis, given that Ecuador's economy is heavily influenced by oil prices, and we can use it to identify which product families are most impacted by fluctuations in oil prices. \n","\n","Meanwhile, the Stores data provides information on the location and type of each store, which can help us better understand regional differences in sales patterns. \n","\n","Finally, the Transaction data is particularly useful, as it is highly correlated with the sales column in the training data, and can be used to identify patterns and trends in store sales over time. \n","\n","Speaking of the training data, it consists of time series data for each store and product family combination, and includes a sales column that provides the total sales for each product family at a particular store on a given date. \n","And a test dataset will be used to evaluate the performance of our models."]},{"cell_type":"markdown","metadata":{},"source":["So define paths..."]},{"cell_type":"code","execution_count":5,"metadata":{"_cell_guid":"82973371-bd35-45ba-8d8f-0b249726ccf6","_uuid":"5b844c03-27f0-4b06-bf08-318ad93908e1","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:02:51.303109Z","iopub.status.busy":"2023-02-20T18:02:51.302864Z","iopub.status.idle":"2023-02-20T18:02:51.313569Z","shell.execute_reply":"2023-02-20T18:02:51.312571Z","shell.execute_reply.started":"2023-02-20T18:02:51.303087Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["pathHolidays = \"./input/holidays_events.csv\"\n","pathOilPrices = \"./input/oil.csv\"\n","pathStores = \"./input/stores.csv\"\n","pathTransactions = \"./input/transactions.csv\"\n","pathTrain = \"./input/train.csv\"\n","pathTest = \"./input/test.csv\""]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["Clean up old csv datasets..."]},{"cell_type":"code","execution_count":6,"metadata":{},"outputs":[],"source":["for dirname, _, filenames in os.walk(\"./input\"):\n","    for filename in filenames:\n","        if filename.endswith(('csv')):\n","            os.remove(os.path.join(dirname, filename))"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["...and extract the new ones."]},{"cell_type":"code","execution_count":7,"metadata":{},"outputs":[],"source":["with zipfile.ZipFile(\"./input/input.zip\",\"r\") as z:\n","    z.extractall(\"./input\")"]},{"cell_type":"markdown","metadata":{},"source":["... and load them in separate Panda's data frame objects."]},{"cell_type":"code","execution_count":8,"metadata":{"_cell_guid":"1613b9a2-34f0-4f77-aa8f-de522baee613","_uuid":"d4450233-95a7-4e47-a7ef-b00d8640bd5e","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:02:51.315415Z","iopub.status.busy":"2023-02-20T18:02:51.315160Z","iopub.status.idle":"2023-02-20T18:02:54.710017Z","shell.execute_reply":"2023-02-20T18:02:54.709400Z","shell.execute_reply.started":"2023-02-20T18:02:51.315392Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["dfHolidays = pd.read_csv(pathHolidays, sep=\",\", parse_dates=['date'])\n","dfHolidays.Name = \"Holidays\"\n","\n","dfOilPrices = pd.read_csv(pathOilPrices, sep=\",\", parse_dates=['date'])\n","dfOilPrices.Name = \"OilPrices\"\n","\n","dfStores = pd.read_csv(pathStores, sep=\",\")\n","dfStores.Name = \"Stores\"\n","\n","dfTransactions = pd.read_csv(pathTransactions, sep=\",\", parse_dates=['date'])\n","dfTransactions.Name = \"transaction\"\n","\n","dfTrain = pd.read_csv(pathTrain, sep=\",\", parse_dates=['date'])\n","dfTrain.Name = \"Train\"\n","\n","dfTest = pd.read_csv(pathTest, sep=\",\", parse_dates=['date'])\n","dfTest.Name = \"Test\""]},{"cell_type":"markdown","metadata":{"_cell_guid":"25593bcd-ddb3-4de6-832a-690ce066ef4d","_uuid":"66dd8bf3-02fe-433d-a744-56e717fa1d5d","trusted":true},"source":["# Profiling datasets <a name=\"profilingdatasets\"></a>"]},{"cell_type":"markdown","metadata":{"_cell_guid":"a46828f8-d6c9-49da-9fd6-5560613d0347","_uuid":"1e32f81b-05af-458f-a58f-b5bb7bc60683","trusted":true},"source":["### Profiling Train and Test <a name=\"profilingtraintest\"></a>"]},{"cell_type":"code","execution_count":9,"metadata":{"_cell_guid":"88bf99e0-4f2c-42ba-b802-5886531bccf6","_uuid":"6614360d-e713-4375-9e45-cf9bbd307331","execution":{"iopub.execute_input":"2023-02-20T18:02:54.711778Z","iopub.status.busy":"2023-02-20T18:02:54.711486Z","iopub.status.idle":"2023-02-20T18:02:55.218868Z","shell.execute_reply":"2023-02-20T18:02:55.217791Z","shell.execute_reply.started":"2023-02-20T18:02:54.711755Z"},"trusted":true},"outputs":[{"data":{"text/markdown":["**Just first five rows**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>id</th>\n","      <th>date</th>\n","      <th>store_nbr</th>\n","      <th>family</th>\n","      <th>sales</th>\n","      <th>onpromotion</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0</td>\n","      <td>2013-01-01</td>\n","      <td>1</td>\n","      <td>AUTOMOTIVE</td>\n","      <td>0.0</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1</td>\n","      <td>2013-01-01</td>\n","      <td>1</td>\n","      <td>BABY CARE</td>\n","      <td>0.0</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2</td>\n","      <td>2013-01-01</td>\n","      <td>1</td>\n","      <td>BEAUTY</td>\n","      <td>0.0</td>\n","      <td>0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   id       date  store_nbr      family  sales  onpromotion\n","0   0 2013-01-01          1  AUTOMOTIVE    0.0            0\n","1   1 2013-01-01          1   BABY CARE    0.0            0\n","2   2 2013-01-01          1      BEAUTY    0.0            0"]},"metadata":{},"output_type":"display_data"},{"data":{"text/plain":["'The Train data set consists of 6 different features which for 3000888 samples.'"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Info about the index dtype and columns, non-null values and memory usage.**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["<class 'pandas.core.frame.DataFrame'>\n","RangeIndex: 3000888 entries, 0 to 3000887\n","Data columns (total 6 columns):\n"," #   Column       Dtype         \n","---  ------       -----         \n"," 0   id           int64         \n"," 1   date         datetime64[ns]\n"," 2   store_nbr    int64         \n"," 3   family       object        \n"," 4   sales        float64       \n"," 5   onpromotion  int64         \n","dtypes: datetime64[ns](1), float64(1), int64(3), object(1)\n","memory usage: 137.4+ MB\n"]},{"data":{"text/plain":["None"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Count na values**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/plain":["id             0\n","date           0\n","store_nbr      0\n","family         0\n","sales          0\n","onpromotion    0\n","dtype: int64"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Show the statistic report of the numeric features of the dataset**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>count</th>\n","      <th>mean</th>\n","      <th>min</th>\n","      <th>25%</th>\n","      <th>50%</th>\n","      <th>75%</th>\n","      <th>max</th>\n","      <th>std</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>id</th>\n","      <td>3.00089e+06</td>\n","      <td>1.50044e+06</td>\n","      <td>0</td>\n","      <td>750222</td>\n","      <td>1.50044e+06</td>\n","      <td>2.25067e+06</td>\n","      <td>3.00089e+06</td>\n","      <td>866282</td>\n","    </tr>\n","    <tr>\n","      <th>date</th>\n","      <td>3000888</td>\n","      <td>2015-04-24 08:27:04.703216128</td>\n","      <td>2013-01-01 00:00:00</td>\n","      <td>2014-02-26 18:00:00</td>\n","      <td>2015-04-24 12:00:00</td>\n","      <td>2016-06-19 06:00:00</td>\n","      <td>2017-08-15 00:00:00</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>store_nbr</th>\n","      <td>3.00089e+06</td>\n","      <td>27.5</td>\n","      <td>1</td>\n","      <td>14</td>\n","      <td>27.5</td>\n","      <td>41</td>\n","      <td>54</td>\n","      <td>15.5858</td>\n","    </tr>\n","    <tr>\n","      <th>sales</th>\n","      <td>3.00089e+06</td>\n","      <td>357.776</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>11</td>\n","      <td>195.847</td>\n","      <td>124717</td>\n","      <td>1102</td>\n","    </tr>\n","    <tr>\n","      <th>onpromotion</th>\n","      <td>3.00089e+06</td>\n","      <td>2.60277</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>741</td>\n","      <td>12.2189</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                   count                           mean                  min  \\\n","id           3.00089e+06                    1.50044e+06                    0   \n","date             3000888  2015-04-24 08:27:04.703216128  2013-01-01 00:00:00   \n","store_nbr    3.00089e+06                           27.5                    1   \n","sales        3.00089e+06                        357.776                    0   \n","onpromotion  3.00089e+06                        2.60277                    0   \n","\n","                             25%                  50%                  75%  \\\n","id                        750222          1.50044e+06          2.25067e+06   \n","date         2014-02-26 18:00:00  2015-04-24 12:00:00  2016-06-19 06:00:00   \n","store_nbr                     14                 27.5                   41   \n","sales                          0                   11              195.847   \n","onpromotion                    0                    0                    0   \n","\n","                             max      std  \n","id                   3.00089e+06   866282  \n","date         2017-08-15 00:00:00      NaN  \n","store_nbr                     54  15.5858  \n","sales                     124717     1102  \n","onpromotion                  741  12.2189  "]},"metadata":{},"output_type":"display_data"}],"source":["basicEDA(dfTrain, \"Train\")"]},{"cell_type":"markdown","metadata":{"_cell_guid":"6d3c8168-0775-4e70-a23e-dfa86c64d04c","_uuid":"50325c34-a7ec-4f33-ac94-28eec3613585","trusted":true},"source":["We can see there is no missing data. And the distinct values of family are:"]},{"cell_type":"code","execution_count":10,"metadata":{"_cell_guid":"aefa14f5-cb63-4b04-bbe2-e44e429de431","_uuid":"3cb07855-da18-42b6-9071-3ac883a907c2","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:02:55.220868Z","iopub.status.busy":"2023-02-20T18:02:55.220544Z","iopub.status.idle":"2023-02-20T18:02:55.402856Z","shell.execute_reply":"2023-02-20T18:02:55.402024Z","shell.execute_reply.started":"2023-02-20T18:02:55.220843Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/markdown":["Distint values in:"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["family -> ['AUTOMOTIVE', 'BABY CARE', 'BEAUTY', 'BEVERAGES', 'BOOKS', 'BREAD/BAKERY', 'CELEBRATION', 'CLEANING', 'DAIRY', 'DELI', 'EGGS', 'FROZEN FOODS', 'GROCERY I', 'GROCERY II', 'HARDWARE', 'HOME AND KITCHEN I', 'HOME AND KITCHEN II', 'HOME APPLIANCES', 'HOME CARE', 'LADIESWEAR', 'LAWN AND GARDEN', 'LINGERIE', 'LIQUOR,WINE,BEER', 'MAGAZINES', 'MEATS', 'PERSONAL CARE', 'PET SUPPLIES', 'PLAYERS AND ELECTRONICS', 'POULTRY', 'PREPARED FOODS', 'PRODUCE', 'SCHOOL AND OFFICE SUPPLIES', 'SEAFOOD']"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"}],"source":["getDistinctValues(dfTrain, [\"family\"])"]},{"cell_type":"markdown","metadata":{"_cell_guid":"08ca0790-61d5-4766-9e2c-da068ab51f8b","_uuid":"16fa287c-d32e-46fb-8e49-d6eda4f591ee","trusted":true},"source":["Same thing about test data."]},{"cell_type":"code","execution_count":11,"metadata":{"_cell_guid":"ffe099d6-16e0-45e8-a7ea-f3532afd6072","_uuid":"f198eb55-bd81-4b3c-beff-e114566f181a","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:02:55.404285Z","iopub.status.busy":"2023-02-20T18:02:55.403718Z","iopub.status.idle":"2023-02-20T18:02:55.453164Z","shell.execute_reply":"2023-02-20T18:02:55.452213Z","shell.execute_reply.started":"2023-02-20T18:02:55.404261Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/markdown":["**Just first five rows**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>id</th>\n","      <th>date</th>\n","      <th>store_nbr</th>\n","      <th>family</th>\n","      <th>onpromotion</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>3000888</td>\n","      <td>2017-08-16</td>\n","      <td>1</td>\n","      <td>AUTOMOTIVE</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>3000889</td>\n","      <td>2017-08-16</td>\n","      <td>1</td>\n","      <td>BABY CARE</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>3000890</td>\n","      <td>2017-08-16</td>\n","      <td>1</td>\n","      <td>BEAUTY</td>\n","      <td>2</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["        id       date  store_nbr      family  onpromotion\n","0  3000888 2017-08-16          1  AUTOMOTIVE            0\n","1  3000889 2017-08-16          1   BABY CARE            0\n","2  3000890 2017-08-16          1      BEAUTY            2"]},"metadata":{},"output_type":"display_data"},{"data":{"text/plain":["'The Test data set consists of 5 different features which for 28512 samples.'"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Info about the index dtype and columns, non-null values and memory usage.**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["<class 'pandas.core.frame.DataFrame'>\n","RangeIndex: 28512 entries, 0 to 28511\n","Data columns (total 5 columns):\n"," #   Column       Non-Null Count  Dtype         \n","---  ------       --------------  -----         \n"," 0   id           28512 non-null  int64         \n"," 1   date         28512 non-null  datetime64[ns]\n"," 2   store_nbr    28512 non-null  int64         \n"," 3   family       28512 non-null  object        \n"," 4   onpromotion  28512 non-null  int64         \n","dtypes: datetime64[ns](1), int64(3), object(1)\n","memory usage: 1.1+ MB\n"]},{"data":{"text/plain":["None"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Count na values**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/plain":["id             0\n","date           0\n","store_nbr      0\n","family         0\n","onpromotion    0\n","dtype: int64"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Show the statistic report of the numeric features of the dataset**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>count</th>\n","      <th>mean</th>\n","      <th>min</th>\n","      <th>25%</th>\n","      <th>50%</th>\n","      <th>75%</th>\n","      <th>max</th>\n","      <th>std</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>id</th>\n","      <td>28512</td>\n","      <td>3.01514e+06</td>\n","      <td>3.00089e+06</td>\n","      <td>3.00802e+06</td>\n","      <td>3.01514e+06</td>\n","      <td>3.02227e+06</td>\n","      <td>3.0294e+06</td>\n","      <td>8230.85</td>\n","    </tr>\n","    <tr>\n","      <th>date</th>\n","      <td>28512</td>\n","      <td>2017-08-23 12:00:00.000363776</td>\n","      <td>2017-08-16 00:00:00</td>\n","      <td>2017-08-19 18:00:00</td>\n","      <td>2017-08-23 12:00:00</td>\n","      <td>2017-08-27 06:00:00</td>\n","      <td>2017-08-31 00:00:00</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>store_nbr</th>\n","      <td>28512</td>\n","      <td>27.5</td>\n","      <td>1</td>\n","      <td>14</td>\n","      <td>27.5</td>\n","      <td>41</td>\n","      <td>54</td>\n","      <td>15.5861</td>\n","    </tr>\n","    <tr>\n","      <th>onpromotion</th>\n","      <td>28512</td>\n","      <td>6.96538</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>6</td>\n","      <td>646</td>\n","      <td>20.684</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["             count                           mean                  min  \\\n","id           28512                    3.01514e+06          3.00089e+06   \n","date         28512  2017-08-23 12:00:00.000363776  2017-08-16 00:00:00   \n","store_nbr    28512                           27.5                    1   \n","onpromotion  28512                        6.96538                    0   \n","\n","                             25%                  50%                  75%  \\\n","id                   3.00802e+06          3.01514e+06          3.02227e+06   \n","date         2017-08-19 18:00:00  2017-08-23 12:00:00  2017-08-27 06:00:00   \n","store_nbr                     14                 27.5                   41   \n","onpromotion                    0                    0                    6   \n","\n","                             max      std  \n","id                    3.0294e+06  8230.85  \n","date         2017-08-31 00:00:00      NaN  \n","store_nbr                     54  15.5861  \n","onpromotion                  646   20.684  "]},"metadata":{},"output_type":"display_data"}],"source":["basicEDA(dfTest, \"Test\")"]},{"cell_type":"markdown","metadata":{"_cell_guid":"3f8adeda-e0be-4c77-b182-264d1d935ec5","_uuid":"4b97f2cb-f791-4eda-9e19-f3c7f6937dc2","trusted":true},"source":["holidays\n","\tdatetime64, object, bool\n","\tno na values\n","oilprices\n","\tdatetime64,\tfloat64\n","\tsome prices missed (3.53%)\n","stores\n","\tint64, object\n","\tno na values\n","transactions\n","\tdatetime64, int64\n","\tno na values\n","train /test\n","\tint64, datetime64[ns], object, float64, int64\n","\tno missed data"]},{"cell_type":"markdown","metadata":{"_cell_guid":"47b873b4-b8a9-4fda-938e-26ae3cdefd05","_uuid":"72a7a508-a905-42f9-bbac-c390843d08ea","trusted":true},"source":["### Profiling Holidays <a name=\"profilingholidays\"></a>"]},{"cell_type":"code","execution_count":12,"metadata":{"_cell_guid":"d3ac1406-2641-449b-98e0-87a1133e4371","_uuid":"94a9bf3b-adb1-409a-847c-2b9fd40f8c7e","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:02:55.454673Z","iopub.status.busy":"2023-02-20T18:02:55.454412Z","iopub.status.idle":"2023-02-20T18:02:55.500820Z","shell.execute_reply":"2023-02-20T18:02:55.499841Z","shell.execute_reply.started":"2023-02-20T18:02:55.454651Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/markdown":["**Just first five rows**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>date</th>\n","      <th>type</th>\n","      <th>locale</th>\n","      <th>locale_name</th>\n","      <th>description</th>\n","      <th>transferred</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>2012-03-02</td>\n","      <td>Holiday</td>\n","      <td>Local</td>\n","      <td>Manta</td>\n","      <td>Fundacion de Manta</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>2012-04-01</td>\n","      <td>Holiday</td>\n","      <td>Regional</td>\n","      <td>Cotopaxi</td>\n","      <td>Provincializacion de Cotopaxi</td>\n","      <td>False</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2012-04-12</td>\n","      <td>Holiday</td>\n","      <td>Local</td>\n","      <td>Cuenca</td>\n","      <td>Fundacion de Cuenca</td>\n","      <td>False</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["        date     type    locale locale_name                    description  \\\n","0 2012-03-02  Holiday     Local       Manta             Fundacion de Manta   \n","1 2012-04-01  Holiday  Regional    Cotopaxi  Provincializacion de Cotopaxi   \n","2 2012-04-12  Holiday     Local      Cuenca            Fundacion de Cuenca   \n","\n","   transferred  \n","0        False  \n","1        False  \n","2        False  "]},"metadata":{},"output_type":"display_data"},{"data":{"text/plain":["'The Holidays data set consists of 6 different features which for 350 samples.'"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Info about the index dtype and columns, non-null values and memory usage.**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["<class 'pandas.core.frame.DataFrame'>\n","RangeIndex: 350 entries, 0 to 349\n","Data columns (total 6 columns):\n"," #   Column       Non-Null Count  Dtype         \n","---  ------       --------------  -----         \n"," 0   date         350 non-null    datetime64[ns]\n"," 1   type         350 non-null    object        \n"," 2   locale       350 non-null    object        \n"," 3   locale_name  350 non-null    object        \n"," 4   description  350 non-null    object        \n"," 5   transferred  350 non-null    bool          \n","dtypes: bool(1), datetime64[ns](1), object(4)\n","memory usage: 14.1+ KB\n"]},{"data":{"text/plain":["None"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Count na values**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/plain":["date           0\n","type           0\n","locale         0\n","locale_name    0\n","description    0\n","transferred    0\n","dtype: int64"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Show the statistic report of the numeric features of the dataset**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>count</th>\n","      <th>mean</th>\n","      <th>min</th>\n","      <th>25%</th>\n","      <th>50%</th>\n","      <th>75%</th>\n","      <th>max</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>date</th>\n","      <td>350</td>\n","      <td>2015-04-24 00:45:15.428571392</td>\n","      <td>2012-03-02</td>\n","      <td>2013-12-23 06:00:00</td>\n","      <td>2015-06-08</td>\n","      <td>2016-07-03</td>\n","      <td>2017-12-26</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["     count                          mean        min                 25%  \\\n","date   350 2015-04-24 00:45:15.428571392 2012-03-02 2013-12-23 06:00:00   \n","\n","            50%        75%        max  \n","date 2015-06-08 2016-07-03 2017-12-26  "]},"metadata":{},"output_type":"display_data"}],"source":["basicEDA(dfHolidays, \"Holidays\")"]},{"cell_type":"markdown","metadata":{"_cell_guid":"415ae803-f228-4d31-8c3b-70632e909b36","_uuid":"e930436e-8bbb-4c4f-9bcf-bc53fbba34ff","trusted":true},"source":["We can see there is no missing data. And the distinct values of locale, locale_name and type are:"]},{"cell_type":"code","execution_count":13,"metadata":{"_cell_guid":"380b0b94-f289-44ff-ba06-f0169f23fed8","_uuid":"d5540864-44e4-41d1-8e35-6f6320a336e3","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:02:55.502578Z","iopub.status.busy":"2023-02-20T18:02:55.502275Z","iopub.status.idle":"2023-02-20T18:02:55.517764Z","shell.execute_reply":"2023-02-20T18:02:55.516038Z","shell.execute_reply.started":"2023-02-20T18:02:55.502555Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/markdown":["Distint values in:"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["locale -> ['Local', 'National', 'Regional']"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["locale_name -> ['Ambato', 'Cayambe', 'Cotopaxi', 'Cuenca', 'Ecuador', 'El Carmen', 'Esmeraldas', 'Guaranda', 'Guayaquil', 'Ibarra', 'Imbabura', 'Latacunga', 'Libertad', 'Loja', 'Machala', 'Manta', 'Puyo', 'Quevedo', 'Quito', 'Riobamba', 'Salinas', 'Santa Elena', 'Santo Domingo', 'Santo Domingo de los Tsachilas']"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["type -> ['Additional', 'Bridge', 'Event', 'Holiday', 'Transfer', 'Work Day']"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"}],"source":["getDistinctValues(dfHolidays, [\"locale\", \"locale_name\", \"type\"])"]},{"cell_type":"markdown","metadata":{"_cell_guid":"fc7870f8-ea4e-492a-ab1f-802f2c9715f5","_uuid":"9daa2ec1-d474-4eb5-ac3c-412977c575ba","trusted":true},"source":["### Profiling Oil prices <a name=\"profilingoilprices\"></a>"]},{"cell_type":"code","execution_count":14,"metadata":{"_cell_guid":"19c17d9e-145a-4e79-af0f-27bbc7168c46","_uuid":"607456cb-3334-4172-88a9-47ecd022b16e","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:02:55.519732Z","iopub.status.busy":"2023-02-20T18:02:55.519439Z","iopub.status.idle":"2023-02-20T18:02:55.568781Z","shell.execute_reply":"2023-02-20T18:02:55.567620Z","shell.execute_reply.started":"2023-02-20T18:02:55.519709Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/markdown":["**Just first five rows**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>date</th>\n","      <th>dcoilwtico</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>2013-01-01</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>2013-01-02</td>\n","      <td>93.14</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2013-01-03</td>\n","      <td>92.97</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["        date  dcoilwtico\n","0 2013-01-01         NaN\n","1 2013-01-02       93.14\n","2 2013-01-03       92.97"]},"metadata":{},"output_type":"display_data"},{"data":{"text/plain":["'The Oil prices data set consists of 2 different features which for 1218 samples.'"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Info about the index dtype and columns, non-null values and memory usage.**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["<class 'pandas.core.frame.DataFrame'>\n","RangeIndex: 1218 entries, 0 to 1217\n","Data columns (total 2 columns):\n"," #   Column      Non-Null Count  Dtype         \n","---  ------      --------------  -----         \n"," 0   date        1218 non-null   datetime64[ns]\n"," 1   dcoilwtico  1175 non-null   float64       \n","dtypes: datetime64[ns](1), float64(1)\n","memory usage: 19.2 KB\n"]},{"data":{"text/plain":["None"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Count na values**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/plain":["date           0\n","dcoilwtico    43\n","dtype: int64"]},"metadata":{},"output_type":"display_data"},{"data":{"text/plain":["\"The columns with missing data are: ['dcoilwtico']\""]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["Percent of missing 'dcoilwtico' records is 3.53 % (43 values on 1218 total)\n"]},{"data":{"text/markdown":["**Show the statistic report of the numeric features of the dataset**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>count</th>\n","      <th>mean</th>\n","      <th>min</th>\n","      <th>25%</th>\n","      <th>50%</th>\n","      <th>75%</th>\n","      <th>max</th>\n","      <th>std</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>date</th>\n","      <td>1218</td>\n","      <td>2015-05-02 12:00:00.000012544</td>\n","      <td>2013-01-01 00:00:00</td>\n","      <td>2014-03-03 06:00:00</td>\n","      <td>2015-05-02 12:00:00</td>\n","      <td>2016-06-30 18:00:00</td>\n","      <td>2017-08-31 00:00:00</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>dcoilwtico</th>\n","      <td>1175</td>\n","      <td>67.7144</td>\n","      <td>26.19</td>\n","      <td>46.405</td>\n","      <td>53.19</td>\n","      <td>95.66</td>\n","      <td>110.62</td>\n","      <td>25.6305</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["           count                           mean                  min  \\\n","date        1218  2015-05-02 12:00:00.000012544  2013-01-01 00:00:00   \n","dcoilwtico  1175                        67.7144                26.19   \n","\n","                            25%                  50%                  75%  \\\n","date        2014-03-03 06:00:00  2015-05-02 12:00:00  2016-06-30 18:00:00   \n","dcoilwtico               46.405                53.19                95.66   \n","\n","                            max      std  \n","date        2017-08-31 00:00:00      NaN  \n","dcoilwtico               110.62  25.6305  "]},"metadata":{},"output_type":"display_data"}],"source":["basicEDA(dfOilPrices, \"Oil prices\")"]},{"cell_type":"markdown","metadata":{"_cell_guid":"78555415-a697-4c9b-b664-2cbd0ddd107b","_uuid":"82a0c8cf-4e41-4a5e-ae55-5f223f0a7140","trusted":true},"source":["We've found 43 dcoilwtico missing values, so let's handling missing data using Linear Regression."]},{"cell_type":"code","execution_count":15,"metadata":{"_cell_guid":"670c0229-53b6-40a5-94a7-12da28fe2158","_uuid":"9cf34640-ab07-42e4-8103-a6d3d0633d3a","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:02:55.575548Z","iopub.status.busy":"2023-02-20T18:02:55.574599Z","iopub.status.idle":"2023-02-20T18:02:55.598761Z","shell.execute_reply":"2023-02-20T18:02:55.597724Z","shell.execute_reply.started":"2023-02-20T18:02:55.575522Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["The r2 is:  1.0\n","The rmse is:  0.0\n","0\n"]}],"source":["dfOilPrices, verifyScore = handleWithLinearRegression(dfOilPrices, \"dcoilwtico\")\n","print(verifyScore) # overfitting?"]},{"cell_type":"markdown","metadata":{"_cell_guid":"bdadeb44-92ac-46f9-8ec9-55e689d584d4","_uuid":"2fdf7583-d593-44a6-a092-aa49d87aba7d","trusted":true},"source":["### Profiling Stores <a name=\"profilingstores\"></a>"]},{"cell_type":"code","execution_count":16,"metadata":{"_cell_guid":"b383b8d4-bf4b-482f-9995-391c3f40420e","_uuid":"2436a63b-0519-44c3-8460-51ce6aeb10f6","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:02:55.600326Z","iopub.status.busy":"2023-02-20T18:02:55.599999Z","iopub.status.idle":"2023-02-20T18:02:55.646455Z","shell.execute_reply":"2023-02-20T18:02:55.645444Z","shell.execute_reply.started":"2023-02-20T18:02:55.600303Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/markdown":["**Just first five rows**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>store_nbr</th>\n","      <th>city</th>\n","      <th>state</th>\n","      <th>type</th>\n","      <th>cluster</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>1</td>\n","      <td>Quito</td>\n","      <td>Pichincha</td>\n","      <td>D</td>\n","      <td>13</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>2</td>\n","      <td>Quito</td>\n","      <td>Pichincha</td>\n","      <td>D</td>\n","      <td>13</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>3</td>\n","      <td>Quito</td>\n","      <td>Pichincha</td>\n","      <td>D</td>\n","      <td>8</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   store_nbr   city      state type  cluster\n","0          1  Quito  Pichincha    D       13\n","1          2  Quito  Pichincha    D       13\n","2          3  Quito  Pichincha    D        8"]},"metadata":{},"output_type":"display_data"},{"data":{"text/plain":["'The Stores data set consists of 5 different features which for 54 samples.'"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Info about the index dtype and columns, non-null values and memory usage.**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["<class 'pandas.core.frame.DataFrame'>\n","RangeIndex: 54 entries, 0 to 53\n","Data columns (total 5 columns):\n"," #   Column     Non-Null Count  Dtype \n","---  ------     --------------  ----- \n"," 0   store_nbr  54 non-null     int64 \n"," 1   city       54 non-null     object\n"," 2   state      54 non-null     object\n"," 3   type       54 non-null     object\n"," 4   cluster    54 non-null     int64 \n","dtypes: int64(2), object(3)\n","memory usage: 2.2+ KB\n"]},{"data":{"text/plain":["None"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Count na values**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/plain":["store_nbr    0\n","city         0\n","state        0\n","type         0\n","cluster      0\n","dtype: int64"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Show the statistic report of the numeric features of the dataset**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>count</th>\n","      <th>mean</th>\n","      <th>std</th>\n","      <th>min</th>\n","      <th>25%</th>\n","      <th>50%</th>\n","      <th>75%</th>\n","      <th>max</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>store_nbr</th>\n","      <td>54.0</td>\n","      <td>27.500000</td>\n","      <td>15.732133</td>\n","      <td>1.0</td>\n","      <td>14.25</td>\n","      <td>27.5</td>\n","      <td>40.75</td>\n","      <td>54.0</td>\n","    </tr>\n","    <tr>\n","      <th>cluster</th>\n","      <td>54.0</td>\n","      <td>8.481481</td>\n","      <td>4.693395</td>\n","      <td>1.0</td>\n","      <td>4.00</td>\n","      <td>8.5</td>\n","      <td>13.00</td>\n","      <td>17.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["           count       mean        std  min    25%   50%    75%   max\n","store_nbr   54.0  27.500000  15.732133  1.0  14.25  27.5  40.75  54.0\n","cluster     54.0   8.481481   4.693395  1.0   4.00   8.5  13.00  17.0"]},"metadata":{},"output_type":"display_data"}],"source":["basicEDA(dfStores, \"Stores\")"]},{"cell_type":"markdown","metadata":{"_cell_guid":"a49ea9f4-8831-4c89-83a6-466952ea3558","_uuid":"631d3731-a7cb-4d01-b5e8-6960f7176069","trusted":true},"source":["We have no missing data here!"]},{"cell_type":"code","execution_count":17,"metadata":{"_cell_guid":"112c1568-795c-4845-8784-4da71bb6012f","_uuid":"ebd92626-66d5-4dca-a66c-5255f0559072","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:02:55.647922Z","iopub.status.busy":"2023-02-20T18:02:55.647699Z","iopub.status.idle":"2023-02-20T18:02:55.660830Z","shell.execute_reply":"2023-02-20T18:02:55.659270Z","shell.execute_reply.started":"2023-02-20T18:02:55.647902Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/markdown":["Distint values in:"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["city -> ['Ambato', 'Babahoyo', 'Cayambe', 'Cuenca', 'Daule', 'El Carmen', 'Esmeraldas', 'Guaranda', 'Guayaquil', 'Ibarra', 'Latacunga', 'Libertad', 'Loja', 'Machala', 'Manta', 'Playas', 'Puyo', 'Quevedo', 'Quito', 'Riobamba', 'Salinas', 'Santo Domingo']"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["state -> ['Azuay', 'Bolivar', 'Chimborazo', 'Cotopaxi', 'El Oro', 'Esmeraldas', 'Guayas', 'Imbabura', 'Loja', 'Los Rios', 'Manabi', 'Pastaza', 'Pichincha', 'Santa Elena', 'Santo Domingo de los Tsachilas', 'Tungurahua']"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["type -> ['A', 'B', 'C', 'D', 'E']"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"}],"source":["getDistinctValues(dfStores, [\"city\", \"state\", \"type\"])"]},{"cell_type":"markdown","metadata":{"_cell_guid":"e07f0695-a4e9-4726-90ce-4e2ee2c61006","_uuid":"c6682fe3-3d61-45f2-8bfd-58c0ac5fd158","trusted":true},"source":["### Profiling Transactions <a name=\"profilingtransaction\"></a>"]},{"cell_type":"code","execution_count":18,"metadata":{"_cell_guid":"b532b810-a37c-4834-8ab9-c64a5b6e27ba","_uuid":"e83fb981-eb62-44e0-b8fc-927ff4725460","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:02:55.662138Z","iopub.status.busy":"2023-02-20T18:02:55.661858Z","iopub.status.idle":"2023-02-20T18:02:55.717007Z","shell.execute_reply":"2023-02-20T18:02:55.716046Z","shell.execute_reply.started":"2023-02-20T18:02:55.662115Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/markdown":["**Just first five rows**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>date</th>\n","      <th>store_nbr</th>\n","      <th>transactions</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>2013-01-01</td>\n","      <td>25</td>\n","      <td>770</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>2013-01-02</td>\n","      <td>1</td>\n","      <td>2111</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2013-01-02</td>\n","      <td>2</td>\n","      <td>2358</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["        date  store_nbr  transactions\n","0 2013-01-01         25           770\n","1 2013-01-02          1          2111\n","2 2013-01-02          2          2358"]},"metadata":{},"output_type":"display_data"},{"data":{"text/plain":["'The Trasactions data set consists of 3 different features which for 83488 samples.'"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Info about the index dtype and columns, non-null values and memory usage.**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["<class 'pandas.core.frame.DataFrame'>\n","RangeIndex: 83488 entries, 0 to 83487\n","Data columns (total 3 columns):\n"," #   Column        Non-Null Count  Dtype         \n","---  ------        --------------  -----         \n"," 0   date          83488 non-null  datetime64[ns]\n"," 1   store_nbr     83488 non-null  int64         \n"," 2   transactions  83488 non-null  int64         \n","dtypes: datetime64[ns](1), int64(2)\n","memory usage: 1.9 MB\n"]},{"data":{"text/plain":["None"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Count na values**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/plain":["date            0\n","store_nbr       0\n","transactions    0\n","dtype: int64"]},"metadata":{},"output_type":"display_data"},{"data":{"text/markdown":["**Show the statistic report of the numeric features of the dataset**"],"text/plain":["<IPython.core.display.Markdown object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>count</th>\n","      <th>mean</th>\n","      <th>min</th>\n","      <th>25%</th>\n","      <th>50%</th>\n","      <th>75%</th>\n","      <th>max</th>\n","      <th>std</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>date</th>\n","      <td>83488</td>\n","      <td>2015-05-20 16:07:40.866227968</td>\n","      <td>2013-01-01 00:00:00</td>\n","      <td>2014-03-27 00:00:00</td>\n","      <td>2015-06-08 00:00:00</td>\n","      <td>2016-07-14 06:00:00</td>\n","      <td>2017-08-15 00:00:00</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>store_nbr</th>\n","      <td>83488</td>\n","      <td>26.9392</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>27</td>\n","      <td>40</td>\n","      <td>54</td>\n","      <td>15.6082</td>\n","    </tr>\n","    <tr>\n","      <th>transactions</th>\n","      <td>83488</td>\n","      <td>1694.6</td>\n","      <td>5</td>\n","      <td>1046</td>\n","      <td>1393</td>\n","      <td>2079</td>\n","      <td>8359</td>\n","      <td>963.287</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["              count                           mean                  min  \\\n","date          83488  2015-05-20 16:07:40.866227968  2013-01-01 00:00:00   \n","store_nbr     83488                        26.9392                    1   \n","transactions  83488                         1694.6                    5   \n","\n","                              25%                  50%                  75%  \\\n","date          2014-03-27 00:00:00  2015-06-08 00:00:00  2016-07-14 06:00:00   \n","store_nbr                      13                   27                   40   \n","transactions                 1046                 1393                 2079   \n","\n","                              max      std  \n","date          2017-08-15 00:00:00      NaN  \n","store_nbr                      54  15.6082  \n","transactions                 8359  963.287  "]},"metadata":{},"output_type":"display_data"}],"source":["basicEDA(dfTransactions, \"Trasactions\")"]},{"cell_type":"markdown","metadata":{"_cell_guid":"e90c4f52-21a6-48a9-8665-19baa38b15b0","_uuid":"65067e86-bbd9-4784-981c-6bda1f2e4b77","trusted":true},"source":["We have no missing data here!"]},{"cell_type":"markdown","metadata":{"_cell_guid":"1c32828a-0407-4f6c-9127-96ddd6d090b4","_uuid":"54084de4-5d42-489d-a07b-1e5513b91971","trusted":true},"source":["# Handling train dataset <a name=\"handlingtraindataset\"></a>"]},{"cell_type":"markdown","metadata":{},"source":["Let's merge combines multiple datasets into a single DataFrame called dfTrainMerged and dfTestMerged!"]},{"cell_type":"code","execution_count":19,"metadata":{"_cell_guid":"c6474fcb-9590-4d7c-894e-40e78ad0d07a","_uuid":"6310aa88-4bfa-4d53-91c5-d1d0708b9c0a","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:02:55.718721Z","iopub.status.busy":"2023-02-20T18:02:55.718452Z","iopub.status.idle":"2023-02-20T18:03:01.920442Z","shell.execute_reply":"2023-02-20T18:03:01.919382Z","shell.execute_reply.started":"2023-02-20T18:02:55.718699Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["dfTrainMerged = dfTrain.merge(dfStores, on = 'store_nbr', how='left') \\\n","                     .merge(dfTransactions, left_on=[\"date\", \"store_nbr\"], right_on=[\"date\", \"store_nbr\"], how=\"left\") \\\n","                     .merge(dfHolidays, left_on=\"date\", right_on=\"date\", how=\"left\") \\\n","                     .rename(columns={\"type_x\": \"type_Store\"}) \\\n","                     .rename(columns={\"type_y\": \"type_Holiday\"}) \\\n","                     .drop_duplicates(subset=\"id\").copy() \\\n","                     .merge(dfOilPrices, left_on=\"date\", right_on=\"date\", how=\"left\").copy()"]},{"cell_type":"code","execution_count":20,"metadata":{"execution":{"iopub.execute_input":"2023-02-20T18:03:01.922130Z","iopub.status.busy":"2023-02-20T18:03:01.921845Z","iopub.status.idle":"2023-02-20T18:03:01.998164Z","shell.execute_reply":"2023-02-20T18:03:01.997236Z","shell.execute_reply.started":"2023-02-20T18:03:01.922107Z"},"trusted":true},"outputs":[],"source":["dfTestMerged = dfTest.merge(dfStores, on = 'store_nbr', how='left') \\\n","                     .merge(dfTransactions, left_on=[\"date\", \"store_nbr\"], right_on=[\"date\", \"store_nbr\"], how=\"left\") \\\n","                     .merge(dfHolidays, left_on=\"date\", right_on=\"date\", how=\"left\") \\\n","                     .rename(columns={\"type_x\": \"type_Store\"}) \\\n","                     .rename(columns={\"type_y\": \"type_Holiday\"}) \\\n","                     .drop_duplicates(subset=\"id\").copy() \\\n","                     .merge(dfOilPrices, left_on=\"date\", right_on=\"date\", how=\"left\").copy()"]},{"cell_type":"markdown","metadata":{},"source":["Clean up memory!"]},{"cell_type":"code","execution_count":21,"metadata":{"_cell_guid":"e71fd782-8627-48be-aaac-70c6b0afccac","_uuid":"bf87d07b-bd34-47b2-9ae3-c68f5a8abb60","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:03:02.000458Z","iopub.status.busy":"2023-02-20T18:03:01.999371Z","iopub.status.idle":"2023-02-20T18:03:02.161698Z","shell.execute_reply":"2023-02-20T18:03:02.160775Z","shell.execute_reply.started":"2023-02-20T18:03:02.000425Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/plain":["0"]},"execution_count":21,"metadata":{},"output_type":"execute_result"}],"source":["del dfHolidays \n","del dfOilPrices \n","del dfStores \n","del dfTransactions\n","del dfTrain\n","del dfTest\n","gc.collect()"]},{"cell_type":"markdown","metadata":{"_cell_guid":"849db587-f95e-4c5b-b105-8b24461b5dc0","_uuid":"067e6fe5-3ace-4981-aafe-b06ffce0ec5d","trusted":true},"source":["## Feature engineering <a name=\"11111\"></a>"]},{"cell_type":"markdown","metadata":{"_cell_guid":"e864efd8-42a5-49c1-ad0d-0d35f0cdf545","_uuid":"46ac5d47-e13a-4a10-a1b2-e0b54cb81304","trusted":true},"source":["We conduct a missing values analysis to see what is the percentage of missing per column."]},{"cell_type":"code","execution_count":22,"metadata":{"_cell_guid":"f0849596-04a2-409d-a312-6d9412a4eaa9","_uuid":"917057f9-e60f-4506-8bd0-4c9655caa197","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:03:02.163314Z","iopub.status.busy":"2023-02-20T18:03:02.162628Z","iopub.status.idle":"2023-02-20T18:03:03.574778Z","shell.execute_reply":"2023-02-20T18:03:03.573862Z","shell.execute_reply.started":"2023-02-20T18:03:02.163289Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/plain":["\"The columns with missing data are: ['transactions', 'type_Holiday', 'locale', 'locale_name', 'description', 'transferred', 'dcoilwtico']\""]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["Percent of missing 'transactions' records is 8.19 % (245784 values on 3000888 total)\n","Percent of missing 'type_Holiday' records is 85.036 % (2551824 values on 3000888 total)\n","Percent of missing 'locale' records is 85.036 % (2551824 values on 3000888 total)\n","Percent of missing 'locale_name' records is 85.036 % (2551824 values on 3000888 total)\n","Percent of missing 'description' records is 85.036 % (2551824 values on 3000888 total)\n","Percent of missing 'transferred' records is 85.036 % (2551824 values on 3000888 total)\n","Percent of missing 'dcoilwtico' records is 28.563 % (857142 values on 3000888 total)\n"]}],"source":["showMissingData(dfTrainMerged)"]},{"cell_type":"code","execution_count":23,"metadata":{"execution":{"iopub.execute_input":"2023-02-20T18:03:03.576072Z","iopub.status.busy":"2023-02-20T18:03:03.575809Z","iopub.status.idle":"2023-02-20T18:03:03.601998Z","shell.execute_reply":"2023-02-20T18:03:03.600715Z","shell.execute_reply.started":"2023-02-20T18:03:03.576049Z"},"trusted":true},"outputs":[{"data":{"text/plain":["\"The columns with missing data are: ['transactions', 'type_Holiday', 'locale', 'locale_name', 'description', 'transferred', 'dcoilwtico']\""]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["Percent of missing 'transactions' records is 100.0 % (28512 values on 28512 total)\n","Percent of missing 'type_Holiday' records is 93.75 % (26730 values on 28512 total)\n","Percent of missing 'locale' records is 93.75 % (26730 values on 28512 total)\n","Percent of missing 'locale_name' records is 93.75 % (26730 values on 28512 total)\n","Percent of missing 'description' records is 93.75 % (26730 values on 28512 total)\n","Percent of missing 'transferred' records is 93.75 % (26730 values on 28512 total)\n","Percent of missing 'dcoilwtico' records is 25.0 % (7128 values on 28512 total)\n"]}],"source":["showMissingData(dfTestMerged)"]},{"cell_type":"markdown","metadata":{},"source":["First of all, we can get rid some feature, as showed below:"]},{"cell_type":"code","execution_count":24,"metadata":{"_cell_guid":"95e7e253-5b98-4d6d-9bd3-a1cbded3c02c","_uuid":"96528438-fab9-403a-a84f-61db84b30865","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:03:03.603931Z","iopub.status.busy":"2023-02-20T18:03:03.603543Z","iopub.status.idle":"2023-02-20T18:03:04.697441Z","shell.execute_reply":"2023-02-20T18:03:04.696461Z","shell.execute_reply.started":"2023-02-20T18:03:03.603894Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>id</th>\n","      <th>date</th>\n","      <th>store_nbr</th>\n","      <th>family</th>\n","      <th>sales</th>\n","      <th>onpromotion</th>\n","      <th>city</th>\n","      <th>state</th>\n","      <th>type_Store</th>\n","      <th>cluster</th>\n","      <th>transactions</th>\n","      <th>type_Holiday</th>\n","      <th>dcoilwtico</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0</td>\n","      <td>2013-01-01</td>\n","      <td>1</td>\n","      <td>AUTOMOTIVE</td>\n","      <td>0.0</td>\n","      <td>0</td>\n","      <td>Quito</td>\n","      <td>Pichincha</td>\n","      <td>D</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>Holiday</td>\n","      <td>104.896636</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1</td>\n","      <td>2013-01-01</td>\n","      <td>1</td>\n","      <td>BABY CARE</td>\n","      <td>0.0</td>\n","      <td>0</td>\n","      <td>Quito</td>\n","      <td>Pichincha</td>\n","      <td>D</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>Holiday</td>\n","      <td>104.896636</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2</td>\n","      <td>2013-01-01</td>\n","      <td>1</td>\n","      <td>BEAUTY</td>\n","      <td>0.0</td>\n","      <td>0</td>\n","      <td>Quito</td>\n","      <td>Pichincha</td>\n","      <td>D</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>Holiday</td>\n","      <td>104.896636</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3</td>\n","      <td>2013-01-01</td>\n","      <td>1</td>\n","      <td>BEVERAGES</td>\n","      <td>0.0</td>\n","      <td>0</td>\n","      <td>Quito</td>\n","      <td>Pichincha</td>\n","      <td>D</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>Holiday</td>\n","      <td>104.896636</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>4</td>\n","      <td>2013-01-01</td>\n","      <td>1</td>\n","      <td>BOOKS</td>\n","      <td>0.0</td>\n","      <td>0</td>\n","      <td>Quito</td>\n","      <td>Pichincha</td>\n","      <td>D</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>Holiday</td>\n","      <td>104.896636</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   id       date  store_nbr      family  sales  onpromotion   city      state  \\\n","0   0 2013-01-01          1  AUTOMOTIVE    0.0            0  Quito  Pichincha   \n","1   1 2013-01-01          1   BABY CARE    0.0            0  Quito  Pichincha   \n","2   2 2013-01-01          1      BEAUTY    0.0            0  Quito  Pichincha   \n","3   3 2013-01-01          1   BEVERAGES    0.0            0  Quito  Pichincha   \n","4   4 2013-01-01          1       BOOKS    0.0            0  Quito  Pichincha   \n","\n","  type_Store  cluster  transactions type_Holiday  dcoilwtico  \n","0          D       13           NaN      Holiday  104.896636  \n","1          D       13           NaN      Holiday  104.896636  \n","2          D       13           NaN      Holiday  104.896636  \n","3          D       13           NaN      Holiday  104.896636  \n","4          D       13           NaN      Holiday  104.896636  "]},"execution_count":24,"metadata":{},"output_type":"execute_result"}],"source":["delColumn(dfTrainMerged, \"locale\")\n","delColumn(dfTrainMerged, \"locale_name\")\n","delColumn(dfTrainMerged, \"description\")\n","delColumn(dfTrainMerged, \"transferred\")\n","gc.collect()\n","dfTrainMerged.head()"]},{"cell_type":"code","execution_count":25,"metadata":{"execution":{"iopub.execute_input":"2023-02-20T18:03:04.701259Z","iopub.status.busy":"2023-02-20T18:03:04.700241Z","iopub.status.idle":"2023-02-20T18:03:04.854823Z","shell.execute_reply":"2023-02-20T18:03:04.854007Z","shell.execute_reply.started":"2023-02-20T18:03:04.701224Z"},"trusted":true},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>id</th>\n","      <th>date</th>\n","      <th>store_nbr</th>\n","      <th>family</th>\n","      <th>onpromotion</th>\n","      <th>city</th>\n","      <th>state</th>\n","      <th>type_Store</th>\n","      <th>cluster</th>\n","      <th>transactions</th>\n","      <th>type_Holiday</th>\n","      <th>dcoilwtico</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>3000888</td>\n","      <td>2017-08-16</td>\n","      <td>1</td>\n","      <td>AUTOMOTIVE</td>\n","      <td>0</td>\n","      <td>Quito</td>\n","      <td>Pichincha</td>\n","      <td>D</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>46.8</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>3000889</td>\n","      <td>2017-08-16</td>\n","      <td>1</td>\n","      <td>BABY CARE</td>\n","      <td>0</td>\n","      <td>Quito</td>\n","      <td>Pichincha</td>\n","      <td>D</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>46.8</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>3000890</td>\n","      <td>2017-08-16</td>\n","      <td>1</td>\n","      <td>BEAUTY</td>\n","      <td>2</td>\n","      <td>Quito</td>\n","      <td>Pichincha</td>\n","      <td>D</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>46.8</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3000891</td>\n","      <td>2017-08-16</td>\n","      <td>1</td>\n","      <td>BEVERAGES</td>\n","      <td>20</td>\n","      <td>Quito</td>\n","      <td>Pichincha</td>\n","      <td>D</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>46.8</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>3000892</td>\n","      <td>2017-08-16</td>\n","      <td>1</td>\n","      <td>BOOKS</td>\n","      <td>0</td>\n","      <td>Quito</td>\n","      <td>Pichincha</td>\n","      <td>D</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>46.8</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["        id       date  store_nbr      family  onpromotion   city      state  \\\n","0  3000888 2017-08-16          1  AUTOMOTIVE            0  Quito  Pichincha   \n","1  3000889 2017-08-16          1   BABY CARE            0  Quito  Pichincha   \n","2  3000890 2017-08-16          1      BEAUTY            2  Quito  Pichincha   \n","3  3000891 2017-08-16          1   BEVERAGES           20  Quito  Pichincha   \n","4  3000892 2017-08-16          1       BOOKS            0  Quito  Pichincha   \n","\n","  type_Store  cluster  transactions type_Holiday  dcoilwtico  \n","0          D       13           NaN          NaN        46.8  \n","1          D       13           NaN          NaN        46.8  \n","2          D       13           NaN          NaN        46.8  \n","3          D       13           NaN          NaN        46.8  \n","4          D       13           NaN          NaN        46.8  "]},"execution_count":25,"metadata":{},"output_type":"execute_result"}],"source":["delColumn(dfTestMerged, \"locale\")\n","delColumn(dfTestMerged, \"locale_name\")\n","delColumn(dfTestMerged, \"description\")\n","delColumn(dfTestMerged, \"transferred\")\n","gc.collect()\n","dfTestMerged.head()"]},{"cell_type":"markdown","metadata":{},"source":["We use fillTypeHolidayColumn() function which iterates over each row in the DataFrame, and for any rows where the \"type_Holiday\" value is missing (i.e. NaN), it determines whether the corresponding date is a weekday or weekend. If the date is a weekend (Saturday or Sunday), it fills in the \"type_Holiday\" column with the value \"Weekend\". Otherwise, it fills in the \"type_Holiday\" column with the value \"Work Day\". That on train and test merged data frames."]},{"cell_type":"code","execution_count":26,"metadata":{"_cell_guid":"87b06603-1f35-4524-928b-d532c63831eb","_uuid":"1bf8820c-2e26-45ac-99d9-9319d7829458","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:03:04.857089Z","iopub.status.busy":"2023-02-20T18:03:04.856075Z","iopub.status.idle":"2023-02-20T18:08:47.012393Z","shell.execute_reply":"2023-02-20T18:08:47.010955Z","shell.execute_reply.started":"2023-02-20T18:03:04.857027Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["dfTrainMerged = fillTypeHolidayColumn(dfTrainMerged)\n","dfTestMerged = fillTypeHolidayColumn(dfTestMerged)"]},{"cell_type":"markdown","metadata":{},"source":["A quick check!"]},{"cell_type":"code","execution_count":27,"metadata":{"_cell_guid":"0812886e-9ce1-455a-8857-ea18f7e87093","_uuid":"36916d02-9e74-42cc-b51c-a9d1fbab308e","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:08:47.014614Z","iopub.status.busy":"2023-02-20T18:08:47.014237Z","iopub.status.idle":"2023-02-20T18:08:47.224298Z","shell.execute_reply":"2023-02-20T18:08:47.222894Z","shell.execute_reply.started":"2023-02-20T18:08:47.014591Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/plain":["\"The 'type_Holiday' column has 0 missing data\""]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["Percent of missing 'type_Holiday' records is 0.0 % (0 values on 3000888 total)\n"]}],"source":["showMissingDataPerCol(dfTrainMerged, \"type_Holiday\")"]},{"cell_type":"code","execution_count":28,"metadata":{"execution":{"iopub.execute_input":"2023-02-20T18:08:47.226677Z","iopub.status.busy":"2023-02-20T18:08:47.226329Z","iopub.status.idle":"2023-02-20T18:08:47.238009Z","shell.execute_reply":"2023-02-20T18:08:47.236762Z","shell.execute_reply.started":"2023-02-20T18:08:47.226654Z"},"trusted":true},"outputs":[{"data":{"text/plain":["\"The 'type_Holiday' column has 0 missing data\""]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["Percent of missing 'type_Holiday' records is 0.0 % (0 values on 28512 total)\n"]}],"source":["showMissingDataPerCol(dfTestMerged, \"type_Holiday\")"]},{"cell_type":"markdown","metadata":{"_cell_guid":"b8d7cc10-8123-4f55-b9c1-bac703bc2ad4","_uuid":"d4d4b37e-4d2f-4445-9340-170f831ac241","trusted":true},"source":["Let's handle categorical data by using \"Binary Encoding\" that is a technique used to encode categorical features as binary digits (0s and 1s). Each distinct value in the categorical feature is represented by a unique binary code, which is then split into multiple binary columns."]},{"cell_type":"code","execution_count":29,"metadata":{"_cell_guid":"8b8539e0-766b-44b1-8cb5-a39a1dab531e","_uuid":"a12aa33c-49bc-40a8-802c-72de18a89c7a","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:08:47.241593Z","iopub.status.busy":"2023-02-20T18:08:47.240103Z","iopub.status.idle":"2023-02-20T18:09:01.385180Z","shell.execute_reply":"2023-02-20T18:09:01.384052Z","shell.execute_reply.started":"2023-02-20T18:08:47.241552Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>id</th>\n","      <th>date</th>\n","      <th>store_nbr</th>\n","      <th>family_0</th>\n","      <th>family_1</th>\n","      <th>family_2</th>\n","      <th>family_3</th>\n","      <th>family_4</th>\n","      <th>family_5</th>\n","      <th>sales</th>\n","      <th>...</th>\n","      <th>state_4</th>\n","      <th>type_Store_0</th>\n","      <th>type_Store_1</th>\n","      <th>type_Store_2</th>\n","      <th>cluster</th>\n","      <th>transactions</th>\n","      <th>type_Holiday_0</th>\n","      <th>type_Holiday_1</th>\n","      <th>type_Holiday_2</th>\n","      <th>dcoilwtico</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0</td>\n","      <td>2013-01-01</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0.0</td>\n","      <td>...</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>104.896636</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1</td>\n","      <td>2013-01-01</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0.0</td>\n","      <td>...</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>104.896636</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2</td>\n","      <td>2013-01-01</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>1</td>\n","      <td>0.0</td>\n","      <td>...</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>104.896636</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3</td>\n","      <td>2013-01-01</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0.0</td>\n","      <td>...</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>104.896636</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>4</td>\n","      <td>2013-01-01</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0.0</td>\n","      <td>...</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>104.896636</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>5 rows × 30 columns</p>\n","</div>"],"text/plain":["   id       date  store_nbr  family_0  family_1  family_2  family_3  family_4  \\\n","0   0 2013-01-01          1         0         0         0         0         0   \n","1   1 2013-01-01          1         0         0         0         0         1   \n","2   2 2013-01-01          1         0         0         0         0         1   \n","3   3 2013-01-01          1         0         0         0         1         0   \n","4   4 2013-01-01          1         0         0         0         1         0   \n","\n","   family_5  sales  ...  state_4  type_Store_0  type_Store_1  type_Store_2  \\\n","0         1    0.0  ...        1             0             0             1   \n","1         0    0.0  ...        1             0             0             1   \n","2         1    0.0  ...        1             0             0             1   \n","3         0    0.0  ...        1             0             0             1   \n","4         1    0.0  ...        1             0             0             1   \n","\n","   cluster  transactions  type_Holiday_0  type_Holiday_1  type_Holiday_2  \\\n","0       13           NaN               0               0               1   \n","1       13           NaN               0               0               1   \n","2       13           NaN               0               0               1   \n","3       13           NaN               0               0               1   \n","4       13           NaN               0               0               1   \n","\n","   dcoilwtico  \n","0  104.896636  \n","1  104.896636  \n","2  104.896636  \n","3  104.896636  \n","4  104.896636  \n","\n","[5 rows x 30 columns]"]},"execution_count":29,"metadata":{},"output_type":"execute_result"}],"source":["dfTrainMerged = BinEncCategory(dfTrainMerged, [\"type_Holiday\",\"family\", \"city\", \"state\", \"type_Store\"])\n","dfTrainMerged.head()"]},{"cell_type":"code","execution_count":30,"metadata":{"execution":{"iopub.execute_input":"2023-02-20T18:09:01.387218Z","iopub.status.busy":"2023-02-20T18:09:01.386886Z","iopub.status.idle":"2023-02-20T18:09:01.614852Z","shell.execute_reply":"2023-02-20T18:09:01.613758Z","shell.execute_reply.started":"2023-02-20T18:09:01.387192Z"},"trusted":true},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>id</th>\n","      <th>date</th>\n","      <th>store_nbr</th>\n","      <th>family_0</th>\n","      <th>family_1</th>\n","      <th>family_2</th>\n","      <th>family_3</th>\n","      <th>family_4</th>\n","      <th>family_5</th>\n","      <th>onpromotion</th>\n","      <th>...</th>\n","      <th>state_3</th>\n","      <th>state_4</th>\n","      <th>type_Store_0</th>\n","      <th>type_Store_1</th>\n","      <th>type_Store_2</th>\n","      <th>cluster</th>\n","      <th>transactions</th>\n","      <th>type_Holiday_0</th>\n","      <th>type_Holiday_1</th>\n","      <th>dcoilwtico</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>3000888</td>\n","      <td>2017-08-16</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>46.8</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>3000889</td>\n","      <td>2017-08-16</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>46.8</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>3000890</td>\n","      <td>2017-08-16</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>1</td>\n","      <td>2</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>46.8</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3000891</td>\n","      <td>2017-08-16</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>20</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>46.8</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>3000892</td>\n","      <td>2017-08-16</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>NaN</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>46.8</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>5 rows × 28 columns</p>\n","</div>"],"text/plain":["        id       date  store_nbr  family_0  family_1  family_2  family_3  \\\n","0  3000888 2017-08-16          1         0         0         0         0   \n","1  3000889 2017-08-16          1         0         0         0         0   \n","2  3000890 2017-08-16          1         0         0         0         0   \n","3  3000891 2017-08-16          1         0         0         0         1   \n","4  3000892 2017-08-16          1         0         0         0         1   \n","\n","   family_4  family_5  onpromotion  ...  state_3  state_4  type_Store_0  \\\n","0         0         1            0  ...        0        1             0   \n","1         1         0            0  ...        0        1             0   \n","2         1         1            2  ...        0        1             0   \n","3         0         0           20  ...        0        1             0   \n","4         0         1            0  ...        0        1             0   \n","\n","   type_Store_1  type_Store_2  cluster  transactions  type_Holiday_0  \\\n","0             0             1       13           NaN               0   \n","1             0             1       13           NaN               0   \n","2             0             1       13           NaN               0   \n","3             0             1       13           NaN               0   \n","4             0             1       13           NaN               0   \n","\n","   type_Holiday_1  dcoilwtico  \n","0               1        46.8  \n","1               1        46.8  \n","2               1        46.8  \n","3               1        46.8  \n","4               1        46.8  \n","\n","[5 rows x 28 columns]"]},"execution_count":30,"metadata":{},"output_type":"execute_result"}],"source":["dfTestMerged = BinEncCategory(dfTestMerged, [\"type_Holiday\",\"family\", \"city\", \"state\", \"type_Store\"])\n","dfTestMerged.head()"]},{"cell_type":"markdown","metadata":{},"source":["Now we can delete labeled features..."]},{"cell_type":"code","execution_count":31,"metadata":{"_cell_guid":"d464be39-555e-41c0-ada0-bf009253015a","_uuid":"6826ca03-7fc1-47ac-a568-9429e0794fb3","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:09:01.616609Z","iopub.status.busy":"2023-02-20T18:09:01.616241Z","iopub.status.idle":"2023-02-20T18:09:01.786171Z","shell.execute_reply":"2023-02-20T18:09:01.785230Z","shell.execute_reply.started":"2023-02-20T18:09:01.616576Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/plain":["0"]},"execution_count":31,"metadata":{},"output_type":"execute_result"}],"source":["delColumn(dfTrainMerged, \"type_Holiday\")\n","delColumn(dfTrainMerged, \"family\")\n","delColumn(dfTrainMerged, \"city\")\n","delColumn(dfTrainMerged, \"state\")\n","delColumn(dfTrainMerged, \"type_Store\")\n","\n","delColumn(dfTestMerged, \"type_Holiday\")\n","delColumn(dfTestMerged, \"family\")\n","delColumn(dfTestMerged, \"city\")\n","delColumn(dfTestMerged, \"state\")\n","delColumn(dfTestMerged, \"type_Store\")\n","gc.collect()"]},{"cell_type":"markdown","metadata":{},"source":["It is better change datas in float32 type."]},{"cell_type":"code","execution_count":32,"metadata":{"_cell_guid":"cc0cc733-3dde-43d9-af44-6bfb4612682e","_uuid":"2bdcf8b2-2a7a-42d4-ac00-7893a65ca7fa","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:09:01.788302Z","iopub.status.busy":"2023-02-20T18:09:01.787551Z","iopub.status.idle":"2023-02-20T18:09:01.803055Z","shell.execute_reply":"2023-02-20T18:09:01.801523Z","shell.execute_reply.started":"2023-02-20T18:09:01.788268Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["dfTrainMerged[(\"date\")] = dfTrainMerged[(\"date\")].values.astype(\"float32\")\n","dfTestMerged[(\"date\")] = dfTestMerged[(\"date\")].values.astype(\"float32\")"]},{"cell_type":"markdown","metadata":{},"source":["Now we want to fill NaN transactions values. To do that we can simply get the mean value and use it."]},{"cell_type":"code","execution_count":33,"metadata":{"execution":{"iopub.execute_input":"2023-02-20T18:09:01.805491Z","iopub.status.busy":"2023-02-20T18:09:01.805072Z","iopub.status.idle":"2023-02-20T18:09:01.827847Z","shell.execute_reply":"2023-02-20T18:09:01.826784Z","shell.execute_reply.started":"2023-02-20T18:09:01.805455Z"},"trusted":true},"outputs":[{"data":{"text/plain":["1694.6021583940208"]},"execution_count":33,"metadata":{},"output_type":"execute_result"}],"source":["transactions_mean = dfTrainMerged[\"transactions\"].mean()\n","transactions_mean"]},{"cell_type":"code","execution_count":34,"metadata":{"_cell_guid":"62d7c1b9-5dd4-4222-bb7c-58cf958f28f1","_uuid":"49d782ef-a453-4f70-9ecd-1697ba4afd41","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:09:01.829888Z","iopub.status.busy":"2023-02-20T18:09:01.829366Z","iopub.status.idle":"2023-02-20T18:09:01.853218Z","shell.execute_reply":"2023-02-20T18:09:01.852180Z","shell.execute_reply.started":"2023-02-20T18:09:01.829863Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>id</th>\n","      <th>date</th>\n","      <th>store_nbr</th>\n","      <th>family_0</th>\n","      <th>family_1</th>\n","      <th>family_2</th>\n","      <th>family_3</th>\n","      <th>family_4</th>\n","      <th>family_5</th>\n","      <th>sales</th>\n","      <th>...</th>\n","      <th>state_4</th>\n","      <th>type_Store_0</th>\n","      <th>type_Store_1</th>\n","      <th>type_Store_2</th>\n","      <th>cluster</th>\n","      <th>transactions</th>\n","      <th>type_Holiday_0</th>\n","      <th>type_Holiday_1</th>\n","      <th>type_Holiday_2</th>\n","      <th>dcoilwtico</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0</td>\n","      <td>1.356998e+18</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0.0</td>\n","      <td>...</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>1694.602158</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>104.896636</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1</td>\n","      <td>1.356998e+18</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0.0</td>\n","      <td>...</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>1694.602158</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>104.896636</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2</td>\n","      <td>1.356998e+18</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>1</td>\n","      <td>0.0</td>\n","      <td>...</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>1694.602158</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>104.896636</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3</td>\n","      <td>1.356998e+18</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0.0</td>\n","      <td>...</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>1694.602158</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>104.896636</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>4</td>\n","      <td>1.356998e+18</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0.0</td>\n","      <td>...</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>1694.602158</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>104.896636</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>5 rows × 30 columns</p>\n","</div>"],"text/plain":["   id          date  store_nbr  family_0  family_1  family_2  family_3  \\\n","0   0  1.356998e+18          1         0         0         0         0   \n","1   1  1.356998e+18          1         0         0         0         0   \n","2   2  1.356998e+18          1         0         0         0         0   \n","3   3  1.356998e+18          1         0         0         0         1   \n","4   4  1.356998e+18          1         0         0         0         1   \n","\n","   family_4  family_5  sales  ...  state_4  type_Store_0  type_Store_1  \\\n","0         0         1    0.0  ...        1             0             0   \n","1         1         0    0.0  ...        1             0             0   \n","2         1         1    0.0  ...        1             0             0   \n","3         0         0    0.0  ...        1             0             0   \n","4         0         1    0.0  ...        1             0             0   \n","\n","   type_Store_2  cluster  transactions  type_Holiday_0  type_Holiday_1  \\\n","0             1       13   1694.602158               0               0   \n","1             1       13   1694.602158               0               0   \n","2             1       13   1694.602158               0               0   \n","3             1       13   1694.602158               0               0   \n","4             1       13   1694.602158               0               0   \n","\n","   type_Holiday_2  dcoilwtico  \n","0               1  104.896636  \n","1               1  104.896636  \n","2               1  104.896636  \n","3               1  104.896636  \n","4               1  104.896636  \n","\n","[5 rows x 30 columns]"]},"execution_count":34,"metadata":{},"output_type":"execute_result"}],"source":["dfTrainMerged[\"transactions\"].fillna(transactions_mean,inplace=True)\n","dfTrainMerged.head()"]},{"cell_type":"markdown","metadata":{},"source":["Unfortunately, we have no data in transactions feature."]},{"cell_type":"code","execution_count":35,"metadata":{"execution":{"iopub.execute_input":"2023-02-20T18:09:01.854542Z","iopub.status.busy":"2023-02-20T18:09:01.854288Z","iopub.status.idle":"2023-02-20T18:09:01.862441Z","shell.execute_reply":"2023-02-20T18:09:01.861111Z","shell.execute_reply.started":"2023-02-20T18:09:01.854517Z"},"trusted":true},"outputs":[{"data":{"text/plain":["nan"]},"execution_count":35,"metadata":{},"output_type":"execute_result"}],"source":["test_transactions_mean = dfTestMerged[\"transactions\"].mean()\n","test_transactions_mean"]},{"cell_type":"markdown","metadata":{},"source":["So we use the train transaction mean value..."]},{"cell_type":"code","execution_count":36,"metadata":{"execution":{"iopub.execute_input":"2023-02-20T18:09:01.864466Z","iopub.status.busy":"2023-02-20T18:09:01.864035Z","iopub.status.idle":"2023-02-20T18:09:01.889800Z","shell.execute_reply":"2023-02-20T18:09:01.888916Z","shell.execute_reply.started":"2023-02-20T18:09:01.864431Z"},"trusted":true},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>id</th>\n","      <th>date</th>\n","      <th>store_nbr</th>\n","      <th>family_0</th>\n","      <th>family_1</th>\n","      <th>family_2</th>\n","      <th>family_3</th>\n","      <th>family_4</th>\n","      <th>family_5</th>\n","      <th>onpromotion</th>\n","      <th>...</th>\n","      <th>state_3</th>\n","      <th>state_4</th>\n","      <th>type_Store_0</th>\n","      <th>type_Store_1</th>\n","      <th>type_Store_2</th>\n","      <th>cluster</th>\n","      <th>transactions</th>\n","      <th>type_Holiday_0</th>\n","      <th>type_Holiday_1</th>\n","      <th>dcoilwtico</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>3000888</td>\n","      <td>1.502842e+18</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>1694.602158</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>46.8</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>3000889</td>\n","      <td>1.502842e+18</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>1694.602158</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>46.8</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>3000890</td>\n","      <td>1.502842e+18</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>1</td>\n","      <td>2</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>1694.602158</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>46.8</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3000891</td>\n","      <td>1.502842e+18</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>20</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>1694.602158</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>46.8</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>3000892</td>\n","      <td>1.502842e+18</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>1694.602158</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>46.8</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>5 rows × 28 columns</p>\n","</div>"],"text/plain":["        id          date  store_nbr  family_0  family_1  family_2  family_3  \\\n","0  3000888  1.502842e+18          1         0         0         0         0   \n","1  3000889  1.502842e+18          1         0         0         0         0   \n","2  3000890  1.502842e+18          1         0         0         0         0   \n","3  3000891  1.502842e+18          1         0         0         0         1   \n","4  3000892  1.502842e+18          1         0         0         0         1   \n","\n","   family_4  family_5  onpromotion  ...  state_3  state_4  type_Store_0  \\\n","0         0         1            0  ...        0        1             0   \n","1         1         0            0  ...        0        1             0   \n","2         1         1            2  ...        0        1             0   \n","3         0         0           20  ...        0        1             0   \n","4         0         1            0  ...        0        1             0   \n","\n","   type_Store_1  type_Store_2  cluster  transactions  type_Holiday_0  \\\n","0             0             1       13   1694.602158               0   \n","1             0             1       13   1694.602158               0   \n","2             0             1       13   1694.602158               0   \n","3             0             1       13   1694.602158               0   \n","4             0             1       13   1694.602158               0   \n","\n","   type_Holiday_1  dcoilwtico  \n","0               1        46.8  \n","1               1        46.8  \n","2               1        46.8  \n","3               1        46.8  \n","4               1        46.8  \n","\n","[5 rows x 28 columns]"]},"execution_count":36,"metadata":{},"output_type":"execute_result"}],"source":["dfTestMerged[\"transactions\"].fillna(transactions_mean,inplace=True)\n","dfTestMerged.head()"]},{"cell_type":"markdown","metadata":{},"source":["Let's fill \"dcoilwtico\" missing values. We choose two ways: by Linear Regression and, on test data simply put into the mean values on test data (remember that \"Percent of missing 'dcoilwtico' records is 25.0 % (7128 values on 28512 total)\")."]},{"cell_type":"code","execution_count":37,"metadata":{"_cell_guid":"f78c73f3-20f8-431a-9175-22dd71f89e0c","_uuid":"1ef4b7f1-8600-411a-b08d-fa4bc83acc9e","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:09:01.891706Z","iopub.status.busy":"2023-02-20T18:09:01.891167Z","iopub.status.idle":"2023-02-20T18:09:04.262318Z","shell.execute_reply":"2023-02-20T18:09:04.261583Z","shell.execute_reply.started":"2023-02-20T18:09:01.891681Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["The r2 is:  1.0\n","The rmse is:  0.0\n","0\n"]}],"source":["dfTrainMerged, countMissing = handleWithLinearRegression(dfTrainMerged, \"dcoilwtico\")\n","print(countMissing)"]},{"cell_type":"code","execution_count":38,"metadata":{"execution":{"iopub.execute_input":"2023-02-20T18:09:04.263993Z","iopub.status.busy":"2023-02-20T18:09:04.263537Z","iopub.status.idle":"2023-02-20T18:09:04.287642Z","shell.execute_reply":"2023-02-20T18:09:04.286884Z","shell.execute_reply.started":"2023-02-20T18:09:04.263965Z"},"trusted":true},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>id</th>\n","      <th>date</th>\n","      <th>store_nbr</th>\n","      <th>family_0</th>\n","      <th>family_1</th>\n","      <th>family_2</th>\n","      <th>family_3</th>\n","      <th>family_4</th>\n","      <th>family_5</th>\n","      <th>onpromotion</th>\n","      <th>...</th>\n","      <th>state_3</th>\n","      <th>state_4</th>\n","      <th>type_Store_0</th>\n","      <th>type_Store_1</th>\n","      <th>type_Store_2</th>\n","      <th>cluster</th>\n","      <th>transactions</th>\n","      <th>type_Holiday_0</th>\n","      <th>type_Holiday_1</th>\n","      <th>dcoilwtico</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>3000888</td>\n","      <td>1.502842e+18</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>1694.602158</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>46.8</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>3000889</td>\n","      <td>1.502842e+18</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>1694.602158</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>46.8</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>3000890</td>\n","      <td>1.502842e+18</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>1</td>\n","      <td>2</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>1694.602158</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>46.8</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>3000891</td>\n","      <td>1.502842e+18</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>20</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>1694.602158</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>46.8</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>3000892</td>\n","      <td>1.502842e+18</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>13</td>\n","      <td>1694.602158</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>46.8</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>5 rows × 28 columns</p>\n","</div>"],"text/plain":["        id          date  store_nbr  family_0  family_1  family_2  family_3  \\\n","0  3000888  1.502842e+18          1         0         0         0         0   \n","1  3000889  1.502842e+18          1         0         0         0         0   \n","2  3000890  1.502842e+18          1         0         0         0         0   \n","3  3000891  1.502842e+18          1         0         0         0         1   \n","4  3000892  1.502842e+18          1         0         0         0         1   \n","\n","   family_4  family_5  onpromotion  ...  state_3  state_4  type_Store_0  \\\n","0         0         1            0  ...        0        1             0   \n","1         1         0            0  ...        0        1             0   \n","2         1         1            2  ...        0        1             0   \n","3         0         0           20  ...        0        1             0   \n","4         0         1            0  ...        0        1             0   \n","\n","   type_Store_1  type_Store_2  cluster  transactions  type_Holiday_0  \\\n","0             0             1       13   1694.602158               0   \n","1             0             1       13   1694.602158               0   \n","2             0             1       13   1694.602158               0   \n","3             0             1       13   1694.602158               0   \n","4             0             1       13   1694.602158               0   \n","\n","   type_Holiday_1  dcoilwtico  \n","0               1        46.8  \n","1               1        46.8  \n","2               1        46.8  \n","3               1        46.8  \n","4               1        46.8  \n","\n","[5 rows x 28 columns]"]},"execution_count":38,"metadata":{},"output_type":"execute_result"}],"source":["dfTestMerged[\"dcoilwtico\"].fillna(dfTestMerged[\"dcoilwtico\"].mean(),inplace=True)\n","dfTestMerged.head()"]},{"cell_type":"code","execution_count":39,"metadata":{"_cell_guid":"5875742f-7623-4b69-9024-9c20fb81fe30","_uuid":"74f41659-d155-4971-a512-b2783f775787","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:09:04.289215Z","iopub.status.busy":"2023-02-20T18:09:04.288796Z","iopub.status.idle":"2023-02-20T18:09:04.318234Z","shell.execute_reply":"2023-02-20T18:09:04.317524Z","shell.execute_reply.started":"2023-02-20T18:09:04.289189Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/plain":["'The columns with missing data are: []'"]},"metadata":{},"output_type":"display_data"}],"source":["showMissingData(dfTrainMerged)"]},{"cell_type":"code","execution_count":40,"metadata":{"execution":{"iopub.execute_input":"2023-02-20T18:09:04.319810Z","iopub.status.busy":"2023-02-20T18:09:04.319385Z","iopub.status.idle":"2023-02-20T18:09:04.326587Z","shell.execute_reply":"2023-02-20T18:09:04.325854Z","shell.execute_reply.started":"2023-02-20T18:09:04.319784Z"},"trusted":true},"outputs":[{"data":{"text/plain":["'The columns with missing data are: []'"]},"metadata":{},"output_type":"display_data"}],"source":["showMissingData(dfTestMerged)"]},{"cell_type":"markdown","metadata":{},"source":["At this point, we can reduce data size. In fact we have:"]},{"cell_type":"code","execution_count":41,"metadata":{"_cell_guid":"e55d7b2f-4c6f-4b4d-b4a7-e5216606db90","_uuid":"fe074cfe-eb0c-4de6-8fcd-ea4426d603b4","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:09:04.328546Z","iopub.status.busy":"2023-02-20T18:09:04.328020Z","iopub.status.idle":"2023-02-20T18:09:04.342848Z","shell.execute_reply":"2023-02-20T18:09:04.342084Z","shell.execute_reply.started":"2023-02-20T18:09:04.328515Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["732216672\n"]}],"source":["print(getMemoryUsage(dfTrainMerged))"]},{"cell_type":"markdown","metadata":{},"source":["So reduce it!"]},{"cell_type":"code","execution_count":42,"metadata":{"_cell_guid":"713d6e19-4f44-47ac-8189-bfc0aed13af4","_uuid":"8e2157dd-9af1-44d2-bf05-d28afdc80960","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:09:04.355718Z","iopub.status.busy":"2023-02-20T18:09:04.353315Z","iopub.status.idle":"2023-02-20T18:09:05.841402Z","shell.execute_reply":"2023-02-20T18:09:05.840395Z","shell.execute_reply.started":"2023-02-20T18:09:04.355677Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["<class 'pandas.core.frame.DataFrame'>\n","Int64Index: 3000888 entries, 0 to 3000887\n","Data columns (total 30 columns):\n"," #   Column          Dtype  \n","---  ------          -----  \n"," 0   id              int64  \n"," 1   date            float32\n"," 2   store_nbr       int64  \n"," 3   family_0        int64  \n"," 4   family_1        int64  \n"," 5   family_2        int64  \n"," 6   family_3        int64  \n"," 7   family_4        int64  \n"," 8   family_5        int64  \n"," 9   sales           float64\n"," 10  onpromotion     int64  \n"," 11  city_0          int64  \n"," 12  city_1          int64  \n"," 13  city_2          int64  \n"," 14  city_3          int64  \n"," 15  city_4          int64  \n"," 16  state_0         int64  \n"," 17  state_1         int64  \n"," 18  state_2         int64  \n"," 19  state_3         int64  \n"," 20  state_4         int64  \n"," 21  type_Store_0    int64  \n"," 22  type_Store_1    int64  \n"," 23  type_Store_2    int64  \n"," 24  cluster         int64  \n"," 25  transactions    float64\n"," 26  type_Holiday_0  int64  \n"," 27  type_Holiday_1  int64  \n"," 28  type_Holiday_2  int64  \n"," 29  dcoilwtico      float64\n","dtypes: float32(1), float64(3), int64(26)\n","memory usage: 698.3 MB\n","None\n","----------------\n","Memory usage of dataframe is 698.30 MB\n","\n","Memory usage AFTER optimization is: 143.09 MB\n","Decreased by 79.5%\n"]}],"source":["dfTrainMerged = reduce_mem_usage(dfTrainMerged)"]},{"cell_type":"code","execution_count":43,"metadata":{"_cell_guid":"06481fbc-51cb-4c88-a2a1-0a3d0ba75dd4","_uuid":"166ccdae-0a35-4e7e-8e13-a1dc98b1bf35","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:09:05.842614Z","iopub.status.busy":"2023-02-20T18:09:05.842369Z","iopub.status.idle":"2023-02-20T18:09:05.850719Z","shell.execute_reply":"2023-02-20T18:09:05.849270Z","shell.execute_reply.started":"2023-02-20T18:09:05.842592Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["6500736\n"]}],"source":["print(getMemoryUsage(dfTestMerged))"]},{"cell_type":"code","execution_count":44,"metadata":{"execution":{"iopub.execute_input":"2023-02-20T18:09:05.852447Z","iopub.status.busy":"2023-02-20T18:09:05.852083Z","iopub.status.idle":"2023-02-20T18:09:05.893188Z","shell.execute_reply":"2023-02-20T18:09:05.892296Z","shell.execute_reply.started":"2023-02-20T18:09:05.852413Z"},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["<class 'pandas.core.frame.DataFrame'>\n","Int64Index: 28512 entries, 0 to 28511\n","Data columns (total 28 columns):\n"," #   Column          Non-Null Count  Dtype  \n","---  ------          --------------  -----  \n"," 0   id              28512 non-null  int64  \n"," 1   date            28512 non-null  float32\n"," 2   store_nbr       28512 non-null  int64  \n"," 3   family_0        28512 non-null  int64  \n"," 4   family_1        28512 non-null  int64  \n"," 5   family_2        28512 non-null  int64  \n"," 6   family_3        28512 non-null  int64  \n"," 7   family_4        28512 non-null  int64  \n"," 8   family_5        28512 non-null  int64  \n"," 9   onpromotion     28512 non-null  int64  \n"," 10  city_0          28512 non-null  int64  \n"," 11  city_1          28512 non-null  int64  \n"," 12  city_2          28512 non-null  int64  \n"," 13  city_3          28512 non-null  int64  \n"," 14  city_4          28512 non-null  int64  \n"," 15  state_0         28512 non-null  int64  \n"," 16  state_1         28512 non-null  int64  \n"," 17  state_2         28512 non-null  int64  \n"," 18  state_3         28512 non-null  int64  \n"," 19  state_4         28512 non-null  int64  \n"," 20  type_Store_0    28512 non-null  int64  \n"," 21  type_Store_1    28512 non-null  int64  \n"," 22  type_Store_2    28512 non-null  int64  \n"," 23  cluster         28512 non-null  int64  \n"," 24  transactions    28512 non-null  float64\n"," 25  type_Holiday_0  28512 non-null  int64  \n"," 26  type_Holiday_1  28512 non-null  int64  \n"," 27  dcoilwtico      28512 non-null  float64\n","dtypes: float32(1), float64(2), int64(25)\n","memory usage: 6.2 MB\n","None\n","----------------\n","Memory usage of dataframe is 6.20 MB\n","\n","Memory usage AFTER optimization is: 1.22 MB\n","Decreased by 80.3%\n"]}],"source":["dfTestMerged = reduce_mem_usage(dfTestMerged)"]},{"cell_type":"code","execution_count":45,"metadata":{"execution":{"iopub.execute_input":"2023-02-20T18:09:05.896060Z","iopub.status.busy":"2023-02-20T18:09:05.895250Z","iopub.status.idle":"2023-02-20T18:09:05.902421Z","shell.execute_reply":"2023-02-20T18:09:05.901524Z","shell.execute_reply.started":"2023-02-20T18:09:05.896026Z"},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["1283040\n"]}],"source":["print(getMemoryUsage(dfTestMerged))"]},{"cell_type":"markdown","metadata":{"_cell_guid":"a6e15897-8759-45ad-8687-761fe308982e","_uuid":"78c7f751-312f-4608-96c8-504fcc3cac09","trusted":true},"source":["# Train and test the model"]},{"cell_type":"code","execution_count":46,"metadata":{"_cell_guid":"11d28149-c48b-40d1-a41a-148458555036","_uuid":"7772199a-9c7f-4617-b9f2-f90dceb4cbed","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:09:05.904540Z","iopub.status.busy":"2023-02-20T18:09:05.903653Z","iopub.status.idle":"2023-02-20T18:09:06.938052Z","shell.execute_reply":"2023-02-20T18:09:06.936879Z","shell.execute_reply.started":"2023-02-20T18:09:05.904514Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["X = dfTrainMerged.drop(\"sales\", axis=1)\n","y = dfTrainMerged[\"sales\"]\n","\n","X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4)"]},{"cell_type":"code","execution_count":47,"metadata":{"_cell_guid":"5fc4cd10-a013-4d49-941b-1fe15ac7d4ee","_uuid":"5c77be6c-bbef-43d8-83f4-c5b870cd332e","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:09:06.939624Z","iopub.status.busy":"2023-02-20T18:09:06.939398Z","iopub.status.idle":"2023-02-20T18:09:07.084953Z","shell.execute_reply":"2023-02-20T18:09:07.083400Z","shell.execute_reply.started":"2023-02-20T18:09:06.939602Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/plain":["0"]},"execution_count":47,"metadata":{},"output_type":"execute_result"}],"source":["makeEmptyDf(dfTrainMerged) # may be redundant!\n","del dfTrainMerged\n","gc.collect()"]},{"cell_type":"markdown","metadata":{},"source":["We've selected four regression models (Linear Regression, Random Forest Regressor, LGBM Regressor, and XGB Regressor), In order to choose what to use, we call executeVoter() function which creates an instance of the VotingRegressor class, that is a type of ensemble model that combines the predictions of multiple base estimators using a weighted average. The VotingRegressor object fits the X_train and y_train data and then scores the model with the X_test and y_test data. After that, it makes predictions on X_test data for the number of samples specified by whatToPredict. Finally, it prints the names of the models used for the ensemble."]},{"cell_type":"code","execution_count":48,"metadata":{"_cell_guid":"ed839d14-c610-4910-ae24-e6c64ca4722e","_uuid":"d5c119c5-e395-4ed4-9246-4ec0a95586a8","collapsed":false,"execution":{"iopub.execute_input":"2023-02-20T18:09:55.477258Z","iopub.status.busy":"2023-02-20T18:09:55.476944Z","iopub.status.idle":"2023-02-20T18:23:22.553848Z","shell.execute_reply":"2023-02-20T18:23:22.552145Z","shell.execute_reply.started":"2023-02-20T18:09:55.477236Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["This are the estimators:\n"," {'lr': LinearRegression(), 'rfr': RandomForestRegressor(), 'lgb': LGBMRegressor(), 'xgb': XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n","             colsample_bynode=1, colsample_bytree=1, enable_categorical=False,\n","             gamma=0, gpu_id=-1, importance_type=None,\n","             interaction_constraints='', learning_rate=0.300000012,\n","             max_delta_step=0, max_depth=6, min_child_weight=1, missing=nan,\n","             monotone_constraints='()', n_estimators=100, n_jobs=6,\n","             num_parallel_tree=1, predictor='auto', random_state=0, reg_alpha=0,\n","             reg_lambda=1, scale_pos_weight=1, subsample=1, tree_method='exact',\n","             validate_parameters=1, verbosity=None)}\n","lr -> 0.22351857622593385\n","rfr -> 0.9531138061875539\n","lgb -> 0.9073586757994937\n","xgb -> 0.8801303120269346\n"]}],"source":["voter_regr = executeVoter(X_train, X_test, y_train, y_test, 5)\n","for name, models in voter_regr.named_estimators_.items():\n","    print(f\"{name} -> {models.score(X_test, y_test)}\")"]},{"cell_type":"code","execution_count":49,"metadata":{"_cell_guid":"f3f8ebac-843d-414b-9b23-f45f6b10ba36","_uuid":"3b2fceb2-4662-4a09-8af2-e3dab406ff92","collapsed":false,"execution":{"iopub.status.busy":"2023-02-20T18:09:07.248463Z","iopub.status.idle":"2023-02-20T18:09:07.248734Z","shell.execute_reply":"2023-02-20T18:09:07.248614Z","shell.execute_reply.started":"2023-02-20T18:09:07.248600Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/plain":["132"]},"execution_count":49,"metadata":{},"output_type":"execute_result"}],"source":["del voter_regr\n","gc.collect()"]},{"cell_type":"markdown","metadata":{},"source":["We choose XGBRegressor."]},{"cell_type":"code","execution_count":50,"metadata":{"_cell_guid":"69843ad6-f05c-4d92-b937-c65887cde54f","_uuid":"cbff5c21-af24-422f-900b-b7dd54877baf","collapsed":false,"execution":{"iopub.status.busy":"2023-02-20T18:09:07.251724Z","iopub.status.idle":"2023-02-20T18:09:07.252019Z","shell.execute_reply":"2023-02-20T18:09:07.251893Z","shell.execute_reply.started":"2023-02-20T18:09:07.251879Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["xgbr = xgb.XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n","             colsample_bynode=1, colsample_bytree=1, enable_categorical=False,\n","             gamma=0, gpu_id=-1, importance_type=None,\n","             interaction_constraints='', learning_rate=0.300000012,\n","             max_delta_step=0, max_depth=6, min_child_weight=1,\n","             monotone_constraints='()', n_estimators=100, n_jobs=6,\n","             num_parallel_tree=1, predictor='auto', random_state=0, reg_alpha=0,\n","             reg_lambda=1, scale_pos_weight=1, subsample=1, tree_method='exact',\n","             validate_parameters=1, verbosity=1)"]},{"cell_type":"markdown","metadata":{},"source":["In order to verify if our model work properly. we call cross_val_score() function that returns a dictionary with the results of the cross-validation, including the score for each fold, the time taken for each fold, and the model instance trained on each fold."]},{"cell_type":"code","execution_count":51,"metadata":{"_cell_guid":"639a5199-d1db-41ba-a474-9b685ac28e06","_uuid":"3f9e2cad-b928-4c6c-8bbb-ffa7015d4170","collapsed":false,"execution":{"iopub.status.busy":"2023-02-20T18:09:07.253107Z","iopub.status.idle":"2023-02-20T18:09:07.253412Z","shell.execute_reply":"2023-02-20T18:09:07.253261Z","shell.execute_reply.started":"2023-02-20T18:09:07.253247Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["Mean cross-validation score: 0.88\n"]}],"source":["from sklearn.model_selection import cross_val_score\n","scores = cross_val_score(xgbr, X_train, y_train, cv=10)\n","print(\"Mean cross-validation score: %.2f\" % scores.mean())"]},{"cell_type":"markdown","metadata":{},"source":["So, we can fit."]},{"cell_type":"code","execution_count":52,"metadata":{"_cell_guid":"400c719d-df30-4681-bd56-1cc72da28ee1","_uuid":"5c3fa377-f29e-4593-99b4-6ec2dde57de3","collapsed":false,"execution":{"iopub.status.busy":"2023-02-20T18:09:07.255076Z","iopub.status.idle":"2023-02-20T18:09:07.255377Z","shell.execute_reply":"2023-02-20T18:09:07.255229Z","shell.execute_reply.started":"2023-02-20T18:09:07.255216Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/plain":["XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n","             colsample_bynode=1, colsample_bytree=1, enable_categorical=False,\n","             gamma=0, gpu_id=-1, importance_type=None,\n","             interaction_constraints='', learning_rate=0.300000012,\n","             max_delta_step=0, max_depth=6, min_child_weight=1, missing=nan,\n","             monotone_constraints='()', n_estimators=100, n_jobs=6,\n","             num_parallel_tree=1, predictor='auto', random_state=0, reg_alpha=0,\n","             reg_lambda=1, scale_pos_weight=1, subsample=1, tree_method='exact',\n","             validate_parameters=1, verbosity=1)"]},"execution_count":52,"metadata":{},"output_type":"execute_result"}],"source":["xgbr.fit(X_train, y_train)"]},{"cell_type":"markdown","metadata":{},"source":["Check scores."]},{"cell_type":"code","execution_count":53,"metadata":{"_cell_guid":"ac6397fa-14dc-4d65-8536-4b3318a302db","_uuid":"b75df3fa-29c0-425d-be27-830c5b4d281a","collapsed":false,"execution":{"iopub.status.busy":"2023-02-20T18:09:07.256689Z","iopub.status.idle":"2023-02-20T18:09:07.256966Z","shell.execute_reply":"2023-02-20T18:09:07.256842Z","shell.execute_reply.started":"2023-02-20T18:09:07.256829Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["scoreTraining = xgbr.score(X_train, y_train)\n","scoreTest = xgbr.score(X_test, y_test)"]},{"cell_type":"code","execution_count":54,"metadata":{"_cell_guid":"9a30a4d6-8af2-42f9-877c-ccc2de3effd5","_uuid":"e4b7af19-a1ba-46c3-949f-e22f59df9f11","collapsed":false,"execution":{"iopub.status.busy":"2023-02-20T18:09:07.257897Z","iopub.status.idle":"2023-02-20T18:09:07.258173Z","shell.execute_reply":"2023-02-20T18:09:07.258049Z","shell.execute_reply.started":"2023-02-20T18:09:07.258035Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["Training score:  0.8836883801863323\n","Test score:  0.8801303120269346\n"]}],"source":["print(\"Training score: \", scoreTraining)\n","print(\"Test score: \", scoreTest)"]},{"cell_type":"markdown","metadata":{},"source":["It's time to make predictions!"]},{"cell_type":"code","execution_count":55,"metadata":{"_cell_guid":"d622befc-bf81-474d-b0e6-97ad3b39eedf","_uuid":"18562c43-5e7f-4195-89bb-ab317e11fd64","collapsed":false,"execution":{"iopub.status.busy":"2023-02-20T18:09:07.259292Z","iopub.status.idle":"2023-02-20T18:09:07.259605Z","shell.execute_reply":"2023-02-20T18:09:07.259474Z","shell.execute_reply.started":"2023-02-20T18:09:07.259459Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":["Mean Absolute Error:\t 155.99464416503906\n","rmse:\t 382.53933621263053\n","Mean Square Error  :\t 382.53933621263053\n"]}],"source":["# Predict target for test data\n","predictions = xgbr.predict(X_test)\n","predictions = predictions.reshape(len(predictions), 1)\n","\n","from sklearn.metrics import mean_absolute_error\n","\n","# Error printing\n","mse = mean_squared_error(y_test, predictions)\n","rmse = mse**.5\n","\n","print(f\"Mean Absolute Error:\\t {mean_absolute_error(y_test, predictions)}\")\n","print(f\"rmse:\\t {rmse}\")\n","print(f\"Mean Square Error  :\\t {rmse}\")"]},{"cell_type":"markdown","metadata":{},"source":["### Submit predictions"]},{"cell_type":"code","execution_count":56,"metadata":{"_cell_guid":"6e505e26-c5f5-42d9-9ead-ef9f0219df66","_uuid":"ab03b296-e2f6-47ff-bfd4-0b87697d7533","collapsed":false,"execution":{"iopub.status.busy":"2023-02-20T18:09:07.261058Z","iopub.status.idle":"2023-02-20T18:09:07.261831Z","shell.execute_reply":"2023-02-20T18:09:07.261698Z","shell.execute_reply.started":"2023-02-20T18:09:07.261682Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[{"data":{"text/plain":["array([2718.6587, 2639.9302, 1380.7925, ..., 2884.8293, 2657.1685,\n","       2744.6748], dtype=float32)"]},"execution_count":56,"metadata":{},"output_type":"execute_result"}],"source":["dfTestMerged['sales'] = np.NaN\n","predictions = xgbr.predict(dfTestMerged)\n","predictions"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["Activate the following code only in Kagle environment, because if output folder isn't empty, we have a \"Submission CSV Not Found\" error!"]},{"cell_type":"code","execution_count":57,"metadata":{"execution":{"iopub.status.busy":"2023-02-20T18:09:07.262901Z","iopub.status.idle":"2023-02-20T18:09:07.263194Z","shell.execute_reply":"2023-02-20T18:09:07.263065Z","shell.execute_reply.started":"2023-02-20T18:09:07.263052Z"},"trusted":true},"outputs":[],"source":["# for dirname, _, filenames in os.walk(\"/kaggle/working/\"):\n","#     for filename in filenames:\n","#         os.remove(os.path.join(dirname, filename))"]},{"cell_type":"code","execution_count":58,"metadata":{"_cell_guid":"9ffb2519-607d-43fd-9bb9-d0ecc6ee6abe","_uuid":"dd05542f-cc9e-4631-ad98-05d9bf91cd6b","collapsed":false,"execution":{"iopub.status.busy":"2023-02-20T18:09:07.264122Z","iopub.status.idle":"2023-02-20T18:09:07.264415Z","shell.execute_reply":"2023-02-20T18:09:07.264269Z","shell.execute_reply.started":"2023-02-20T18:09:07.264256Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["myId = dfTestMerged['id']\n","\n","submission = pd.DataFrame({\"id\": myId,\"sales\": predictions})\n","submission.to_csv('submission.csv', index=False)\n"]}],"metadata":{"kernelspec":{"display_name":"Python 3.6.9 64-bit","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.9"},"vscode":{"interpreter":{"hash":"31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"}}},"nbformat":4,"nbformat_minor":4}