notebook.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "notebook.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.9"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wd0jNlcZdAFi"
      },
      "source": [
        "# Federated Learning Tutorial\n",
        "\n",
        "Author: Daniyal Shahrokhian\n",
        "\n",
        "## Problem\n",
        "\n",
        "Worldline has open sourced some of their data of credit card transactions to\n",
        "try to predict fraud:\n",
        "\n",
        "https://www.kaggle.com/mlg-ulb/creditcardfraud\n",
        "\n",
        "Imagine this dataset is cut in half horizontally. Alice has half of the data\n",
        "and Bob has the other half. Neither of them wants to send their raw data to us.\n",
        "However, we convince them to let our model learn from their data in a federated\n",
        "setting. Implement a way for our model to train on the combined data of both\n",
        "Alice and Bob without either of them sending us any raw data. Compare it with the model with the traditional approach that can see all the data at once."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nAxuAYSpdAFk"
      },
      "source": [
        "## Dependencies & Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BzYAWfvUdAFl"
      },
      "source": [
        "%%shell\n",
        "pip install sklearn\n",
        "pip install pandas\n",
        "pip install matplotlib\n",
        "pip install tensorflow\n",
        "\n",
        "pip uninstall --yes tensorboard tb-nightly\n",
        "\n",
        "pip install --quiet --upgrade tensorflow-federated\n",
        "pip install --quiet --upgrade nest-asyncio\n",
        "pip install --quiet --upgrade tensorboard"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bZAnYwAkdAFl"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "%matplotlib notebook\n",
        "%matplotlib inline\n",
        "\n",
        "import nest_asyncio\n",
        "nest_asyncio.apply()\n",
        "\n",
        "import pandas as pd\n",
        "import random\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.preprocessing import MinMaxScaler\n",
        "import tensorflow as tf\n",
        "import tensorflow_federated as tff\n",
        "from tensorflow.keras.metrics import BinaryAccuracy, Precision, Recall\n",
        "\n",
        "SEED = 1337\n",
        "tf.random.set_seed(SEED)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KTm_q5vvdAFm"
      },
      "source": [
        "## Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RAPWGDTEdAFm"
      },
      "source": [
        "df = pd.read_csv('creditcard.csv')\n",
        "df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WhVAV43ZdAFm"
      },
      "source": [
        "# Creating Alice and Bob's splits:\n",
        "alice_df = df[:len(df.index)//2]\n",
        "bob_df = df[len(df.index)//2:]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QNP4fyRtdAFm"
      },
      "source": [
        "### Exploratory Analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GJcJ0XKpdAFn"
      },
      "source": [
        "Fraudulent transactions only account for 0.17% of the total transactions. Given the large distribution difference, class weighting applied to the classifier won't cut it, so it is very likely that we will need to rely on under/over-sampling."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tzo3e8h1dAFn"
      },
      "source": [
        "df['Class'].value_counts()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vrV-C0stdAFn"
      },
      "source": [
        "When splitting the data horizontally, the class distribution does not change drastically."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_zv4s-5QdAFo"
      },
      "source": [
        "alice_df['Class'].value_counts()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gZJr_3bjdAFo"
      },
      "source": [
        "bob_df['Class'].value_counts()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lIhA1oPHdAFo"
      },
      "source": [
        "As it can be seen by plotting the density estimation of the two datasets, there is some differences between the two. In many Federated scenarios, the data sources are non-i.i.d. (Independent and Identically Distributed). At first glance, this also happens in our dataset, but given the small variance it won't be much of a problem. The only variables that have significant differences are `Time` and `Amount`, the former we will not even include on our classifier."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NkJlLNJqdAFp"
      },
      "source": [
        "for col in df:\n",
        "    combined = pd.concat([alice_df[col].reset_index(drop=True), bob_df[col].reset_index(drop=True)], axis=1, ignore_index=True, keys=['Alice', 'Bob'])\n",
        "    fig, ax = plt.subplots(figsize=(3,2))\n",
        "    combined.sample(1000, random_state=SEED).plot(kind='density', ax=ax) # Random sample of 1000 to ease computation\n",
        "    ax.title.set_text(col)\n",
        "    ax.legend(['Alice', 'Bob'])\n",
        "    plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MmYleGGcdAFp"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZYYi42tYdAFp"
      },
      "source": [
        "EPOCHS = 100\n",
        "BATCH_SIZE = 64"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QpgncmHIdAFp"
      },
      "source": [
        "## Federated Learning Approach"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DIg-Ji57dAFp"
      },
      "source": [
        "### Data Loading"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QIx3CIRydAFq"
      },
      "source": [
        "def make_tf_dataset(dataframe, negative_ratio=None, batch_size=None):\n",
        "    dataset = dataframe.drop(['Time'], axis=1)\n",
        "\n",
        "    # Class balancing\n",
        "    pos_df = dataset[dataset['Class'] == 1]\n",
        "    neg_df = dataset[dataset['Class'] == 0]\n",
        "    if negative_ratio:\n",
        "        neg_df = neg_df.iloc[random.sample(range(0, len(neg_df)), len(pos_df)*negative_ratio), :]\n",
        "    balanced_df = pd.concat([pos_df, neg_df], ignore_index=True, sort=False)\n",
        "\n",
        "    y = balanced_df.pop('Class')\n",
        "    \n",
        "    # Dataset creation\n",
        "    dataset = tf.data.Dataset.from_tensor_slices((balanced_df.values, y.to_frame().values))\n",
        "    dataset = dataset.shuffle(2048, seed=SEED)\n",
        "    if batch_size:\n",
        "        dataset = dataset.batch(batch_size)\n",
        "\n",
        "    return dataset"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zBj7fQxFdAFq"
      },
      "source": [
        "train_data, val_data = [], []\n",
        "for client_data in [alice_df, bob_df]:\n",
        "    train_df, val_df = train_test_split(client_data, test_size=0.1, random_state=SEED)\n",
        "\n",
        "    # Scaling (Standardization actually hurts performance) \n",
        "    scaler = MinMaxScaler() \n",
        "    train_features = scaler.fit_transform(train_df.drop(['Class'], axis=1))\n",
        "    val_features = scaler.transform(val_df.drop(['Class'], axis=1))\n",
        "\n",
        "    train_df[train_df.columns.difference(['Class'])] = train_features\n",
        "    val_df[val_df.columns.difference(['Class'])] = val_features\n",
        "\n",
        "    # TF Datasets\n",
        "    train_data.append(make_tf_dataset(train_df, negative_ratio=10, batch_size=BATCH_SIZE))\n",
        "    val_data.append(make_tf_dataset(val_df, batch_size=1))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eP8byNHDdAFq"
      },
      "source": [
        "### Model Definition"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rCci5e2-dAFq"
      },
      "source": [
        "def input_spec():\n",
        "    return (\n",
        "        tf.TensorSpec([None, 29], tf.float64),\n",
        "        tf.TensorSpec([None, 1], tf.int64)\n",
        "    )\n",
        "\n",
        "def model_fn():\n",
        "    model = tf.keras.models.Sequential([\n",
        "        tf.keras.layers.InputLayer(input_shape=(29,)),\n",
        "        tf.keras.layers.Dense(32, activation='relu'),\n",
        "        tf.keras.layers.Dense(64, activation='relu'),\n",
        "        tf.keras.layers.Dense(32, activation='relu'),\n",
        "        tf.keras.layers.Dense(1, activation='sigmoid'),\n",
        "    ])\n",
        "\n",
        "    return tff.learning.from_keras_model(\n",
        "        model,\n",
        "        input_spec=input_spec(),\n",
        "        loss=tf.keras.losses.BinaryCrossentropy(),\n",
        "        metrics=[BinaryAccuracy(), Precision(), Recall()])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sd99tw3RdAFr"
      },
      "source": [
        "### Training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4XUqmO4HdAFs"
      },
      "source": [
        "Each time the `next` method is called, the server model is broadcast to each client using a broadcast function. For each client, one epoch of local training is performed. Each client computes the difference between the client model after training and the initial broadcast model. These model deltas are then aggregated at the server using some aggregation function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x1YqmgzhdAFr"
      },
      "source": [
        "trainer = tff.learning.build_federated_averaging_process(\n",
        "    model_fn,\n",
        "    client_optimizer_fn=lambda: tf.keras.optimizers.Adam(),\n",
        "    server_optimizer_fn=lambda: tf.keras.optimizers.Adam()\n",
        ")\n",
        "\n",
        "state = trainer.initialize()\n",
        "train_hist = []\n",
        "for i in range(EPOCHS):\n",
        "    state, metrics = trainer.next(state, train_data)\n",
        "    train_hist.append(metrics)\n",
        "\n",
        "    print(f\"\\rRun {i+1}/{EPOCHS}\", end=\"\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WOBx91fpdAFs"
      },
      "source": [
        "### Evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hp0PJWjydAFs"
      },
      "source": [
        "evaluator = tff.learning.build_federated_evaluation(model_fn)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XHy9ihAZdAFs"
      },
      "source": [
        "federated_metrics = evaluator(state.model, val_data)\n",
        "federated_metrics"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GnAXQMx_dAFs"
      },
      "source": [
        "## Single Model with all Data at once (for comparison)\n",
        "\n",
        "### Data Loading"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yqL-buDmdAFt"
      },
      "source": [
        "train_data = train_data[0].concatenate(train_data[1])\n",
        "val_data = val_data[0].concatenate(val_data[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MDjQc-u0dAFt"
      },
      "source": [
        "### Model Definition"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oj7Wsal0dAFt"
      },
      "source": [
        "def model_fn():\n",
        "    model = tf.keras.models.Sequential([\n",
        "        tf.keras.layers.InputLayer(input_shape=(29,)),\n",
        "        tf.keras.layers.Dense(32, activation='relu'),\n",
        "        tf.keras.layers.Dense(64, activation='relu'),\n",
        "        tf.keras.layers.Dense(32, activation='relu'),\n",
        "        tf.keras.layers.Dense(1, activation='sigmoid'),\n",
        "    ])\n",
        "    \n",
        "    model.compile(\n",
        "        loss=tf.keras.losses.BinaryCrossentropy(),\n",
        "        optimizer=tf.keras.optimizers.Adam(),\n",
        "        metrics=[BinaryAccuracy(), Precision(), Recall()],\n",
        "    )\n",
        "    \n",
        "    return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pDYXpu6WdAFt"
      },
      "source": [
        "### Training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jm8nwUyDdAFt"
      },
      "source": [
        "model = model_fn()\n",
        "history = model.fit(train_data, epochs=EPOCHS)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pQ_J02EAdAFt"
      },
      "source": [
        "### Evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y_xbTiDpdAFu"
      },
      "source": [
        "test_scores = model.evaluate(val_data)\n",
        "single_metrics = {\n",
        "    'loss': test_scores[0],\n",
        "    'binary_accuracy': test_scores[1],\n",
        "    'precision': test_scores[2],\n",
        "    'recall': test_scores[3]\n",
        "}\n",
        "single_metrics"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vfJ1_ak4dAFu"
      },
      "source": [
        "## Conclusion\n",
        "\n",
        "Comparing both models:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zp2npXnodAFu"
      },
      "source": [
        "print(f\"---Single model metrics---\\n{single_metrics}\\n\")\n",
        "print(f\"---Federated model metrics---\\n{dict(federated_metrics)}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3jgEX9PadAFu"
      },
      "source": [
        "The Federated Learning approach has a better balance between precision and recall, which might be an indicator of better handling of the imbalanced dataset."
      ]
    }
  ]
}