demos/graph-classification/gcn-supervised-graph-classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0",
   "metadata": {},
   "source": [
    "# Supervised graph classification with GCN\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1",
   "metadata": {
    "nbsphinx": "hidden",
    "tags": [
     "CloudRunner"
    ]
   },
   "source": [
    "<table><tr><td>Run the latest release of this notebook:</td><td><a href=\"https://mybinder.org/v2/gh/stellargraph/stellargraph/master?urlpath=lab/tree/demos/graph-classification/gcn-supervised-graph-classification.ipynb\" alt=\"Open In Binder\" target=\"_parent\"><img src=\"https://mybinder.org/badge_logo.svg\"/></a></td><td><a href=\"https://colab.research.google.com/github/stellargraph/stellargraph/blob/master/demos/graph-classification/gcn-supervised-graph-classification.ipynb\" alt=\"Open In Colab\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\"/></a></td></tr></table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2",
   "metadata": {},
   "source": [
    "This notebook demonstrates how to train a graph classification model in a supervised setting using graph convolutional layers followed by a mean pooling layer as well as any number of fully connected layers.\n",
    "\n",
    "The graph convolutional classification model architecture is based on the one proposed in [1] (see Figure 5 in [1]) using the graph convolutional layers from [2]. This demo differs from [1] in the dataset, MUTAG, used here; MUTAG is a collection of static graphs representing chemical compounds with each graph associated with a binary label. Furthermore, none of the graph convolutional layers in our model utilise an attention head as proposed in [1].\n",
    "\n",
    "Evaluation data for graph kernel-based approaches shown in the very last cell in this notebook are taken from [3].\n",
    "\n",
    "**References**\n",
    "\n",
    "[1] Fake News Detection on Social Media using Geometric Deep Learning, F. Monti, F. Frasca, D. Eynard, D. Mannion, and M. M. Bronstein, ICLR 2019. ([link](https://arxiv.org/abs/1902.06673))\n",
    "\n",
    "[2] Semi-supervised Classification with Graph Convolutional Networks, T. N. Kipf and M. Welling, ICLR 2017. ([link](https://arxiv.org/abs/1609.02907))\n",
    "\n",
    "[3] An End-to-End Deep Learning Architecture for Graph Classification, M. Zhang, Z. Cui, M. Neumann, Y. Chen, AAAI-18. ([link](https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewPaper/17146))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3",
   "metadata": {
    "nbsphinx": "hidden",
    "tags": [
     "CloudRunner"
    ]
   },
   "outputs": [],
   "source": [
    "# install StellarGraph if running on Google Colab\n",
    "import sys\n",
    "if 'google.colab' in sys.modules:\n",
    "  %pip install -q stellargraph[demos]==1.3.0b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4",
   "metadata": {
    "nbsphinx": "hidden",
    "tags": [
     "VersionCheck"
    ]
   },
   "outputs": [],
   "source": [
    "# verify that we're using the correct version of StellarGraph for this notebook\n",
    "import stellargraph as sg\n",
    "\n",
    "try:\n",
    "    sg.utils.validate_notebook_version(\"1.3.0b\")\n",
    "except AttributeError:\n",
    "    raise ValueError(\n",
    "        f\"This notebook requires StellarGraph version 1.3.0b, but a different version {sg.__version__} is installed.  Please see <https://github.com/stellargraph/stellargraph/issues/1172>.\"\n",
    "    ) from None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import stellargraph as sg\n",
    "from stellargraph.mapper import PaddedGraphGenerator\n",
    "from stellargraph.layer import GCNSupervisedGraphClassification\n",
    "from stellargraph import StellarGraph\n",
    "\n",
    "from stellargraph import datasets\n",
    "\n",
    "from sklearn import model_selection\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "from tensorflow.keras import Model\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.losses import binary_crossentropy\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6",
   "metadata": {},
   "source": [
    "## Import the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7",
   "metadata": {
    "tags": [
     "DataLoadingLinks"
    ]
   },
   "source": [
    "(See [the \"Loading from Pandas\" demo](../basics/loading-pandas.ipynb) for details on how data can be loaded.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8",
   "metadata": {
    "tags": [
     "DataLoading"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "Each graph represents a chemical compound and graph labels represent 'their mutagenic effect on a specific gram negative bacterium.'The dataset includes 188 graphs with 18 nodes and 20 edges on average for each graph. Graph nodes have 7 labels and each graph is labelled as belonging to 1 of 2 classes."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dataset = datasets.MUTAG()\n",
    "display(HTML(dataset.description))\n",
    "graphs, graph_labels = dataset.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9",
   "metadata": {},
   "source": [
    "The `graphs` value is a list of many `StellarGraph` instances, each of which has a few node features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "10",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "StellarGraph: Undirected multigraph\n",
      " Nodes: 17, Edges: 38\n",
      "\n",
      " Node types:\n",
      "  default: [17]\n",
      "    Features: float32 vector, length 7\n",
      "    Edge types: default-default->default\n",
      "\n",
      " Edge types:\n",
      "    default-default->default: [38]\n",
      "        Weights: all 1 (default)\n",
      "        Features: none\n"
     ]
    }
   ],
   "source": [
    "print(graphs[0].info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "11",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "StellarGraph: Undirected multigraph\n",
      " Nodes: 13, Edges: 28\n",
      "\n",
      " Node types:\n",
      "  default: [13]\n",
      "    Features: float32 vector, length 7\n",
      "    Edge types: default-default->default\n",
      "\n",
      " Edge types:\n",
      "    default-default->default: [28]\n",
      "        Weights: all 1 (default)\n",
      "        Features: none\n"
     ]
    }
   ],
   "source": [
    "print(graphs[1].info())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12",
   "metadata": {},
   "source": [
    "Summary statistics of the sizes of the graphs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "13",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nodes</th>\n",
       "      <th>edges</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>188.0</td>\n",
       "      <td>188.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>17.9</td>\n",
       "      <td>39.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>4.6</td>\n",
       "      <td>11.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>10.0</td>\n",
       "      <td>20.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>14.0</td>\n",
       "      <td>28.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>17.5</td>\n",
       "      <td>38.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>22.0</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>28.0</td>\n",
       "      <td>66.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       nodes  edges\n",
       "count  188.0  188.0\n",
       "mean    17.9   39.6\n",
       "std      4.6   11.4\n",
       "min     10.0   20.0\n",
       "25%     14.0   28.0\n",
       "50%     17.5   38.0\n",
       "75%     22.0   50.0\n",
       "max     28.0   66.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary = pd.DataFrame(\n",
    "    [(g.number_of_nodes(), g.number_of_edges()) for g in graphs],\n",
    "    columns=[\"nodes\", \"edges\"],\n",
    ")\n",
    "summary.describe().round(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14",
   "metadata": {},
   "source": [
    "The labels are `1` or `-1`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "15",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-1</th>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    label\n",
       "1     125\n",
       "-1     63"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "graph_labels.value_counts().to_frame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "16",
   "metadata": {},
   "outputs": [],
   "source": [
    "graph_labels = pd.get_dummies(graph_labels, drop_first=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17",
   "metadata": {},
   "source": [
    "### Prepare graph generator\n",
    "\n",
    "To feed data to the `tf.Keras` model that we will create later, we need a data generator. For supervised graph classification, we create an instance of `StellarGraph`'s `PaddedGraphGenerator` class. Note that `graphs` is a list of `StellarGraph` graph objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "18",
   "metadata": {},
   "outputs": [],
   "source": [
    "generator = PaddedGraphGenerator(graphs=graphs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19",
   "metadata": {},
   "source": [
    "### Create the Keras graph classification model\n",
    "\n",
    "We are now ready to create a `tf.Keras` graph classification model using `StellarGraph`'s `GraphClassification` class together with standard `tf.Keras` layers, e.g., `Dense`. \n",
    "\n",
    "The input is the graph represented by its adjacency and node features matrices. The first two layers are Graph Convolutional as in [2] with each layer having 64 units and `relu` activations. The next layer is a mean pooling layer where the learned node representation are summarized to create a graph representation. The graph representation is input to two fully connected layers with 32 and 16 units respectively and `relu` activations. The last layer is the output layer with a single unit and `sigmoid` activation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20",
   "metadata": {},
   "source": [
    "![](graph_classification_architecture.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "21",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_graph_classification_model(generator):\n",
    "    gc_model = GCNSupervisedGraphClassification(\n",
    "        layer_sizes=[64, 64],\n",
    "        activations=[\"relu\", \"relu\"],\n",
    "        generator=generator,\n",
    "        dropout=0.5,\n",
    "    )\n",
    "    x_inp, x_out = gc_model.in_out_tensors()\n",
    "    predictions = Dense(units=32, activation=\"relu\")(x_out)\n",
    "    predictions = Dense(units=16, activation=\"relu\")(predictions)\n",
    "    predictions = Dense(units=1, activation=\"sigmoid\")(predictions)\n",
    "\n",
    "    # Let's create the Keras model and prepare it for training\n",
    "    model = Model(inputs=x_inp, outputs=predictions)\n",
    "    model.compile(optimizer=Adam(0.005), loss=binary_crossentropy, metrics=[\"acc\"])\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22",
   "metadata": {},
   "source": [
    "### Train the model\n",
    "\n",
    "We can now train the model using the model's `fit` method. First, we specify some important training parameters such as the number of training epochs, number of fold for cross validation and the number of time to repeat cross validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "23",
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "epochs = 200  # maximum number of training epochs\n",
    "folds = 10  # the number of folds for k-fold cross validation\n",
    "n_repeats = 5  # the number of repeats for repeated k-fold cross validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "24",
   "metadata": {},
   "outputs": [],
   "source": [
    "es = EarlyStopping(\n",
    "    monitor=\"val_loss\", min_delta=0, patience=25, restore_best_weights=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25",
   "metadata": {},
   "source": [
    "The method `train_fold` is used to train a graph classification model for a single fold of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "26",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_fold(model, train_gen, test_gen, es, epochs):\n",
    "    history = model.fit(\n",
    "        train_gen, epochs=epochs, validation_data=test_gen, verbose=0, callbacks=[es],\n",
    "    )\n",
    "    # calculate performance on the test data and return along with history\n",
    "    test_metrics = model.evaluate(test_gen, verbose=0)\n",
    "    test_acc = test_metrics[model.metrics_names.index(\"acc\")]\n",
    "\n",
    "    return history, test_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "27",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_generators(train_index, test_index, graph_labels, batch_size):\n",
    "    train_gen = generator.flow(\n",
    "        train_index, targets=graph_labels.iloc[train_index].values, batch_size=batch_size\n",
    "    )\n",
    "    test_gen = generator.flow(\n",
    "        test_index, targets=graph_labels.iloc[test_index].values, batch_size=batch_size\n",
    "    )\n",
    "\n",
    "    return train_gen, test_gen"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28",
   "metadata": {},
   "source": [
    "The code below puts all the above functionality together in a training loop for repeated k-fold cross-validation where the number of folds is 10, `folds=10`; that is we do 10-fold cross validation `n_repeats` times where `n_repeats=5`.\n",
    "\n",
    "**Note**: The below code may take a long time to run depending on the value set for `n_repeats`. The larger the latter, the longer it takes since for each repeat we train and evaluate 10 graph classification models, one for each fold of the data. For progress updates, we recommend that you set `verbose=2` in the call to the `fit` method is cell 10, line 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "29",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training and evaluating on fold 1 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 2 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 3 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 4 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 5 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 6 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 7 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 8 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 9 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 10 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 11 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 12 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 13 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 14 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 15 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 16 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 17 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 18 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 19 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 20 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 21 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 22 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 23 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 24 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 25 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 26 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 27 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 28 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 29 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 30 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 31 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 32 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 33 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 34 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 35 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 36 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 37 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 38 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 39 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 40 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 41 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 42 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 43 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 44 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 45 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 46 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 47 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 48 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 49 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n",
      "Training and evaluating on fold 50 out of 50...\n",
      "  ['...']\n",
      "  ['...']\n",
      "  ['...']\n"
     ]
    }
   ],
   "source": [
    "test_accs = []\n",
    "\n",
    "stratified_folds = model_selection.RepeatedStratifiedKFold(\n",
    "    n_splits=folds, n_repeats=n_repeats\n",
    ").split(graph_labels, graph_labels)\n",
    "\n",
    "for i, (train_index, test_index) in enumerate(stratified_folds):\n",
    "    print(f\"Training and evaluating on fold {i+1} out of {folds * n_repeats}...\")\n",
    "    train_gen, test_gen = get_generators(\n",
    "        train_index, test_index, graph_labels, batch_size=30\n",
    "    )\n",
    "\n",
    "    model = create_graph_classification_model(generator)\n",
    "\n",
    "    history, acc = train_fold(model, train_gen, test_gen, es, epochs)\n",
    "\n",
    "    test_accs.append(acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "30",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy over all folds mean: 76.4% and std: 6.7%\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    f\"Accuracy over all folds mean: {np.mean(test_accs)*100:.3}% and std: {np.std(test_accs)*100:.2}%\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31",
   "metadata": {},
   "source": [
    "Finally, we plot a histogram of the accuracy of all `n_repeats x folds` models trained (50 in total)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "32",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Count')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfgAAAFzCAYAAADSXxtkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAY50lEQVR4nO3df7BkZX3n8fdHRiCrKCA3iMAwRAkJmoDuFeOPbPglwkhEDVEmiRkN7hhXXa1ks4VrlVqmtoqtjTEVcWUnwKIWQaNIQhYUJ4oaEkUGwi9RhJBxmZHACAoSda1hv/tHn4nNte/c5t7b3befeb+quvqc5zzn9Lefmdufe37cPqkqJElSWx436QIkSdLyM+AlSWqQAS9JUoMMeEmSGmTAS5LUIANekqQGrZp0AcvpgAMOqDVr1ky6DEmSxuL666//dlXNDFrWVMCvWbOGzZs3T7oMSZLGIsk351vmIXpJkhpkwEuS1CADXpKkBhnwkiQ1yICXJKlBBrwkSQ0y4CVJapABL0lSgwx4SZIaZMBLktQgA16SpAYZ8JIkNciAlySpQU3dTU5q0Zqzr5h0CQvacs5LJ12CpDncg5ckqUEGvCRJDTLgJUlqkAEvSVKDDHhJkhpkwEuS1CADXpKkBhnwkiQ1yICXJKlBBrwkSQ0y4CVJatDIAj7JoUmuTnJbkq8meWvXvn+STUnu6J73m2f99V2fO5KsH1WdkiS1aJR78DuA36+qo4BfAt6U5CjgbOCzVXUE8Nlu/lGS7A+8C3gecCzwrvl+EZAkST9pZAFfVfdU1Q3d9PeArwEHA6cDH+q6fQh4+YDVXwJsqqoHquo7wCbglFHVKklSa8ZyDj7JGuDZwLXAgVV1T7fon4EDB6xyMHB33/zWrm3Qtjck2Zxk8/bt25etZkmSptnIAz7JE4FLgbdV1UP9y6qqgFrK9qtqY1XNVtXszMzMUjYlSVIzRhrwSR5PL9wvrqpPds33JjmoW34QcN+AVbcBh/bNH9K1SZKkIYzyKvoAFwBfq6o/7lt0ObDzqvj1wF8NWP0q4OQk+3UX153ctUmSpCGMcg/+hcBrgBOS3Ng91gLnAC9OcgdwUjdPktkk5wNU1QPAHwLXdY/3dG2SJGkIq0a14aq6Bsg8i08c0H8z8Pq++QuBC0dTnSRJbfOb7CRJapABL0lSgwx4SZIaZMBLktQgA16SpAYZ8JIkNciAlySpQQa8JEkNMuAlSWqQAS9JUoMMeEmSGmTAS5LUIANekqQGGfCSJDXIgJckqUEGvCRJDTLgJUlqkAEvSVKDDHhJkhpkwEuS1CADXpKkBhnwkiQ1yICXJKlBBrwkSQ0y4CVJapABL0lSgwx4SZIaZMBLktSgVaPacJILgdOA+6rqWV3bx4Ajuy77At+tqmMGrLsF+B7wCLCjqmZHVackSS0aWcADFwHnAh/e2VBVr945neS9wIO7WP/4qvr2yKqTJKlhIwv4qvpikjWDliUJ8CrghFG9viRJu7NJnYP/ZeDeqrpjnuUFfCbJ9Uk27GpDSTYk2Zxk8/bt25e9UEmSptGkAn4dcMkulr+oqp4DnAq8Kcm/m69jVW2sqtmqmp2ZmVnuOiVJmkpjD/gkq4BXAh+br09Vbeue7wMuA44dT3WSJLVhEnvwJwFfr6qtgxYmeUKSfXZOAycDt46xPkmSpt7IAj7JJcCXgCOTbE1yVrfoTOYcnk/ytCRXdrMHAtckuQn4CnBFVX16VHVKktSiUV5Fv26e9tcOaPsWsLabvgs4elR1SZK0O/Cb7CRJapABL0lSgwx4SZIaZMBLktQgA16SpAYZ8JIkNciAlySpQQa8JEkNMuAlSWqQAS9JUoMMeEmSGmTAS5LUIANekqQGGfCSJDXIgJckqUEGvCRJDTLgJUlqkAEvSVKDDHhJkhpkwEuS1CADXpKkBhnwkiQ1yICXJKlBBrwkSQ0y4CVJapABL0lSgwx4SZIaNLKAT3JhkvuS3NrX9u4k25Lc2D3WzrPuKUluT3JnkrNHVaMkSa0a5R78RcApA9rfV1XHdI8r5y5MsgfwAeBU4ChgXZKjRlinJEnNGVnAV9UXgQcWseqxwJ1VdVdV/Qj4KHD6shYnSVLjJnEO/s1Jbu4O4e83YPnBwN1981u7NkmSNKRxB/wHgacDxwD3AO9d6gaTbEiyOcnm7du3L3VzkiQ1YawBX1X3VtUjVfX/gD+jdzh+rm3AoX3zh3Rt821zY1XNVtXszMzM8hYsSdKUGmvAJzmob/YVwK0Dul0HHJHk8CR7AmcCl4+jPkmSWrFqVBtOcglwHHBAkq3Au4DjkhwDFLAFeEPX92nA+VW1tqp2JHkzcBWwB3BhVX11VHVKktSikQV8Va0b0HzBPH2/Baztm78S+Ik/oZMkScPxm+wkSWqQAS9JUoMMeEmSGmTAS5LUIANekqQGGfCSJDXIgJckqUEGvCRJDTLgJUlqkAEvSVKDDHhJkhpkwEuS1CADXpKkBhnwkiQ1yICXJKlBBrwkSQ0y4CVJapABL0lSgwx4SZIaZMBLktQgA16SpAYZ8JIkNciAlySpQQa8JEkNMuAlSWqQAS9JUoMMeEmSGmTAS5LUoJEFfJILk9yX5Na+tv+e5OtJbk5yWZJ951l3S5JbktyYZPOoapQkqVWj3IO/CDhlTtsm4FlV9YvAN4C372L946vqmKqaHVF9kiQ1a2QBX1VfBB6Y0/aZqtrRzX4ZOGRUry9J0u5skufgfwf41DzLCvhMkuuTbBhjTZIkNWHVJF40yTuAHcDF83R5UVVtS/LTwKYkX++OCAza1gZgA8Dq1atHUq8kSdNm7HvwSV4LnAb8ZlXVoD5Vta17vg+4DDh2vu1V1caqmq2q2ZmZmRFULEnS9BlrwCc5BfjPwMuq6vvz9HlCkn12TgMnA7cO6itJkgYb5Z/JXQJ8CTgyydYkZwHnAvvQO+x+Y5Lzur5PS3Jlt+qBwDVJbgK+AlxRVZ8eVZ2SJLVoZOfgq2rdgOYL5un7LWBtN30XcPSo6pIkaXfgN9lJktQgA16SpAYZ8JIkNciAlySpQUMFfJIXDtMmSZJWhmH34N8/ZJskSVoBdvlnckmeD7wAmEnye32LngTsMcrCJEnS4i30d/B7Ak/s+u3T1/4QcMaoipIkSUuzy4Cvqi8AX0hyUVV9c0w1SZKkJRr2m+z2SrIRWNO/TlWdMIqiJEnS0gwb8B8HzgPOBx4ZXTmSJGk5DBvwO6rqgyOtRJIkLZth/0zur5P8hyQHJdl/52OklUmSpEUbdg9+fff8B31tBfzM8pYjSZKWw1ABX1WHj7oQSZK0fIYK+CS/Pai9qj68vOVIkqTlMOwh+uf2Te8NnAjcABjwkiStQMMeon9L/3ySfYGPjqQiSZK0ZIu9Xey/AJ6XlyRphRr2HPxf07tqHno3mfl54C9GVZQkSVqaYc/B/1Hf9A7gm1W1dQT1SJKkZTDUIfrupjNfp3dHuf2AH42yKEmStDRDBXySVwFfAX4deBVwbRJvFytJ0go17CH6dwDPrar7AJLMAH8DfGJUhUmSpMUb9ir6x+0M9879j2FdSZI0ZsPuwX86yVXAJd38q4ErR1OSJElaql0GfJJnAAdW1R8keSXwom7Rl4CLR12cJElanIX24P8EeDtAVX0S+CRAkl/olv3qSKuTJEmLstB59AOr6pa5jV3bmoU2nuTCJPclubWvbf8km5Lc0T3vN8+667s+dyRZP6iPJEkabKGA33cXy35qiO1fBJwyp+1s4LNVdQTw2W7+UZLsD7wLeB5wLPCu+X4RkCRJP2mhgN+c5N/PbUzyeuD6hTZeVV8EHpjTfDrwoW76Q8DLB6z6EmBTVT1QVd8BNvGTvyhIkqR5LHQO/m3AZUl+kx8H+iywJ/CKRb7mgVV1Tzf9z8CBA/ocDNzdN7+1a5MkSUPYZcBX1b3AC5IcDzyra76iqj63HC9eVZWkFu45vyQbgA0Aq1evXo6y/tWas69Y1u0tty3nvHTSJUiSVqhh7wd/NXD1Mr3mvUkOqqp7khwE3DegzzbguL75Q4DPz1PbRmAjwOzs7JJ+WZAkqRWT+Da6y4GdV8WvB/5qQJ+rgJOT7NddXHdy1yZJkoYw0oBPcgm9L8U5MsnWJGcB5wAvTnIHcFI3T5LZJOcDVNUDwB8C13WP93RtkiRpCMN+Ve2iVNW6eRadOKDvZuD1ffMXAheOqDRJkprmDWMkSWqQAS9JUoMMeEmSGmTAS5LUIANekqQGGfCSJDXIgJckqUEGvCRJDTLgJUlqkAEvSVKDDHhJkhpkwEuS1CADXpKkBhnwkiQ1yICXJKlBBrwkSQ0y4CVJapABL0lSgwx4SZIaZMBLktQgA16SpAYZ8JIkNciAlySpQQa8JEkNMuAlSWqQAS9JUoMMeEmSGmTAS5LUoLEHfJIjk9zY93goydvm9DkuyYN9fd457jolSZpmq8b9glV1O3AMQJI9gG3AZQO6/m1VnTbO2iRJasWkD9GfCPxjVX1zwnVIktSUSQf8mcAl8yx7fpKbknwqyTPHWZQkSdNuYgGfZE/gZcDHByy+ATisqo4G3g/85S62syHJ5iSbt2/fPppiJUmaMpPcgz8VuKGq7p27oKoeqqqHu+krgccnOWDQRqpqY1XNVtXszMzMaCuWJGlKTDLg1zHP4fkkT02SbvpYenXeP8baJEmaamO/ih4gyROAFwNv6Gv7XYCqOg84A3hjkh3AD4Azq6omUaskSdNoIgFfVf8CPGVO23l90+cC5467LkmSWjHpq+glSdIIGPCSJDXIgJckqUEGvCRJDTLgJUlqkAEvSVKDDHhJkhpkwEuS1CADXpKkBhnwkiQ1yICXJKlBBrwkSQ0y4CVJapABL0lSgwx4SZIaZMBLktQgA16SpAYZ8JIkNciAlySpQQa8JEkNMuAlSWqQAS9JUoMMeEmSGmTAS5LUIANekqQGGfCSJDXIgJckqUEGvCRJDZpYwCfZkuSWJDcm2TxgeZL8aZI7k9yc5DmTqFOSpGm0asKvf3xVfXueZacCR3SP5wEf7J4lSdICVvIh+tOBD1fPl4F9kxw06aIkSZoGkwz4Aj6T5PokGwYsPxi4u29+a9cmSZIWMMlD9C+qqm1JfhrYlOTrVfXFx7qR7peDDQCrV69e7hq1RGvOvmLSJSxoyzkvnXQJkrTsJrYHX1Xbuuf7gMuAY+d02QYc2jd/SNc2dzsbq2q2qmZnZmZGVa4kSVNlIgGf5AlJ9tk5DZwM3Dqn2+XAb3dX0/8S8GBV3TPmUiVJmkqTOkR/IHBZkp01/HlVfTrJ7wJU1XnAlcBa4E7g+8DrJlSrJElTZyIBX1V3AUcPaD+vb7qAN42zLkmSWrGS/0xOkiQtkgEvSVKDDHhJkhpkwEuS1CADXpKkBhnwkiQ1yICXJKlBBrwkSQ0y4CVJapABL0lSgyZ5u1gt0TTcilVaCabhZ8XbFmu5uQcvSVKDDHhJkhpkwEuS1CADXpKkBhnwkiQ1yICXJKlBBrwkSQ0y4CVJapABL0lSgwx4SZIaZMBLktQgA16SpAYZ8JIkNci7yUmShrLS78rnHfkezT14SZIaZMBLktQgA16SpAaNPeCTHJrk6iS3JflqkrcO6HNckgeT3Ng93jnuOiVJmmaTuMhuB/D7VXVDkn2A65Nsqqrb5vT726o6bQL1SZI09ca+B19V91TVDd3094CvAQePuw5Jklo20XPwSdYAzwauHbD4+UluSvKpJM8ca2GSJE25if0dfJInApcCb6uqh+YsvgE4rKoeTrIW+EvgiHm2swHYALB69eoRVixJ0vSYyB58ksfTC/eLq+qTc5dX1UNV9XA3fSXw+CQHDNpWVW2sqtmqmp2ZmRlp3ZIkTYtJXEUf4ALga1X1x/P0eWrXjyTH0qvz/vFVKUnSdJvEIfoXAq8BbklyY9f2X4DVAFV1HnAG8MYkO4AfAGdWVU2gVkmSptLYA76qrgGyQJ9zgXPHU5EkSe3xm+wkSWqQAS9JUoO8Xax2eyv9FpiStBjuwUuS1CADXpKkBhnwkiQ1yICXJKlBBrwkSQ0y4CVJapABL0lSgwx4SZIaZMBLktQgA16SpAYZ8JIkNciAlySpQQa8JEkNMuAlSWqQt4uVJDVhGm79vOWcl47ttdyDlySpQQa8JEkNMuAlSWqQAS9JUoMMeEmSGmTAS5LUIANekqQGGfCSJDXIgJckqUEGvCRJDZpIwCc5JcntSe5McvaA5Xsl+Vi3/Noka8ZfpSRJ02vsAZ9kD+ADwKnAUcC6JEfN6XYW8J2qegbwPuC/jbdKSZKm2yT24I8F7qyqu6rqR8BHgdPn9Dkd+FA3/QngxCQZY42SJE21SQT8wcDdffNbu7aBfapqB/Ag8JSxVCdJUgOm/naxSTYAG7rZh5PcPsl6VqADgG9PuogVzjFa2C7HKJ5EgyX+P9pNxnC3/1kb4t/5sY7RYfMtmETAbwMO7Zs/pGsb1GdrklXAk4H7B22sqjYCG0dQZxOSbK6q2UnXsZI5RgtzjBbmGC3MMVrYco7RJA7RXwcckeTwJHsCZwKXz+lzObC+mz4D+FxV1RhrlCRpqo19D76qdiR5M3AVsAdwYVV9Ncl7gM1VdTlwAfCRJHcCD9D7JUCSJA1pIufgq+pK4Mo5be/sm/4h8OvjrqtRnr5YmGO0MMdoYY7RwhyjhS3bGMUj35IktcevqpUkqUEG/JQa4ut+35fkxu7xjSTf7Vu2Pskd3WP93HVbscQxeqRv2dyLQJsxxBitTnJ1kn9IcnOStX3L3t6td3uSl4y38vFZ7BglWZPkB33/j84bf/XjMcQYHZbks934fD7JIX3L/DxiwTFa3OdRVfmYsge9ixP/EfgZYE/gJuCoXfR/C72LGQH2B+7qnvfrpveb9HtaSWPUzT886fewEsaI3vnAN3bTRwFb+qZvAvYCDu+2s8ek39MKG6M1wK2Tfg8rZIw+Dqzvpk8APtJN+3m0wBh184v6PHIPfjoN83W//dYBl3TTLwE2VdUDVfUdYBNwykirnYyljNHuYpgxKuBJ3fSTgW9106cDH62q/1tV/wTc2W2vNUsZo93FMGN0FPC5bvrqvuV+Hv3YfGO0aAb8dBrm636B3mEfentYO//jDL3ulFvKGAHsnWRzki8nefnoypyoYcbo3cBvJdlK7y9f3vIY1m3BUsYI4PDu0P0XkvzySCudnGHG6Cbgld30K4B9kjxlyHVbsJQxgkV+Hhnw7TsT+ERVPTLpQlawQWN0WPW+Teo3gD9J8vTJlDZx64CLquoQYC2976fwc+PR5huje4DVVfVs4PeAP0/ypF1sp2X/CfiVJP8A/Aq9byv1M+nRdjVGi/o88gd1Og3zdb87ncmjDz0/lnWn2VLGiKra1j3fBXweePbylzhxw4zRWcBfAFTVl4C96X1Xtv+PfmzgGHWnL+7v2q+ndw72Z0de8fgtOEZV9a2qemX3y847urbvDrNuI5YyRov/PJr0xQc+FnXBxip6F6Mczo8v2HjmgH4/B2yh+76Drm1/4J/oXdCyXze9/6Tf0wobo/2AvbrpA4A72MUFetP6GGaMgE8Br+2mf57e+eUAz+TRF9ndRZsX2S1ljGZ2jgm9i6u27a4/a93P0eO66f8KvKeb9vNo4TFa9OfRxN+4j0X/h1kLfIPeXsE7urb3AC/r6/Nu4JwB6/4OvYui7gReN+n3stLGCHgBcEv3Q3gLcNak38ukxojehT9/143FjcDJfeu+o1vvduDUSb+XlTZGwK8BX+3abgB+ddLvZYJjdEYXTN8Azt8ZWN0yP492MUZL+Tzym+wkSWqQ5+AlSWqQAS9JUoMMeEmSGmTAS5LUIANekqQGGfDSbibJy5NUkp+bdC2SRseAl3Y/64BruueRSLLHqLYtaTgGvLQbSfJE4EX0vl71zK5tjyR/lOTW7l7Ub+nan5vk75PclOQrSfZJ8tok5/Zt738nOa6bfjjJe5PcBDw/yTuTXNdtd2OSdP2ekeRvuu3ekOTpST7cfxONJBcnWfLdtKTdmQEv7V5OBz5dVd8A7k/yb4EN9O5dfkxV/SJwcZI9gY8Bb62qo4GTgB8ssO0nANdW1dFVdQ1wblU9t6qeBfwUcFrX72LgA912X0DvpiwXAK8FSPLkrv2KZXrP0m7JgJd2L+vo3Yua7nkdvfD+n1W1A6CqHgCOBO6pquu6tod2Lt+FR4BL++aPT3JtkluAE4BnJtkHOLiqLuu2+8Oq+n5VfQE4IslMV9OlQ7yepF1YNekCJI1Hkv3pBe0vJClgD6CA6x7DZnbw6B2Dvfumf1jdLXeT7A38D2C2qu5O8u45fQf5MPBb9E4dvO4x1CRpAPfgpd3HGcBHquqwqlpTVYfSu3vXTcAbkqyCf/1F4HbgoCTP7dr26ZZvAY5J8rgkhwLHzvNaO8P82915/zMAqup7wNad59uT7JXk33R9LwLe1vW7bRnft7RbMuCl3cc64LI5bZcCBwH/B7i5u0DuN6rqR8Crgfd3bZvohfbf0ful4DbgT+ndJe0nVO8+1n8G3ApcxaOPErwG+I9Jbgb+Hnhqt869wNeA/7XkdyrJu8lJWhm6PflbgOdU1YOTrkeadu7BS5q4JCfR23t/v+EuLQ/34CVJapB78JIkNciAlySpQQa8JEkNMuAlSWqQAS9JUoMMeEmSGvT/AfucVoi/M3FTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 6))\n",
    "plt.hist(test_accs)\n",
    "plt.xlabel(\"Accuracy\")\n",
    "plt.ylabel(\"Count\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33",
   "metadata": {},
   "source": [
    "The histogram shown above indicates the difficulty of training a good model on the MUTAG dataset due to the following factors,\n",
    "- small amount of available data, i.e., only 188 graphs\n",
    "- small amount of validation data since for a single fold only 19 graphs are used for validation\n",
    "- the data are unbalanced since the majority class is twice as prevalent in the data\n",
    "\n",
    "Given the above, average performance as estimated using repeated 10-fold cross validation displays high variance but overall good performance for a straightforward application of graph convolutional neural networks to supervised graph classification. The high variance is likely the result of the small dataset size.\n",
    "\n",
    "Generally, performance is a bit lower than SOTA in recent literature. However, we have not tuned the model for the best performance possible so some improvement over the current baseline may be attainable.\n",
    "\n",
    "When comparing to graph kernel-based approaches, our straightforward GCN with mean pooling graph classification model is competitive with the WL kernel being the exception.\n",
    "\n",
    "For comparison, some performance numbers repeated from [3] for graph kernel-based approaches are, \n",
    "- Graphlet Kernel (GK): $81.39\\pm1.74$\n",
    "- Random Walk Kernel (RW): $79.17\\pm2.07$\n",
    "- Propagation Kernel (PK): $76.00\\pm2.69$\n",
    "- Weisfeiler-Lehman Subtree Kernel (WL): $84.11\\pm1.91$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34",
   "metadata": {
    "nbsphinx": "hidden",
    "tags": [
     "CloudRunner"
    ]
   },
   "source": [
    "<table><tr><td>Run the latest release of this notebook:</td><td><a href=\"https://mybinder.org/v2/gh/stellargraph/stellargraph/master?urlpath=lab/tree/demos/graph-classification/gcn-supervised-graph-classification.ipynb\" alt=\"Open In Binder\" target=\"_parent\"><img src=\"https://mybinder.org/badge_logo.svg\"/></a></td><td><a href=\"https://colab.research.google.com/github/stellargraph/stellargraph/blob/master/demos/graph-classification/gcn-supervised-graph-classification.ipynb\" alt=\"Open In Colab\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\"/></a></td></tr></table>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}