colab_online.ipynb

﻿{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "[![GitHub stars](https://img.shields.io/github/stars/EssayKillerBrain/EssayKiller?style=social)](https://github.com/imcaspar/gpt2-ml)\n",
        "[![GitHub](https://img.shields.io/github/license/EssayKillerBrain/EssayKiller)](https://github.com/imcaspar/gpt2-ml)\n",
        "[![Twitter URL](https://img.shields.io/twitter/url?style=social&url=https%3A%2F%2Fgithub.com%2Fimcaspar%2Fgpt2-ml)](https://twitter.com/intent/tweet?text=Wow:&url=https://github.com/imcaspar/gpt2-ml)\n",
        "### 高考作文生成指南:\n",
        "\n",
        "* 点击代码框左上角的▶️，选择RUN ANYWAY\n",
        "* 等待对应代码文件、模型文件下载\n",
        "* 输入参数，代表长度\n",
        "* 输入作文题目（摘要模块没有部署，请自己提炼作文题中的主旨句）\n",
        "\n",
        "之后就会生成对应文章，效果如下:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "#@title #参数与开源框架预加载\n",
        "!pip install pandas==0.24.2 &> tmp.log\n",
        "!pip install regex==2019.4.14 &> tmp.log\n",
        "!pip install h5py==2.10.0 &> tmp.log\n",
        "!pip install numpy==1.18.4 &> tmp.log\n",
        "!pip install tensorboard==1.15.0 &> tmp.log\n",
        "!pip install tensorflow-gpu==1.15.2 &> tmp.log\n",
        "!pip install tensorflow-estimator==1.15.1 &> tmp.log\n",
        "!pip install tqdm==4.31.1 &> tmp.log\n",
        "!pip install requests==2.22.0 &> tmp.log\n",
        "!pip install ujson==2.0.3 &> tmp.log\n",
        "!echo '预加载模块结束，请继续点击下方的▶️'\n",
        "#@title #Prerequisites"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "#@title #模型框架加载模块\n",
        "%cd /home\n",
        "!git clone -b master https://github.com/EssayKillerBrain/EssayKiller_V2.git \n",
        "\n",
        "#%tensorflow_version 1.5.2-GPU/TPU\n",
        "!mkdir -p /home/EssayKiller_V2/LanguageNetwork/GPT2/finetune/trained_models\n",
        "\n",
        "%cd /home/EssayKiller_V2/LanguageNetwork/GPT2/finetune/\n",
        "#!perl /home/EssayKiller_V2/LanguageNetwork/GPT2/scripts/gdown.pl https://drive.google.com/open?id=1ujWYTOvRLGJX0raH-f-lPZa3-RN58ZQx trained_models/model.ckpt-220000.data-00000-of-00001\n",
        "!perl /home/EssayKiller_V2/LanguageNetwork/GPT2/scripts/gdown.pl https://drive.google.com/open?id=1mT_qCQg4AWnAXTwKfsyyRWCRpgPrBJS3 trained_models/model.ckpt-280000.data-00000-of-00001\n",
        "#!python3 scripts/down_gdrive_file.py -file_id='1ujWYTOvRLGJX0raH-f-lPZa3-RN58ZQx' -file_path='trained_models/model.ckpt-280000.data-00000-of-00001'　&> predict.log\n"
        "!wget -q --show-progress https://github.com/EssayKillerBrain/EssayKiller/releases/download/v1.0/model.ckpt-280000.index -P /home/EssayKiller_V2/LanguageNetwork/GPT2/finetune/models/mega\n",
        "!wget -q --show-progress https://github.com/EssayKillerBrain/EssayKiller/releases/download/v1.0/model.ckpt-280000.meta -P /home/EssayKiller_V2/LanguageNetwork/GPT2/finetune/models/mega\n",
        "\n",
        "!echo '模型下载完成，Git项目已构建，请继续点击下方的▶️'\n",
        "# If gdown.pl failed, please uncomment following code & exec"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 模型参数设置:\n",
        "\n",
        "* 1.作文最小篇幅：\n",
        "生成对应字数的高考作文，可自己调节长度，最长为1024个汉字。\n",
        "一般来说越短的文章AI表现越好。\n",
        "\n",
        "* 2.生成作文篇数：\n",
        "默认会生成1篇议论文，生成时间取决于服务器状态\n",
        "一般不超过60秒。受限于线上GPU内存，篇数最多为100。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "#!cat /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/tpu/tpu_feed.py\n",
        "#@title #文章生成模块\n",
        "作文最小篇幅 = 1024 #@param {type:\"number\", min:800, max:1024, step:1}\n",
        "生成作文篇数 = 1 #@param {type:\"number\", min:1, max:100, step:1}\n",
        "%mv /home/EssayKiller_V2/LanguageNetwork/GPT2/finetune/models/mega/* /home/EssayKiller_V2/LanguageNetwork/GPT2/finetune/trained_models/\n",
        "%cd /home/EssayKiller_V2/LanguageNetwork/GPT2/\n",
        "!export TF_CPP_MIN_LOG_LEVEL=2\n",
        "!echo '模型加载中，请稍后......'\n",
        "!PYTHONPATH=$(pwd) python scripts/demo.py -ckpt_fn finetune/trained_models/model.ckpt-280000 -min_len $作文最小篇幅 -samples $生成作文篇数\n",
        "!PYTHONPATH=$(pwd) python scripts/formatter.py -org_text result.txt"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "17亿参数-高考作文生成AI | 1.7B GPT2 Pretrained Essay Killer Brain",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}