Skip to content
This repository has been archived by the owner on Jan 2, 2021. It is now read-only.
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: alexjc/neural-enhance
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v0.1
Choose a base ref
...
head repository: alexjc/neural-enhance
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref
Loading
Showing with 371 additions and 141 deletions.
  1. +67 −17 README.rst
  2. +34 −0 docker-cpu.df
  3. +33 −0 docker-gpu.df
  4. BIN docs/EnhanceCSI_example.png
  5. BIN docs/{Chinatown_example.gif → StreetView_example.gif}
  6. +201 −124 enhance.py
  7. +18 −0 train/ne1x-photo-repair.sh
  8. +18 −0 train/ne2x-photo-default.sh
84 changes: 67 additions & 17 deletions README.rst
Original file line number Diff line number Diff line change
@@ -3,13 +3,13 @@ Neural Enhance

.. image:: docs/OldStation_example.gif

**Example #1** — Old Station: `view comparison <http://5.9.70.47:4141/w/0f5177f4-9ce6-11e6-992c-c86000be451f/view>`_ in 24-bit HD, `original photo <https://flic.kr/p/oYhbBv>`_ CC-BY-SA @siv-athens.
**Example #1** — Old Station: `view comparison <http://enhance.nucl.ai/w/0f5177f4-9ce6-11e6-992c-c86000be451f/view>`_ in 24-bit HD, `original photo <https://flic.kr/p/oYhbBv>`_ CC-BY-SA @siv-athens.

----

`As seen on TV! <https://www.youtube.com/watch?v=LhF_56SxrGk>`_ What if you could increase the resolution of your photos using technology from CSI laboratories? Thanks to deep learning and ``#NeuralEnhance``, it's now possible to train a neural network to zoom in to your images at 2x or even 4x. You'll get even better results by increasing the number of neurons or training with a dataset similar to your low resolution image.

The catch? The neural network is hallucinating details based on its training from example images. It's not reconstructing your photo exactly as it would have been if it was HD. That's only possible in Holywood — but using deep learning as "Creative AI" works and its just as cool! Here's how you can get started...
The catch? The neural network is hallucinating details based on its training from example images. It's not reconstructing your photo exactly as it would have been if it was HD. That's only possible in Hollywood — but using deep learning as "Creative AI" works and it is just as cool! Here's how you can get started...

1. `Examples & Usage <#1-examples--usage>`_
2. `Installation <#2-installation--setup>`_
@@ -19,6 +19,9 @@ The catch? The neural network is hallucinating details based on its training fro

|Python Version| |License Type| |Project Stars|

.. image:: docs/EnhanceCSI_example.png
:target: http://enhance.nucl.ai/w/8581db92-9d61-11e6-990b-c86000be451f/view

1. Examples & Usage
===================

@@ -32,16 +35,26 @@ The default is to use ``--device=cpu``, if you have NVIDIA card setup with CUDA
1.a) Enhancing Images
---------------------

A list of example command lines you can use with the pre-trained models provided in the GitHub releases:

.. code:: bash
# Run the super-resolution script for one image.
python3 enhance.py example.png
# Run the super-resolution script to repair JPEG artefacts, zoom factor 1:1.
python3 enhance.py --type=photo --model=repair --zoom=1 broken.jpg
# Process multiple good quality images with a single run, zoom factor 2:1.
python3 enhance.py --type=photo --zoom=2 file1.jpg file2.jpg
# Also process multiple files with a single run.
python3 enhance.py file1.jpg file2.jpg
# Display output images that were given `_ne?x.png` suffix.
open *_ne?x.png
# Display output images that were given `_ne4x.png` suffix.
open *_ne4x.png
Here's a list of currently supported models, image types, and zoom levels in one table.

================== ===================== ==================== ===================== ====================
FEATURES ``--model=default`` ``--model=repair`` ``--model=denoise`` ``--model=deblur``
================== ===================== ==================== ===================== ====================
``--type=photo`` 2x 1x … …
================== ===================== ==================== ===================== ====================


1.b) Training Super-Resolution
@@ -52,31 +65,61 @@ Pre-trained models are provided in the GitHub releases. Training your own is a
.. code:: bash
# Remove the model file as don't want to reload the data to fine-tune it.
rm -f ne4x*.pkl.bz2
rm -f ne?x*.pkl.bz2
# Pre-train the model using perceptual loss from paper [1] below.
python3.4 enhance.py --train --scales=2 --epochs=50 \
python3.4 enhance.py --train "data/*.jpg" --model custom --scales=2 --epochs=50 \
--perceptual-layer=conv2_2 --smoothness-weight=1e7 --adversary-weight=0.0 \
--generator-blocks=4 --generator-filters=64
# Train the model using an adversarial setup based on [4] below.
python3.4 enhance.py --train --scales=2 --epochs=250 \
python3.4 enhance.py --train "data/*.jpg" --model custom --scales=2 --epochs=250 \
--perceptual-layer=conv5_2 --smoothness-weight=2e4 --adversary-weight=1e3 \
--generator-start=5 --discriminator-start=0 --adversarial-start=5 \
--discriminator-size=64
# The newly trained model is output into this file...
ls ne?x-custom-*.pkl.bz2
.. image:: docs/BankLobby_example.gif

**Example #2** — Bank Lobby: `view comparison <http://5.9.70.47:4141/w/38d10880-9ce6-11e6-becb-c86000be451f/view>`_ in 24-bit HD, `original photo <https://flic.kr/p/6a8cwm>`_ CC-BY-SA @benarent.
**Example #2** — Bank Lobby: `view comparison <http://enhance.nucl.ai/w/38d10880-9ce6-11e6-becb-c86000be451f/view>`_ in 24-bit HD, `original photo <https://flic.kr/p/6a8cwm>`_ CC-BY-SA @benarent.

2. Installation & Setup
=======================

2.a) Using Docker Image [recommended]
-------------------------------------

(work in progress)
The easiest way to get up-and-running is to `install Docker <https://www.docker.com/>`_. Then, you should be able to download and run the pre-built image using the ``docker`` command line tool. Find out more about the ``alexjc/neural-enhance`` image on its `Docker Hub <https://hub.docker.com/r/alexjc/neural-enhance/>`_ page.

Here's the simplest way you can call the script using ``docker``, assuming you're familiar with using ``-v`` argument to mount folders you can use this directly to specify files to enhance:

.. code:: bash
# Download the Docker image and show the help text to make sure it works.
docker run --rm -v `pwd`:/ne/input -it alexjc/neural-enhance --help
**Single Image** — In practice, we suggest you setup an alias called ``enhance`` to automatically expose the folder containing your specified image, so the script can read it and store results where you can access them. This is how you can do it in your terminal console on OSX or Linux:

.. code:: bash
# Setup the alias. Put this in your .bashrc or .zshrc file so it's available at startup.
alias enhance='function ne() { docker run --rm -v "$(pwd)/`dirname ${@:$#}`":/ne/input -it alexjc/neural-enhance ${@:1:$#-1} "input/`basename ${@:$#}`"; }; ne'
# Now run any of the examples above using this alias, without the `.py` extension.
enhance --zoom=1 --model=repair images/broken.jpg
**Multiple Images** — To enhance multiple images in a row (faster) from a folder or wildcard specification, make sure to quote the argument to the alias command:

.. code:: bash
# Process multiple images, make sure to quote the argument!
enhance --zoom=2 "images/*.jpg"
If you want to run on your NVIDIA GPU, you can instead change the alias to use the image ``alexjc/neural-enhance:gpu`` which comes with CUDA and CUDNN pre-installed. Then run it within `nvidia-docker <https://github.com/NVIDIA/nvidia-docker>`_ and it should use your physical hardware!


2.b) Manual Installation [developers]
-------------------------------------
@@ -104,6 +147,7 @@ After this, you should have ``pillow``, ``theano`` and ``lasagne`` installed in

.. image:: docs/Faces_example.png

**Example #3** — Specialized super-resolution for faces, trained on HD examples of celebrity faces only. The quality is significantly higher when narrowing the domain from "photos" in general.

3. Background & Research
========================
@@ -115,14 +159,20 @@ This code uses a combination of techniques from the following papers, as well as
3. `Deeply-Recursive Convolutional Network for Image Super-Resolution <https://arxiv.org/abs/1511.04491>`_
4. `Photo-Realistic Super-Resolution Using a Generative Adversarial Network <https://arxiv.org/abs/1609.04802>`_

Special thanks for their help and support in various ways:

* Eder Santana — Discussions, encouragement, and his ideas on `sub-pixel deconvolution <https://github.com/Tetrachrome/subpixel>`_.
* Andrew Brock — This sub-pixel layer code is based on `his project repository <https://github.com/ajbrock/Neural-Photo-Editor>`_ using Lasagne.
* Casper Kaae Sønderby — For suggesting a more stable alternative to sigmoid + log as GAN loss functions.


4. Troubleshooting Problems
===========================

Can't install or Unable to find pgen, not compiling formal grammar.
-------------------------------------------------------------------

There's a Python extension compiler called Cython, and it's missing or inproperly installed. Try getting it directly from the system package manager rather than PIP.
There's a Python extension compiler called Cython, and it's missing or improperly installed. Try getting it directly from the system package manager rather than PIP.

**FIX:** ``sudo apt-get install cython3``

@@ -146,13 +196,13 @@ You need to install Lasagne and Theano directly from the versions specified in `
ValueError: unknown locale: UTF-8
---------------------------------

It seems your terminal is misconfigured and not compatible with the way Python treats locales. You may need to change this in your ``.bash_rc`` or other startup script. Alternatively, this command will fix it once for this shell instance.
It seems your terminal is misconfigured and not compatible with the way Python treats locales. You may need to change this in your ``.bashrc`` or other startup script. Alternatively, this command will fix it once for this shell instance.

**FIX:** ``export LC_ALL=en_US.UTF-8``

.. image:: docs/Chinatown_example.gif
.. image:: docs/StreetView_example.gif

**Example #3** — China Town: `view comparison <http://5.9.70.47:4141/w/3b3c8054-9d00-11e6-9558-c86000be451f/view>`_ in 24-bit HD, `original photo <https://flic.kr/p/gnxcXH>`_ CC-BY-SA @cyalex.
**Example #4** — Street View: `view comparison <http://enhance.nucl.ai/w/3b3c8054-9d00-11e6-9558-c86000be451f/view>`_ in 24-bit HD, `original photo <https://flic.kr/p/gnxcXH>`_ CC-BY-SA @cyalex.

----

34 changes: 34 additions & 0 deletions docker-cpu.df
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
FROM ubuntu:14.04

# Install dependencies
RUN apt-get -qq update && \
apt-get -qq install --assume-yes \
"build-essential" \
"git" \
"wget" \
"libopenblas-dev" \
"liblapack-dev" \
"pkg-config" && \
rm -rf /var/lib/apt/lists/*

# Miniconda.
RUN wget --quiet https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \
/bin/bash ~/miniconda.sh -b -p /opt/conda && \
rm ~/miniconda.sh

# Install requirements before copying project files
WORKDIR /ne
COPY requirements.txt .
RUN /opt/conda/bin/conda install -q -y conda numpy scipy pip pillow
RUN /opt/conda/bin/python3.5 -m pip install -q -r "requirements.txt"

# Copy only required project files
COPY enhance.py .

# Get a pre-trained neural networks, non-commercial & attribution.
RUN wget -q "https://github.com/alexjc/neural-enhance/releases/download/v0.3/ne1x-photo-deblur-0.3.pkl.bz2"
RUN wget -q "https://github.com/alexjc/neural-enhance/releases/download/v0.3/ne1x-photo-repair-0.3.pkl.bz2"
RUN wget -q "https://github.com/alexjc/neural-enhance/releases/download/v0.3/ne2x-photo-default-0.3.pkl.bz2"
RUN wget -q "https://github.com/alexjc/neural-enhance/releases/download/v0.3/ne4x-photo-default-0.3.pkl.bz2"
# Set an entrypoint to the main enhance.py script
ENTRYPOINT ["/opt/conda/bin/python3.5", "enhance.py", "--device=cpu"]
33 changes: 33 additions & 0 deletions docker-gpu.df
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
FROM nvidia/cuda:8.0-cudnn5-devel

# Install dependencies
RUN apt-get -qq update && \
apt-get -qq install --assume-yes \
"build-essential" \
"git" \
"wget" \
"pkg-config" && \
rm -rf /var/lib/apt/lists/*

# Miniconda.
RUN wget --quiet https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \
/bin/bash ~/miniconda.sh -b -p /opt/conda && \
rm ~/miniconda.sh

# Install requirements before copying project files
WORKDIR /ne
COPY requirements.txt .
RUN /opt/conda/bin/conda install -q -y conda numpy scipy pip pillow
RUN /opt/conda/bin/python3.5 -m pip install -q -r "requirements.txt"

# Copy only required project files
COPY enhance.py .

# Get a pre-trained neural networks, non-commercial & attribution.
RUN wget -q "https://github.com/alexjc/neural-enhance/releases/download/v0.3/ne1x-photo-deblur-0.3.pkl.bz2"
RUN wget -q "https://github.com/alexjc/neural-enhance/releases/download/v0.3/ne1x-photo-repair-0.3.pkl.bz2"
RUN wget -q "https://github.com/alexjc/neural-enhance/releases/download/v0.3/ne2x-photo-default-0.3.pkl.bz2"
RUN wget -q "https://github.com/alexjc/neural-enhance/releases/download/v0.3/ne4x-photo-default-0.3.pkl.bz2"

# Set an entrypoint to the main enhance.py script
ENTRYPOINT ["/opt/conda/bin/python3.5", "enhance.py", "--device=gpu"]
Binary file added docs/EnhanceCSI_example.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes
Loading