Skip to content

pallets is a project to explore pixel art generation in neural networks using toy model building blocks

Notifications You must be signed in to change notification settings

jmsdnns/pallets

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Pallets

A cute robot staring at blocks laying on the floor. The background is made of pieces of wood stained in the different bright colors

Pallets is a project to explore pixel art generation in neural networks using toy model building blocks.

Abstract

We explore the application of convolutional autoencoders to generate pixel art. Our approach is twofold: first, we implement a standard RGB-based autoencoder model to understand baseline performance and highlight issues with current approaches - including those using very large generative models. Second, we introduce a novel one-hot encoded color mapping autoencoder designed to adhere strictly to predefined color palettes, a critical aspect of pixel art.

We demonstrate a dimensionality reduction of the problem alongside order of magnitude performance improvement relative to the aesthetic of pixel art. We then suggest that this can be more broadly applied on larger images, more complex model architectures and ambitious generative pieces.

Notebooks

The Dataset

Autoencoders

  • AE: A basic autoencoder for images with 4 color channels for RGBA
  • ConvAE: A convolutionary form of AE
  • AEOneHot: A basic autoencoder for images with a one-hot encoded representation of each unique color in the dataset
  • ConvAEOneHot: A convolutionary form of AEOneHot

Variational Autoencoders

  • VAE: A basic variational autoencoder for one-hot encoded images
  • ConvVAE: A convolutionary form of VAE
  • LabeledVAE: A labeled form of VAE
  • LabeledConvVae: A combination convolutionary & labeled form of VAE

Conditional Variational Autoencoders

  • CVAE: A basic conditional variational autoencoder for one-hot encoded images
  • ConvCVAE: A convolutionary form of VAE
  • LabeledCVAE: A labeled form of VAE
  • LabeledConvCVae: A combination convolutionary & labeled form of VAE

Gumbel Softmax

  • GSVAE: A basic variational autoencoder with gumbel softmax reparameterization
  • LabeledGSVAE: A labeled form of GSVAE

Math Visualizations

  • Visualizing Convolutions: Applies different types of convolutions to graphical representations of an X and an O, and displays an image for every layer of math applied.
  • Notes On The Simplex: A visual representation of a simplex, the structure that underlines Gumbel Softmax.
  • Using Gumbel Softmax: An implementation of Eric Jang's gumbel softax (one of the original authors of Gumbel Softmax) is compared with the implementation packaged with PyTorch.

Setup

Get both the cpunks-10k repo and this one

git clone https://github.com/tnn1t1s/cpunks-10k
git clone https://github.com/jmsdnns/pallets

Then setup a venv for pallets and install it

cd pallets
python -mvenv venv
source venv/bin/activate
pip install -r requirements.txt

Credits

A huge shoutout to @tnn1t1s for starting this project and for mentoring me as I study machine learning.

About

pallets is a project to explore pixel art generation in neural networks using toy model building blocks

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published