Skip to content

Robust way to explore GAN model latent space on web using three.js & t-SNE

Notifications You must be signed in to change notification settings

pandas9/gan-latent-space-tsne-js

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

87 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

GAN Latent Space Web Explorer w/ t-SNE Visualization

gan-latent-space-web-explorer

Robust way to explore GAN model latent space on web using t-SNE statistical method
GAN explorer example

This is example on how you can create latent space web explorer for any GAN model using js and python
Example is based on three.js library, t-SNE statistical method and vanilla js

t-SNE allows us to position images near other similar-looking images in the 2D coordinate space and three.js library to build a WebGL-powered visualization that can display tens of thousands of images in an interactive 3D environment

Inside examples folder there is celeba_gan_e30.h5 that has been trained on celeba dataset in case someone wants to try and build custom GAN explorer but doesn't have machines to train model

Run Locally

  1. clone github project and set githubPages to false inside main.js
  2. npm install
    command inside project root folder
  3. node app.js
    serve app using node

Build Gan Explorer For Your Model

  1. Clone the project and run calculate_latent_size.py and set variables inside file that will fit your model this is important for atlases and how your latent space is visualized with three.js
    python calculate_latent_size.py

  2. Using your GAN model generate latent points and save each as image i.e

# example.py

def generate_images(examples, n):
	for i in range(n):
		pyplot.imsave(f'./faces/generated_face_{i}.png', examples[i, :, :])

model = load_model('celeba_gan_e30.h5')
n = 35301
latent_points = generate_latent_points(100, n)
X  = model.predict(latent_points)
X = (X + 1) / 2.0
generate_images(X, n)
  1. Use classify_images.py to create vectorized representation of each image
    python classify_images.py 'faces/*'

  2. Run create_tsne_projection.py to get t-SNE coordinates
    python create_tsne_projection.py
    inside file make sure to change latent_size and other variables same goes for other files below

  3. Split latent space images into equal parts for atlas
    python split_latent_space.py

  4. Generate atlases and serve the app
    python create_atlases.py

    by now you should have image_tsne_projections.json and images_..jpg those are your atlases next is to create new folder inside /src/assets/ and copy generated files inside, create new config config.js inside /src/ folder that will fit your model and add inside main.js - index.html option to have your config selectable inside web page

    serve app using node node app.js and visit localhost:3000 to explore your GAN latent space

Requirements

python3
node.js
imagemagick

To do

  • Port to React.js
  • Convert classify_images.py to tf2 and have script only create vectorized representations of each image
  • Add 3D visualization
  • Allow objects to be interactive for interpolation / zoom / vector arithmetic ... etc
  • Dynamic atlas size
  • Indication of load progress
  • Guide through the visualization

Acknowledgements

machinelearningmastery.com/how-to-interpolate-and-perform-vector-arithmetic-with-faces-using-a-generative-adversarial-network/
medium.com/mlearning-ai/latent-space-representation-a-hands-on-tutorial-on-autoencoders-in-tensorflow
douglasduhaime.com/posts/visualizing-tsne-maps-with-three-js
github.com/YaleDHLab/pix-plot
artsexperiments.withgoogle.com/tsnemap
towardsdatascience.com/understanding-latent-space-in-machine-learning-de5a7c687d8d

About

Robust way to explore GAN model latent space on web using three.js & t-SNE

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published