DeepLens is an open-source differentiable ray tracing framework for automated optical design, end-to-end optics-network co-design, and computational photography. DeepLens enables researchers and engineers to build custom optical systems and end-to-end imaging pipelines with fully differentiable optimization.

News

Please contact Xinge Yang ([email protected]) for any questions, assistance, or collaboration.

• We now have a slack group. Welcome to join the discussion via this link.
• DeepLens is finall published on Nature Communications, check it here!
• We now have a WeChat group. Please contact Xinge Yang (singeryang1999) to join the discussion!

What is DeepLens

DeepLens aims to combines deep learning and optical design to create:

1. More powerful optical design algorithms enhanced by deep learning.
2. Next-generation computational cameras integrating optical encoding with deep learning decoding.

Key Features

DeepLens differs from others in the following aspects:

1. Open-source ray tracer with accuracy aligned with commercial software.
2. Differentiable optimization providing outstanding design capabilities.
3. Image simulation for camera sensors and image signal processing (ISP), enabling end-to-end optics-network co-design.

Additional features available via request or collaboration:

1. Memory-efficient ray-tracing capable of handling millions of rays on a desktop machine, with strategies to scale up further.
2. Physical optics simulation including phase and polarization tracing.
3. Neural representation to represent camera lenses as networks.
4. Complex optical systems including non-sequential and non-coaxial optical models.
5. Large-scale optimization with multi-GPU parallelization.

Applications

1. Automated lens design

Fully automated lens design from scratch. Try it at AutoLens!

2. End-to-End lens design

Lens-network co-design from scratch using final images (or classification/detection/segmentation) as objective.

3. Implicit Lens Representation

A surrogate network for fast (aberration + defocus) image simulation.

4. Hybrid Refractive-Difractive Lens Model

Design hybrid refractive-diffractive lenses with a new ray-wave model.

How to use

Here are two methods to use deeplens in your research:

Method 1 (recommended)

Clone this repo and write your code inside it.

git clone deeplens
cd deeplens
python 0_hello_deeplens.py
python your_optical_design_pipeline.py

Directory

deeplens/
│
├── deeplens/
│   ├── optics/ (core functions for optical components)
|   ├── network/ (image restoration and implicit representation networks)
|   ├── geolens.py (refractive lensgroup using ray tracing)
|   ├── diffraclens.py (diffractive lensgroup using wave propagation)
|   └── your_own_optical_system.py (your own optical system)
│
├── ... (other files)
├── 0_hello_deeplens.py (main scripts)
└── your_optical_design_pipeline.py (your own optical design pipeline)

Method 2

Clone the repo and install deeplens as a python package.

git clone deeplens
pip install -e ./deeplens

Then in your code:

import deeplens
lens = deeplens.GeoLens(filename='./lenses/cellphone80deg.json')

Reference

Citations

DeepLens is first developed by Dr. Congli Wang (previously named dO), then developed and maintained by Xinge Yang. If you use DeepLens in your research, please cite the corresponding papers:

• [TCI 2022] dO: A differentiable engine for deep lens design of computational imaging systems. Paper, BibTex
• [NatComm 2024] Curriculum Learning for ab initio Deep Learned Refractive Optics. Paper, BibTex
• [SiggraphAsia 2024] End-to-End Hybrid Refractive-Diffractive Lens Design with Differentiable Ray-Wave Model. Paper, BibTex

Projects built on top of DeepLens/dO

(If you donot want to list your paper here, we can remove it.)

• [SiggraphAsia 2024] End-to-end Optimization of Fluidic Lenses. Paper, BibTex
• [TPAMI 2023] Aberration-Aware Depth-From-Focus. Paper, BibTex
• [Arxiv 2024] Centimeter-Scale Achromatic Hybrid Metalens Design: A New Paradigm Based on Differentiable Ray Tracing in the Visible Spectrum. Paper