-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
0 parents
commit b25e63e
Showing
25 changed files
with
3,455 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
*.log | ||
temp/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
MIT License | ||
|
||
Copyright (c) 2023 Threedle (University of Chicago) | ||
|
||
Permission is hereby granted, free of charge, to any person obtaining a copy | ||
of this software and associated documentation files (the "Software"), to deal | ||
in the Software without restriction, including without limitation the rights | ||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
copies of the Software, and to permit persons to whom the Software is | ||
furnished to do so, subject to the following conditions: | ||
|
||
The above copyright notice and this permission notice shall be included in all | ||
copies or substantial portions of the Software. | ||
|
||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
SOFTWARE. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
# MeshUp | ||
This is the repository that contains source code for the [MeshUp website](https://threedle.github.io/meshup). | ||
|
||
If you find MeshUp useful for your work please cite: | ||
``` | ||
@misc{kim2024meshupmultitargetmeshdeformation, | ||
title={MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation}, | ||
author={Hyunwoo Kim and Itai Lang and Noam Aigerman and Thibault Groueix and Vladimir G. Kim and Rana Hanocka}, | ||
year={2024}, | ||
eprint={2408.14899}, | ||
archivePrefix={arXiv}, | ||
primaryClass={cs.CV}, | ||
url={https://arxiv.org/abs/2408.14899}, | ||
} | ||
``` | ||
|
||
# Website License | ||
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,332 @@ | ||
<!DOCTYPE html> | ||
<html> | ||
<head> | ||
<meta charset="utf-8"> | ||
<meta name="description" | ||
content="Multi-Target Mesh Deformation via Blended Score Distillation"> | ||
<meta name="keywords" content="MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation"> | ||
<meta name="viewport" content="width=device-width, initial-scale=1"> | ||
<title>3D Paintbrush</title> | ||
|
||
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" | ||
rel="stylesheet"> | ||
|
||
<link rel="stylesheet" href="./static/css/bulma.min.css"> | ||
<link rel="stylesheet" href="./static/css/bulma-carousel.min.css"> | ||
<link rel="stylesheet" href="./static/css/bulma-slider.min.css"> | ||
<link rel="stylesheet" href="./static/css/fontawesome.all.min.css"> | ||
<link rel="stylesheet" | ||
href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> | ||
<link rel="stylesheet" href="./static/css/index.css"> | ||
<!-- <link rel="icon" href="./static/images/favicon.ico"> --> | ||
<link rel="icon" href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%220 0 100 100%22><text y=%22.9em%22 font-size=%2290%22>🎨</text></svg>"> | ||
|
||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script> | ||
<script defer src="./static/js/fontawesome.all.min.js"></script> | ||
<script src="./static/js/bulma-carousel.min.js"></script> | ||
<script src="./static/js/bulma-slider.min.js"></script> | ||
<script src="./static/js/index.js"></script> | ||
</head> | ||
<body> | ||
|
||
<nav class="navbar" role="navigation" aria-label="main navigation"> | ||
<div class="navbar-brand"> | ||
<a role="button" class="navbar-burger" aria-label="menu" aria-expanded="false"> | ||
<span aria-hidden="true"></span> | ||
<span aria-hidden="true"></span> | ||
<span aria-hidden="true"></span> | ||
</a> | ||
</div> | ||
<div class="navbar-menu"> | ||
<div class="navbar-start" style="flex-grow: 1; justify-content: center;"> | ||
<a class="navbar-item" href="https://3dl.cs.uchicago.edu/"> | ||
<span class="threedle-icon"></span> | ||
</a> | ||
</div> | ||
|
||
</div> | ||
</nav> | ||
|
||
|
||
<section class="hero"> | ||
<div class="hero-body"> | ||
<div class="container is-max-desktop"> | ||
<div class="columns is-centered"> | ||
<div class="column has-text-centered"> | ||
<h1 class="title is-1 publication-title"> MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation</h1> | ||
<div class="is-size-5 publication-authors"> | ||
<span class="author-block"> | ||
<a>Hyunwoo Kim</a><sup>1</sup>, | ||
</span> | ||
<span class="author-block"> | ||
<a>Itai Lang</a><sup>1</sup>, | ||
</span> | ||
<span class="author-block"> | ||
<a>Thibault Groueix</a><sup>2</sup>, | ||
</span> | ||
<span class="author-block"> | ||
<a>Noam Aigerman</a><sup>3</sup>, | ||
</span> | ||
<span class="author-block"> | ||
<a>Vladimir G. Kim</a><sup>2</sup>, | ||
</span> | ||
<span class="author-block"> | ||
<a>Rana Hanocka</a><sup>1</sup> | ||
</span> | ||
</div> | ||
|
||
<div class="is-size-5 publication-authors"> | ||
<span class="author-block"><sup>1</sup>University of Chicago</span> | ||
<span class="author-block"><sup>2</sup>Adobe research</span> | ||
<span class="author-block"><sup>3</sup>University of Montreal</span> | ||
</div> | ||
|
||
<div class="column has-text-centered"> | ||
<div class="publication-links"> | ||
<!-- PDF Link. --> | ||
|
||
<!-- arXiv Link. --> | ||
<span class="link-block"> | ||
<a href="https://arxiv.org/abs/2408.14899" | ||
class="external-link button is-normal is-rounded is-dark"> | ||
<span class="icon"> | ||
<i class="ai ai-arxiv"></i> | ||
</span> | ||
<span>arXiv</span> | ||
</a> | ||
</span> | ||
|
||
<!-- Code Link. --> | ||
<span class="link-block"> | ||
<a href="#" | ||
class="external-link button is-normal is-rounded is-dark"> | ||
<span class="icon"> | ||
<i class="fab fa-github"></i> | ||
</span> | ||
<span>Comming Soon</span> | ||
</a> | ||
</span> | ||
</div> | ||
|
||
</div> | ||
</div> | ||
</div> | ||
</div> | ||
</div> | ||
</section> | ||
|
||
<section class="hero teaser"> | ||
<div class="container is-max-desktop"> | ||
<div class="hero-body"> | ||
<!-- <image id="teaser"> | ||
<img src="./static/images/figures/teaser.png" | ||
type="image/png"> | ||
</image> --> | ||
<div style="display: flex; justify-content: space-between;"> | ||
<video poster="" id="rabbit" autoplay muted loop playsinline height="60%" width="60%"> | ||
<source src="./static/videos/rabbitanimation.gif" type="video/mp4"> | ||
</video> | ||
<video poster="" id="couch" autoplay muted loop playsinline height="60%" width="60%"> | ||
<source src="./static/videos/couchanimation.gif" type="video/mp4"> | ||
</video> | ||
<video poster="" id="moose" autoplay muted loop playsinline height="60%" width="60%"> | ||
<source src="./static/videos/mooseanimation.gif" type="video/mp4"> | ||
</video> | ||
</div> | ||
<h2 class="subtitle has-text-centered"> | ||
MeshUp is capable of deforming a source mesh into various concepts and into their weighted blends. The target objectives can | ||
be text prompts, images, or even meshes. Users can also input a set of control vertices to explicitly define where on the mesh each concept | ||
concept should be expressed. | ||
</h2> | ||
</div> | ||
</div> | ||
</section> | ||
|
||
|
||
<section class="section"> | ||
<div class="container is-max-desktop"> | ||
<!-- Abstract. --> | ||
<div class="columns is-centered has-text-centered"> | ||
<div class="column is-four-fifths"> | ||
<h2 class="title is-3">Abstract</h2> | ||
<div class="content has-text-justified"> | ||
<p> | ||
We propose MeshUp, a technique that deforms a 3D mesh towards multiple target | ||
concepts, and intuitively controls the region where each concept is expressed. | ||
Conveniently, the concepts can be defined as either text queries, e.g., "a dog" | ||
and "a turtle," or inspirational images, and the local regions can be selected | ||
as any number of vertices on the mesh. We can effectively control the influence | ||
of the concepts and mix them together using a novel score distillation | ||
approach, referred to as the Blended Score Distillation (BSD). BSD operates on | ||
each attention layer of the denoising U-Net of a diffusion model as it extracts | ||
and injects the per-objective activations into a unified denoising pipeline | ||
from which the deformation gradients are calculated. To localize the expression | ||
of these activations, we create a probabilistic Region of Interest (ROI) map on | ||
the surface of the mesh, and turn it into 3D-consistent masks that we use to | ||
control the expression of these activations. We demonstrate the effectiveness | ||
of BSD empirically and show that it can deform various meshes towards multiple | ||
objectives. | ||
</p> | ||
</div> | ||
</div> | ||
</div> | ||
<!--/ Abstract. --> | ||
|
||
|
||
<!-- Composite. --> | ||
<div class="columns is-centered has-text-centered"> | ||
<div class="column is-four-fifths"> | ||
<hr class="divider" /> | ||
<h2 class="title is-3">Precise composition of multiple local textures</h2> | ||
<div class="overview-image"> | ||
<img src="./static/images/figures/gallery.png" type="image/png"> | ||
</div> | ||
<div class="content has-text-justified"> | ||
<p> | ||
MeshUp takes as input a 3D mesh and several target objectives, and deforms the source mesh by optimizing the | ||
per-triangle Jacobians of the mesh. MeshUp produces a deformation that blends multiple concepts together, respective of the user-defined weights for each concept. | ||
Notice how the deformation smoothly transitions the source mesh into a mixture of different concepts, and weights dictate the magnitude of their expression. | ||
MeshUp supports as many targets as desired. | ||
</p> | ||
</div> | ||
</div> | ||
</div> | ||
<!--/ Composite. --> | ||
<!-- Gallery. --> | ||
<div class="columns is-centered has-text-centered"> | ||
<div class="column is-four-fifths"> | ||
<hr class="divider" /> | ||
<h2 class="title is-3">Blended Score Distillation</h2> | ||
<div class="overview-image"> | ||
<img src="./static/images/figures/system_local.png" type="image/png"> | ||
</div> | ||
<div class="content has-text-justified"> | ||
<p> | ||
MeshUp can also localize the deformation to specific regions on the mesh. The user can select a set of control vertices to define where on the mesh each concept should be expressed. | ||
The deformation (of one or multiple concepts) is then constrained to these regions, producing a more controlled and localized result. | ||
</p> | ||
</div> | ||
</div> | ||
</div> | ||
<!--/ Gallery. --> | ||
<!-- Gallery. --> | ||
<div class="columns is-centered has-text-centered"> | ||
<div class="column is-four-fifths"> | ||
<hr class="divider" /> | ||
<h2 class="title is-3">Localization</h2> | ||
<div class="overview-image"> | ||
<img src="./static/images/figures/system_local.png" type="image/png"> | ||
</div> | ||
<div class="content has-text-justified"> | ||
<p> | ||
MeshUp can also localize the deformation to specific regions on the mesh. The user can select a set of control vertices to define where on the mesh each concept should be expressed. | ||
The deformation (of one or multiple concepts) is then constrained to these regions, producing a more controlled and localized result. | ||
</p> | ||
</div> | ||
</div> | ||
</div> | ||
<!--/ Gallery. --> | ||
|
||
<!-- Overview. --> | ||
<div class="columns is-centered has-text-centered"> | ||
<div class="column is-four-fifths"> | ||
<hr class="divider" /> | ||
<h2 class="title is-3">Network Overview</h2> | ||
<div class="overview-image"> | ||
<img src="./static/images/figures/system.png" type="image/png"> | ||
</div> | ||
|
||
<div class="content has-text-justified"> | ||
<p> | ||
At each iteration, it creates parallel UNET branches, assigned each to the input target objectives (we call these the "Target Branches"). | ||
Then, it passes the same noised renderings and the corresponding text input through | ||
the UNet of a pretrained text-to-image model. On a different branch (the "Blending Branch"), it interpolates the activations | ||
extracted from the Target Branches, and interpolates them, respective to the assigned weights. Replacing the activations in the blending branch with the interpolated activations, | ||
MeshUp backpropagates the gradients from the blending branch via Score Distillation Sampling (SDS), updating the Jacobians of the mesh accordingly. | ||
</p> | ||
</div> | ||
</div> | ||
</div> | ||
<div class="columns is-centered has-text-centered"> | ||
<div class="column is-four-fifths"> | ||
<hr class="divider" /> | ||
<h2 class="title is-3">Localiztion Overview</h2> | ||
<div class="overview-image"> | ||
<img src="./static/images/figures/system_local.png" type="image/png"> | ||
</div> | ||
<div class="content has-text-justified"> | ||
<p> | ||
To locaalize the deformation, MeshUp creates a probabilistic Region of Interest (ROI) map on the surface of the mesh using the self-attention maps of the selected control vertices. | ||
It then uses this 3D ROI map to restrain the jacobians of the mesh, ensuring that the deformation is localized to the selected regions. | ||
For localization objectives with multiple targets, we rasterize the 3D ROI map of each target from the same viewpoint as the renderings, and use them as binary masks to control the expression of the activations. | ||
</p> | ||
</div> | ||
</div> | ||
</div> | ||
<!--/ Overview. --> | ||
|
||
|
||
|
||
<!--/ Comparison. --> | ||
<div class="columns is-centered has-text-centered"> | ||
<div class="column is-four-fifths"> | ||
<hr class="divider" /> | ||
<h2 class="title is-3">Comparison with Various Methods</h2> | ||
<div class="overview-image"> | ||
<img src="./static/images/figures/comparison.png" type="image/png"> | ||
</div> | ||
<div class="content has-text-justified"> | ||
<p> | ||
Compared to the state-of-the-art text-to-3D generative models, MeshUp produces results that have both better triangulation and geometric details. | ||
Our method is also first to support local deformation capabilities, which not only gurantees the preservation of unselected regions, but also greatly surpasses the quality of meshes generated solely using text descriptions. | ||
</p> | ||
</div> | ||
</div> | ||
</div> | ||
|
||
|
||
|
||
</div> | ||
</section> | ||
|
||
<section class="section" id="BibTeX"> | ||
<div class="container is-max-desktop content"> | ||
<h2 class="title">BibTeX</h2> | ||
<pre><code>@misc{kim2024meshupmultitargetmeshdeformation, | ||
title={MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation}, | ||
author={Hyunwoo Kim and Itai Lang and Noam Aigerman and Thibault Groueix and Vladimir G. Kim and Rana Hanocka}, | ||
year={2024}, | ||
eprint={2408.14899}, | ||
archivePrefix={arXiv}, | ||
primaryClass={cs.CV}, | ||
url={https://arxiv.org/abs/2408.14899}, | ||
}</code></pre> | ||
</div> | ||
</section> | ||
|
||
|
||
<footer class="footer"> | ||
<div class="container"> | ||
<div class="content has-text-centered"> | ||
<a class="icon-link" href="https://github.com/threedle/meshup" class="external-link" disabled> | ||
<i class="fab fa-github"></i> | ||
</a> | ||
</div> | ||
<div class="columns is-centered"> | ||
<div class="column is-8"> | ||
<div class="content"> | ||
<p> | ||
This website is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-sa/4.0/">Creative | ||
Commons Attribution-ShareAlike 4.0 International License</a>. | ||
</p> | ||
<p> | ||
Parts of the code for this website are reused from the <a href="https://github.com/nerfies/nerfies.github.io">source code</a> contributed by the authors of <a href="https://nerfies.github.io/">Nerfies</a>. | ||
</p> | ||
</div> | ||
</div> | ||
</div> | ||
</div> | ||
</footer> | ||
|
||
</body> | ||
</html> |
Oops, something went wrong.