Skip to content

Commit

Permalink
initial commit
Browse files Browse the repository at this point in the history
  • Loading branch information
@hywkim-private
hywkim-private committed Sep 13, 2024
0 parents commit b25e63e
Show file tree
Hide file tree
Showing 25 changed files with 3,455 additions and 0 deletions.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
*.log
temp/
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2023 Threedle (University of Chicago)

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
18 changes: 18 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# MeshUp
This is the repository that contains source code for the [MeshUp website](https://threedle.github.io/meshup).

If you find MeshUp useful for your work please cite:
```
@misc{kim2024meshupmultitargetmeshdeformation,
title={MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation},
author={Hyunwoo Kim and Itai Lang and Noam Aigerman and Thibault Groueix and Vladimir G. Kim and Rana Hanocka},
year={2024},
eprint={2408.14899},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2408.14899},
}
```

# Website License
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.
332 changes: 332 additions & 0 deletions index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,332 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="description"
content="Multi-Target Mesh Deformation via Blended Score Distillation">
<meta name="keywords" content="MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>3D Paintbrush</title>

<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
rel="stylesheet">

<link rel="stylesheet" href="./static/css/bulma.min.css">
<link rel="stylesheet" href="./static/css/bulma-carousel.min.css">
<link rel="stylesheet" href="./static/css/bulma-slider.min.css">
<link rel="stylesheet" href="./static/css/fontawesome.all.min.css">
<link rel="stylesheet"
href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
<link rel="stylesheet" href="./static/css/index.css">
<!-- <link rel="icon" href="./static/images/favicon.ico"> -->
<link rel="icon" href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%220 0 100 100%22><text y=%22.9em%22 font-size=%2290%22>🎨</text></svg>">

<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script defer src="./static/js/fontawesome.all.min.js"></script>
<script src="./static/js/bulma-carousel.min.js"></script>
<script src="./static/js/bulma-slider.min.js"></script>
<script src="./static/js/index.js"></script>
</head>
<body>

<nav class="navbar" role="navigation" aria-label="main navigation">
<div class="navbar-brand">
<a role="button" class="navbar-burger" aria-label="menu" aria-expanded="false">
<span aria-hidden="true"></span>
<span aria-hidden="true"></span>
<span aria-hidden="true"></span>
</a>
</div>
<div class="navbar-menu">
<div class="navbar-start" style="flex-grow: 1; justify-content: center;">
<a class="navbar-item" href="https://3dl.cs.uchicago.edu/">
<span class="threedle-icon"></span>
</a>
</div>

</div>
</nav>


<section class="hero">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<h1 class="title is-1 publication-title"> MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation</h1>
<div class="is-size-5 publication-authors">
<span class="author-block">
<a>Hyunwoo Kim</a><sup>1</sup>,
</span>
<span class="author-block">
<a>Itai Lang</a><sup>1</sup>,
</span>
<span class="author-block">
<a>Thibault Groueix</a><sup>2</sup>,
</span>
<span class="author-block">
<a>Noam Aigerman</a><sup>3</sup>,
</span>
<span class="author-block">
<a>Vladimir G. Kim</a><sup>2</sup>,
</span>
<span class="author-block">
<a>Rana Hanocka</a><sup>1</sup>
</span>
</div>

<div class="is-size-5 publication-authors">
<span class="author-block"><sup>1</sup>University of Chicago</span>
<span class="author-block"><sup>2</sup>Adobe research</span>
<span class="author-block"><sup>3</sup>University of Montreal</span>
</div>

<div class="column has-text-centered">
<div class="publication-links">
<!-- PDF Link. -->

<!-- arXiv Link. -->
<span class="link-block">
<a href="https://arxiv.org/abs/2408.14899"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="ai ai-arxiv"></i>
</span>
<span>arXiv</span>
</a>
</span>

<!-- Code Link. -->
<span class="link-block">
<a href="#"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-github"></i>
</span>
<span>Comming Soon</span>
</a>
</span>
</div>

</div>
</div>
</div>
</div>
</div>
</section>

<section class="hero teaser">
<div class="container is-max-desktop">
<div class="hero-body">
<!-- <image id="teaser">
<img src="./static/images/figures/teaser.png"
type="image/png">
</image> -->
<div style="display: flex; justify-content: space-between;">
<video poster="" id="rabbit" autoplay muted loop playsinline height="60%" width="60%">
<source src="./static/videos/rabbitanimation.gif" type="video/mp4">
</video>
<video poster="" id="couch" autoplay muted loop playsinline height="60%" width="60%">
<source src="./static/videos/couchanimation.gif" type="video/mp4">
</video>
<video poster="" id="moose" autoplay muted loop playsinline height="60%" width="60%">
<source src="./static/videos/mooseanimation.gif" type="video/mp4">
</video>
</div>
<h2 class="subtitle has-text-centered">
MeshUp is capable of deforming a source mesh into various concepts and into their weighted blends. The target objectives can
be text prompts, images, or even meshes. Users can also input a set of control vertices to explicitly define where on the mesh each concept
concept should be expressed.
</h2>
</div>
</div>
</section>


<section class="section">
<div class="container is-max-desktop">
<!-- Abstract. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Abstract</h2>
<div class="content has-text-justified">
<p>
We propose MeshUp, a technique that deforms a 3D mesh towards multiple target
concepts, and intuitively controls the region where each concept is expressed.
Conveniently, the concepts can be defined as either text queries, e.g., "a dog"
and "a turtle," or inspirational images, and the local regions can be selected
as any number of vertices on the mesh. We can effectively control the influence
of the concepts and mix them together using a novel score distillation
approach, referred to as the Blended Score Distillation (BSD). BSD operates on
each attention layer of the denoising U-Net of a diffusion model as it extracts
and injects the per-objective activations into a unified denoising pipeline
from which the deformation gradients are calculated. To localize the expression
of these activations, we create a probabilistic Region of Interest (ROI) map on
the surface of the mesh, and turn it into 3D-consistent masks that we use to
control the expression of these activations. We demonstrate the effectiveness
of BSD empirically and show that it can deform various meshes towards multiple
objectives.
</p>
</div>
</div>
</div>
<!--/ Abstract. -->


<!-- Composite. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<hr class="divider" />
<h2 class="title is-3">Precise composition of multiple local textures</h2>
<div class="overview-image">
<img src="./static/images/figures/gallery.png" type="image/png">
</div>
<div class="content has-text-justified">
<p>
MeshUp takes as input a 3D mesh and several target objectives, and deforms the source mesh by optimizing the
per-triangle Jacobians of the mesh. MeshUp produces a deformation that blends multiple concepts together, respective of the user-defined weights for each concept.
Notice how the deformation smoothly transitions the source mesh into a mixture of different concepts, and weights dictate the magnitude of their expression.
MeshUp supports as many targets as desired.
</p>
</div>
</div>
</div>
<!--/ Composite. -->
<!-- Gallery. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<hr class="divider" />
<h2 class="title is-3">Blended Score Distillation</h2>
<div class="overview-image">
<img src="./static/images/figures/system_local.png" type="image/png">
</div>
<div class="content has-text-justified">
<p>
MeshUp can also localize the deformation to specific regions on the mesh. The user can select a set of control vertices to define where on the mesh each concept should be expressed.
The deformation (of one or multiple concepts) is then constrained to these regions, producing a more controlled and localized result.
</p>
</div>
</div>
</div>
<!--/ Gallery. -->
<!-- Gallery. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<hr class="divider" />
<h2 class="title is-3">Localization</h2>
<div class="overview-image">
<img src="./static/images/figures/system_local.png" type="image/png">
</div>
<div class="content has-text-justified">
<p>
MeshUp can also localize the deformation to specific regions on the mesh. The user can select a set of control vertices to define where on the mesh each concept should be expressed.
The deformation (of one or multiple concepts) is then constrained to these regions, producing a more controlled and localized result.
</p>
</div>
</div>
</div>
<!--/ Gallery. -->

<!-- Overview. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<hr class="divider" />
<h2 class="title is-3">Network Overview</h2>
<div class="overview-image">
<img src="./static/images/figures/system.png" type="image/png">
</div>

<div class="content has-text-justified">
<p>
At each iteration, it creates parallel UNET branches, assigned each to the input target objectives (we call these the "Target Branches").
Then, it passes the same noised renderings and the corresponding text input through
the UNet of a pretrained text-to-image model. On a different branch (the "Blending Branch"), it interpolates the activations
extracted from the Target Branches, and interpolates them, respective to the assigned weights. Replacing the activations in the blending branch with the interpolated activations,
MeshUp backpropagates the gradients from the blending branch via Score Distillation Sampling (SDS), updating the Jacobians of the mesh accordingly.
</p>
</div>
</div>
</div>
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<hr class="divider" />
<h2 class="title is-3">Localiztion Overview</h2>
<div class="overview-image">
<img src="./static/images/figures/system_local.png" type="image/png">
</div>
<div class="content has-text-justified">
<p>
To locaalize the deformation, MeshUp creates a probabilistic Region of Interest (ROI) map on the surface of the mesh using the self-attention maps of the selected control vertices.
It then uses this 3D ROI map to restrain the jacobians of the mesh, ensuring that the deformation is localized to the selected regions.
For localization objectives with multiple targets, we rasterize the 3D ROI map of each target from the same viewpoint as the renderings, and use them as binary masks to control the expression of the activations.
</p>
</div>
</div>
</div>
<!--/ Overview. -->



<!--/ Comparison. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<hr class="divider" />
<h2 class="title is-3">Comparison with Various Methods</h2>
<div class="overview-image">
<img src="./static/images/figures/comparison.png" type="image/png">
</div>
<div class="content has-text-justified">
<p>
Compared to the state-of-the-art text-to-3D generative models, MeshUp produces results that have both better triangulation and geometric details.
Our method is also first to support local deformation capabilities, which not only gurantees the preservation of unselected regions, but also greatly surpasses the quality of meshes generated solely using text descriptions.
</p>
</div>
</div>
</div>



</div>
</section>

<section class="section" id="BibTeX">
<div class="container is-max-desktop content">
<h2 class="title">BibTeX</h2>
<pre><code>@misc{kim2024meshupmultitargetmeshdeformation,
title={MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation},
author={Hyunwoo Kim and Itai Lang and Noam Aigerman and Thibault Groueix and Vladimir G. Kim and Rana Hanocka},
year={2024},
eprint={2408.14899},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2408.14899},
}</code></pre>
</div>
</section>


<footer class="footer">
<div class="container">
<div class="content has-text-centered">
<a class="icon-link" href="https://github.com/threedle/meshup" class="external-link" disabled>
<i class="fab fa-github"></i>
</a>
</div>
<div class="columns is-centered">
<div class="column is-8">
<div class="content">
<p>
This website is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-sa/4.0/">Creative
Commons Attribution-ShareAlike 4.0 International License</a>.
</p>
<p>
Parts of the code for this website are reused from the <a href="https://github.com/nerfies/nerfies.github.io">source code</a> contributed by the authors of <a href="https://nerfies.github.io/">Nerfies</a>.
</p>
</div>
</div>
</div>
</div>
</footer>

</body>
</html>
Loading

0 comments on commit b25e63e

Please sign in to comment.