Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Questions about ICON+Keypoint #231

Open
Jarvisss opened this issue Sep 13, 2023 · 2 comments
Open

Questions about ICON+Keypoint #231

Jarvisss opened this issue Sep 13, 2023 · 2 comments

Comments

@Jarvisss
Copy link

Jarvisss commented Sep 13, 2023

ICON/lib/net/HGPIFuNet.py

Lines 301 to 305 in 8919d96

if self.prior_type == "keypoint":
kpt_feat = self.sp_encoder.forward(
cxyz=xyz.permute(0, 2, 1).contiguous(),
kptxyz=self.smpl_feat_dict["smpl_joint"],
)

I find the implementation of keypoint+ICON projects query points to camera space, but smpl joints are not projected.
But in original keypointNerf paper, the spatial encoding is calculated by the projected keypoints.

ICON/lib/net/HGPIFuNet.py

Lines 333 to 340 in 8919d96

if self.prior_type == "keypoint":
if "vis" in self.smpl_feats:
point_local_feat = feat_select(self.index(im_feat, xy), smpl_feat[:, [-1], :])
point_feat_list = [point_local_feat, kpt_feat, smpl_feat[:, :-1, :]]
else:
point_local_feat = self.index(im_feat, xy)
point_feat_list = [point_local_feat, kpt_feat, smpl_feat[:, :, :]]

And SMPL features are still used in the keypoint version, is this what it supposed to be?

@Jarvisss
Copy link
Author

@YuliangXiu

@YuliangXiu
Copy link
Owner

smpl_joints should be projected into camera space, you could visualize the keypoints when running infer.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants