What about beard?

[luigi-bar]

Hi,
thanks for the nice work! What about beard removal? Can this work be adapted to that?

Thanks!
L.

[oneThousand1000]

Hi,

Our work can be adjusted to handle beards. What's more, given that the 'beard' semantic attribute is typically associated with males, you can leverage our work from sig21. It would only require minor modifications to our existing framework to implement beard removal.

Yiqian

[luigi-bar]

Thanks! I will check it out

[luigi-bar]

Sorry to bother you again.... do you have an idea of the minimum GPU memory needed for inference with these models? I would like to understand if they fit my system... thanks!

[oneThousand1000]

Sorry to bother you again.... do you have an idea of the minimum GPU memory needed for inference with these models? I would like to understand if they fit my system... thanks!

We tested hairmapper and coarse-to-fine on 2060Ti.

[luigi-bar]

Hi again

so... I have tried to modify it to remove ALSO beard (so the goal is to remove hair and beard BOTH) with not so good results....

Here is what it did:

1. create a model to detect beard ( i have used https://www.kaggle.com/code/dima806/beard-face-image-detection-vit as a starting point) --> testing it on some test images it seems fine
2. modify check_hair function in classifier/classify.py as follows:

def check_hair(img,model):
    """  # Hair score: 1 for hair and 0 for bald """

    #    # original code:
    #    output= model.process(img)
    #    return output[0][1]>0.09


    global  beard_pipe

    output_hair= model.process(img)

    if output_hair[0][1]>0.09:
        return True

    if beard_pipe is None:
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        print("loading beard transformer model on ",device)
        beard_pipe = pipeline('image-classification', model='beard_face_image_detection', device=device)

    imgPIL= Image.fromarray(np.uint8(img)).convert('RGB')
    output_beard=beard_pipe(imgPIL)
    # [
    #   {'label': 'Beard', 'score': 0.9772831797599792},
    #   {'label': 'No Beard', 'score': 0.022716792300343513}
    # ]
    output_beard=max(output_beard, key=lambda x: x['score'])['label']
    
    if output_beard=='Beard':
        return True
    
    return False

4. generate 500 images D0 , generate 500 images Dnoise - i have modified step1 so that 10% of images are "hair=0"
5. follow all the steps in the README file, with 50k as final training

Do you see anything obviously wrong? Should I update other sections of the repo?

Thanks for any help you may be able to share, much appreciated!

[oneThousand1000]

Hi again

so... I have tried to modify it to remove ALSO beard (so the goal is to remove hair and beard BOTH) with not so good results....

Here is what it did:

1. create a model to detect beard ( i have used https://www.kaggle.com/code/dima806/beard-face-image-detection-vit as a starting point) --> testing it on some test images it seems fine
2. modify check_hair function in classifier/classify.py as follows:

def check_hair(img,model):
    """  # Hair score: 1 for hair and 0 for bald """

    #    # original code:
    #    output= model.process(img)
    #    return output[0][1]>0.09


    global  beard_pipe

    output_hair= model.process(img)

    if output_hair[0][1]>0.09:
        return True

    if beard_pipe is None:
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        print("loading beard transformer model on ",device)
        beard_pipe = pipeline('image-classification', model='beard_face_image_detection', device=device)

    imgPIL= Image.fromarray(np.uint8(img)).convert('RGB')
    output_beard=beard_pipe(imgPIL)
    # [
    #   {'label': 'Beard', 'score': 0.9772831797599792},
    #   {'label': 'No Beard', 'score': 0.022716792300343513}
    # ]
    output_beard=max(output_beard, key=lambda x: x['score'])['label']
    
    if output_beard=='Beard':
        return True
    
    return False

4. generate 500 images D0 , generate 500 images Dnoise - i have modified step1 so that 10% of images are "hair=0"
5. follow all the steps in the README file, with 50k as final training

Do you see anything obviously wrong? Should I update other sections of the repo?

Thanks for any help you may be able to share, much appreciated!

Could you show me your results?

[luigi-bar]

Beards are not even touched , as simile as that 😥 Il Dom 17 Mar 2024, 16:02 Yiqian Wu ***@***.***> ha scritto:

…

Hi again so... I have tried to modify it to remove ALSO beard (so the goal is to remove hair and beard BOTH) with not so good results.... Here is what it did: 1. create a model to detect beard ( i have used https://www.kaggle.com/code/dima806/beard-face-image-detection-vit as a starting point) --> testing it on some test images it seems fine 2. modify check_hair function in classifier/classify.py as follows: def check_hair(img,model): """ # Hair score: 1 for hair and 0 for bald """ # # original code: # output= model.process(img) # return output[0][1]>0.09 global beard_pipe output_hair= model.process(img) if output_hair[0][1]>0.09: return True if beard_pipe is None: device = torch.device("cuda" if torch.cuda.is_available() else "cpu") print("loading beard transformer model on ",device) beard_pipe = pipeline('image-classification', model='beard_face_image_detection', device=device) imgPIL= Image.fromarray(np.uint8(img)).convert('RGB') output_beard=beard_pipe(imgPIL) # [ # {'label': 'Beard', 'score': 0.9772831797599792}, # {'label': 'No Beard', 'score': 0.022716792300343513} # ] output_beard=max(output_beard, key=lambda x: x['score'])['label'] if output_beard=='Beard': return True return False 1. generate 500 images D0 , generate 500 images Dnoise - i have modified step1 so that 10% of images are "hair=0" 2. follow all the steps in the README file, with 50k as final training Do you see anything obviously wrong? Should I update other sections of the repo? Thanks for any help you may be able to share, much appreciated! Could you show me your results? — Reply to this email directly, view it on GitHub <#20 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADEDI3H2ILYUZLCNN7HYB6DYYWV7RAVCNFSM6AAAAABEGTZDRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBSGUYDAOBWGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[oneThousand1000]

Could you please show me some beard removal results using coarse_boundary (the one that trained using the D0 and Dnoise you generated)? It is important for the subsequent process.

[luigi-bar]

Hi , and thanks for your guidance.

I am a bit of a noob so let me ask you which step are you referring to? for the inference I have adapted this notebook example https://github.com/oneThousand1000/HairMapper/tree/master/notebooks , using the "best" model after training (

ckpt = torch.load('./training_runs/male_mapper/checkpoints/best_model.pt')
) . I have totally skipped the female part, so my complete workflow looks like this:

export DATASETSIZE=500
python3 -u step1_generate_data.py --dataset_name D0 --num $DATASETSIZE --truncation_psi=0.85
python3 -u step1_generate_data.py --dataset_name Dnoise --num $DATASETSIZE --add_noise --truncation_psi=0.85

python3 step2_train_man_hair_coarse_boundary.py  --output_dir sfHairBoundaryDir  --dataset_path ./training_runs/dataset/D0

python3 -u step3_train_bald_male_data.py --dataset_name D0  --hair_boundary_dir sfHairBoundaryDir --num 250000
python3 -u step3_train_bald_male_data.py --dataset_name Dnoise  --hair_boundary_dir sfHairBoundaryDir --num 250000

python3 -u step4_male_mapper_data_preparation.py  --dataset_name D0  --noise_dataset_name Dnoise --mapper_name male_mapper

python3 -u train_mapper.py --mapper_name male_mapper --max_steps 50000

inference:

git clone https://github.com/Puzer/stylegan-encoder.git
# ... download models
# ... place raw images in test_datta/raw
cd stylegan-encoder
python3 align_images.py  ../test_data/raw/ ../test_data/origin
cd ..
# run_inference.py is just the export of the sample notebook with minor changes (paths, models were alread downloaded, etc)
python3 -u run_inference.py 

[oneThousand1000]

python3 step2_train_man_hair_coarse_boundary.py  --output_dir sfHairBoundaryDir  --dataset_path ./training_runs/dataset/D0

The above command will output a coarse_boundary.

python3 -u step3_train_bald_male_data.py --dataset_name D0  --hair_boundary_dir sfHairBoundaryDir --num 250000
python3 -u step3_train_bald_male_data.py --dataset_name Dnoise  --hair_boundary_dir sfHairBoundaryDir --num 250000

The above commands will output the beard edting results using coarse_boundary. Could you show me some results of the two above commands?

[luigi-bar]

This is comparison between

• training_runs/dataset/D0/000004.jpg
• training_runs/male_training/D0/res_img/000004.jpg
  for a couple of cases: you may see that beard is hardly touched

[oneThousand1000]

It seems that the training data of coarse_boundary has some problems...
It looks like that the boundary you trained removes hair instead of beard.

The beard scores there should be 1 for having a beard and 0 for without a beard.

HairMapper/step1_generate_data.py

Line 152 in 40e85da

f'============= {np.sum(hair_scores_array)} images with hair, {args.num - np.sum(hair_scores_array)} images without hair =============')

Could you please check if the scores are aligned with the generated images?

[luigi-bar]

Hi,
I have checked and everything seems in order....
I have seen that you use a "Classifier" class that does a number of things. Should I modify additional methods (besides the check_hair() one ) if i use my custom model?

Apart from this, I don't have other idea 😭

[oneThousand1000]

Ok, I found the problem in your modified check_hair() function.

Since your check_hair() function returns true once it detects hair:

    output_hair= model.process(img)

    if output_hair[0][1]>0.09:
        return True

your score actually means :
1: Have hair or have beard
0: no hair and no beard

Note that it is not a binary classification, so your boundary training result is not good.

I recommend not using a single boundary to remove hair and beard at the same time....

Instead, you can train a "beardMapper" for only removing beard.
Next, use my pre-trained hairmapper to remove hair first, and then use your own beardMapper to remove beard.

The check_beard function could be :

def check_hair(img,model):
    """  # Hair score: 1 for hair and 0 for bald """

    #    # original code:
    #    output= model.process(img)
    #    return output[0][1]>0.09


    global  beard_pipe

    if beard_pipe is None:
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        print("loading beard transformer model on ",device)
        beard_pipe = pipeline('image-classification', model='beard_face_image_detection', device=device)

    imgPIL= Image.fromarray(np.uint8(img)).convert('RGB')
    output_beard=beard_pipe(imgPIL)
    # [
    #   {'label': 'Beard', 'score': 0.9772831797599792},
    #   {'label': 'No Beard', 'score': 0.022716792300343513}
    # ]
    output_beard=max(output_beard, key=lambda x: x['score'])['label']
    
    if output_beard=='Beard':
        return True
    
    return False

[luigi-bar]

Hi,
thanks for the hint - I have tried with no luck - I still get no edits between the images generated in step1 and the output of step3 .

Navigating the code, I have seen a number of actions in classifier/src/feature_extractor/hair_mask_extractor.py that are related to the "hair". Should I change anything here for "beard" ?

Thanks!

[oneThousand1000]

Yes, you're right.
You should have a beard_mask_extractor so that it can extract beard mask, and replace the hair_mask_extractor with it in step3