Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot reproduce the results of Table2 in the paper #24

Open
vifi2021 opened this issue Jul 16, 2024 · 0 comments
Open

Cannot reproduce the results of Table2 in the paper #24

vifi2021 opened this issue Jul 16, 2024 · 0 comments

Comments

@vifi2021
Copy link

Hello,
I am having some trouble reproducing the results of Table2 of your paper (https://arxiv.org/pdf/2107.05908) on the HDFS dataset.

For the unsupervised methods (LSTM, Transformer, and Autoencoder), I am following the scripts in the benchmark/ folder.
The script I tried is:
$ python transformer_demo.py --label_type next_log --feature_type semantics --use_tfidf --topk 10 --dataset HDFS --data_dir ../data/processed/HDFS/hdfs_0.0_tar/

During evaluation phase, it outputs 10 sets of f1-scores/precision/recall because k=10. But non of them matches the scores (0.9+) in the paper. The best f1-score I observed is around 0.8 for top-5.

Could you please clarify on this issue? When training using --label_type = next_log, which accuracy should we look at if we want to reproduce the numbers in Table2?

Thank you for your help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant