Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

switch to tfds array_record, improve visualization, dataset descriptions #190

Merged
merged 22 commits into from
Aug 28, 2023

Conversation

jpata
Copy link
Owner

@jpata jpata commented Aug 11, 2023

  • update plots for the paper
  • switch to ARRAY_RECORD for tfds storage to allow the same datasets to be easily loaded from tf and pytorch
    • this implies new versions of the tfds datasets, propagated to the training configs
  • disable tf2onnx as it seems incompatible with the latest tensorflow (due to the flatbuffers requirement)
  • reorganize and clean up some older datasets
  • switch tests to python 3.10 and ubuntu 22.04

The latest 1.5.0 versions of the tensorflow datasets are available from:

gfal-copy -r root://xrootd.hep.kbfi.ee:1094/local/joosep/mlpf/tensorflow_datasets/clusters
gfal-copy -r root://xrootd.hep.kbfi.ee:1094/local/joosep/mlpf/tensorflow_datasets/hits

and also here at lxplus for the cluster-based dataset:

$ du -csh /eos/user/j/jpata/mlpf/tensorflow_datasets/clusters/*
11G	/eos/user/j/jpata/mlpf/tensorflow_datasets/clusters/clic_edm_qq_pf
11G	/eos/user/j/jpata/mlpf/tensorflow_datasets/clusters/clic_edm_ttbar_pf
13G	/eos/user/j/jpata/mlpf/tensorflow_datasets/clusters/clic_edm_ttbar_pu10_pf
8.6G	/eos/user/j/jpata/mlpf/tensorflow_datasets/clusters/clic_edm_ww_fullhad_pf
4.1G	/eos/user/j/jpata/mlpf/tensorflow_datasets/clusters/clic_edm_zh_tautau_pf

Validation
on the cluster-based dataset, training parameters/clic.yaml, I'm getting
jet_res
met_res

Screenshot from 2023-08-28 13-48-33
Screenshot from 2023-08-28 13-48-51
Screenshot from 2023-08-28 13-49-02

@jpata jpata changed the title improve visualization improve visualization, dataset descriptions Aug 23, 2023
@jpata jpata changed the title improve visualization, dataset descriptions switch to tfds array_record, improve visualization, dataset descriptions Aug 24, 2023
@jpata jpata merged commit ac4c6f3 into main Aug 28, 2023
11 checks passed
@jpata jpata deleted the clicvis branch August 28, 2023 15:33
jpata added a commit that referenced this pull request Sep 15, 2023
…ons (#190)

* improve visualization

* habana vis

* update plots

* fix horovod training

* update click vis

* update visualization

* swap back y and z

---------

Co-authored-by: Javier Duarte <[email protected]>
jpata added a commit that referenced this pull request Sep 15, 2023
…ons (#190)

* improve visualization

* habana vis

* update plots

* fix horovod training

* update click vis

* update visualization

* swap back y and z

---------

Co-authored-by: Javier Duarte <[email protected]>
Former-commit-id: 1176df0
jpata added a commit that referenced this pull request Sep 15, 2023
…ons (#190)

* improve visualization

* habana vis

* update plots

* fix horovod training

* update click vis

* update visualization

* swap back y and z

---------

Co-authored-by: Javier Duarte <[email protected]>
Former-commit-id: 1176df0
jpata added a commit that referenced this pull request Sep 25, 2023
…ons (#190)

* improve visualization

* habana vis

* update plots

* fix horovod training

* update click vis

* update visualization

* swap back y and z

---------

Co-authored-by: Javier Duarte <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants