Gb/optm state #242
Gb/optm state #242
Conversation
… output at end of epoch to history
… could underestimate changes
not really, i think we'll need more training time to see if there's a trajectory. Some layers are definitely gaining momentum on gradients but not sure if they'll converge. |
grantbuster
force-pushed
the
gb/optm_state
branch
from
ef77b81
to
0bc3f6c
Compare
grantbuster
force-pushed
the
gb/optm_state
branch
from
0bc3f6c
to
5134c3e
Compare
sup3r/preprocessing/cachers/base.py
Outdated
| f'received error: "{e}"') | ||
| logger.warning(msg) | ||
| warn(msg) | ||
| ncfile.setncattr(attr_name, json.dumps(attr_value)) |
There was a problem hiding this comment.
@bnb32 fyi i was getting a weird error in production runs where the model meta data had objects that were not compatible with NetCDF. I added some extra error handling here.
There was a problem hiding this comment.
@grantbuster ok cool. I removed some of the default serializing I had before bc it was casting too many things to strings, guess I removed too many. you should use the safe_serialize function:
sup3r/sup3r/utilities/utilities.py
Line 69 in e43e4bb
…ta as attrs, needs to be serialized another way, added error handling here
grantbuster
force-pushed
the
gb/optm_state
branch
from
5134c3e
to
a74ed5c
Compare
No description provided.