Next

• [X] Build ops inside transformer.ml
• [X] Add workload specs
• [X] Add optimizer state
• [ ] Add forward pass for ops - ops_intf.ml
• [ ] Model memory first
• [ ] Save all activations first
• [ ] Add dropout
• [ ] Diff types of stats
  • [ ] Weight stats - keeps track of mem per op + optimizer states
  • [ ] fwd pass stats - keeps track of mem per op per iteration
  • [ ] bkwd pass stats