Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate forward/backward pass #72

Open
Ebanflo42 opened this issue Apr 4, 2024 · 2 comments
Open

Separate forward/backward pass #72

Ebanflo42 opened this issue Apr 4, 2024 · 2 comments

Comments

@Ebanflo42
Copy link
Contributor

We need to carefully design the API such that the user has access to both an executable that returns the desired diff calls (for training) and an executable that returns everything except that (for testing).

This is part of a larger array of issues that will emerge from the need to embed contexts in other contexts (for example, separating the optimizer step, or designing recurrent architectures). In this case it might make sense to allow the user to design a forward pass context which doesn't take labels or output gradients, then allow them to clone that context and recover all desired node identifiers in order to create another context that takes both labels and inputs and outputs both predictions and loss and gradients. Then both executables can be used separately.

@Ebanflo42
Copy link
Contributor Author

Also would make sense to add gradient clipping as a part of the solution to this issue.

@Ebanflo42
Copy link
Contributor Author

Scratch that, gradient clipping should be an extra feature in the core autodiff engine, i don't know why I was thinking these things are related.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant