Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add new features and fixes to Sparse Update #47

Open
wants to merge 95 commits into
base: main
Choose a base branch
from
Open

Add new features and fixes to Sparse Update #47

wants to merge 95 commits into from

Conversation

dnadalini
Copy link
Collaborator

@dnadalini dnadalini commented Jul 25, 2024 •
edited
Loading

This PR adds the following:

  • Leaky Relu Activation Function and related integration in TrainLib_Deployer
  • Integration of Sigmoid Activation Function in TrainLib_Deployer
  • Pseudo-random number generator for Dropout layer
  • L1Loss implementation
  • berHu loss implementation (missing test, to do)

Missing:

  • Integration of pseudo-random number generation in TrainLib_Deployer
  • Integration of L1Loss in TrainLib_Deployer

dnadalini and others added 30 commits April 3, 2024 11:21
@dnadalini
Fix minors for sparse update and add train loss monitoring during epochs
5e914fe
@dnadalini
Fix code generation (minors)
15ce4e4
@dnadalini
Enhance GM for Conv2D test (FP32)
81cdd47
@dnadalini
Merge branch 'main' of github.com:pulp-platform/pulp-trainlib into pr…
dcd195e 
…/SparseUpdate
@dnadalini
Merge branch 'main' of github.com:pulp-platform/pulp-trainlib into pr…
ec037d1 
…/SparseUpdate
@dnadalini
Fix code generation for sparse update in TrainLib_Deployer without si…
3aaae79 
…ngle/double buffering
@dnadalini
Introduce sparse update in single buffer more (code generator is bugg…
438716b 
…edgit status)
@dnadalini
Merge branch 'main' of github.com:pulp-platform/pulp-trainlib into pr…
bd8d8cb 
…/SparseUpdate
@dnadalini
Merge branch 'main' of github.com:pulp-platform/pulp-trainlib into pr…
d0a981e 
…/SparseUpdate
@dnadalini
Fix merge issues in TrainLib_Deployer
d8ebd50
@dnadalini
Merge branch 'main' of github.com:pulp-platform/pulp-trainlib into pr…
1c29032 
…/SparseUpdate
@dnadalini
Merge branch 'main' of github.com:pulp-platform/pulp-trainlib into pr…
29d07a3 
…/SparseUpdate
@dnadalini
Partially fix single buffering with sparse update
b11257a
@dnadalini
Fix partial update with single buffering mode
315fb17
@dnadalini
Fix single buffer mode (no sparse update)
bde9f48
@dnadalini
Optimize memory occupation and fix single buffer mode with sparse update
35c8dc0
@dnadalini
Fix sparse update (still existing bugs) with no buffer and single buf…
a84f8ba 
…fer mode
@dnadalini
Add error in case of double buffering selection (currently unavailable)
793d864
@dnadalini
Fix memory bug with single buffering
4dd71f8
@diaconuccalin
Merge pull request #3 from pulp-platform/main
b566118 
Bring changes from main repo
@dnadalini
Fix functional bug with memory allocation for ReLU backpropagation wi…
2fe4089 
…th sparse update
@dnadalini
Add testing setup for sparse update
417ae28
@dnadalini
Add optimized naive conv2d algorithms for 5x5 kernels with stride 2 a…
c95b43a 
…nd padding 1
@dnadalini
Fix InstanceNorm primitives in both forward and backward (FP32, FP16)
e5d34dc
@dnadalini
Merge with main
d2e4bd1
@diaconuccalin
Merge pull request #4 from pulp-platform/main
216520e 
Merge changes from main repo
@dnadalini
Merge branch 'main' of github.com:pulp-platform/pulp-trainlib into pr…
c08ef3a 
…/SparseUpdate
@diaconuccalin
Merge pull request #6 from pulp-platform/main
f5c6621 
Rebase main branch to updated main of original repo
@dnadalini
Refactor code for the data assignment between blobs (no and single bu…
8f55d8c 
…ffer mode)
@diaconuccalin
Merge pull request #8 from pulp-platform/main
378ce6b 
Update main branch to original repo
alberto and others added 30 commits August 1, 2024 11:59
Added dropout kernels for FP32 and FP16
ae3cf78
@diaconuccalin
Changed the mhsa fp32 implementation to replace the previous big line…
aa4ada7 
…ar layer, that generates qkv from the input, with 3 smaller ones, as the original implementation is designed (making it compatible with publicly available pretrained models). Ran the Black code formatter on the 2 modified py files
@diaconuccalin
Changed the mhsa fp16 implementation to replace the previous big line…
c75cce6 
…ar layer, that generates qkv from the input, with 3 smaller ones, as the original implementation is designed (making it compatible with publicly available pretrained models). Ran the Black code formatter on the 2 modified py files. Other small fixes
Modified makefile
959cf24
Merge branch 'mhsa_fix' of github.com:diaconuccalin/pulp-trainlib int…
6802ead 
…o diaconuccalin-mhsa_fix
Merge branch 'diaconuccalin-mhsa_fix' into pulp-trainlib-dev
4fd1fd3
@diaconuccalin
Included biases for input projection layers for the forward fp32 step…
778a830 
…. Changed some variable names of mhsa fp32 to match official ViT implementation. Other fixes
@diaconuccalin
Fully included biases for input projection layers for fp32 and fp64. …
6a1e311 
…Other fixes
@dnadalini
Add first version of fp32 transposed convolution 2D (incomplete, to b…
0db8db7 
…e debugged) and files to support the fp16 version
Merge branch 'mhsa_fix' of github.com:diaconuccalin/pulp-trainlib int…
78136ef 
…o diaconuccalin-mhsa_fix
Merge branch 'diaconuccalin-mhsa_fix' into pulp-trainlib-dev
95a4f92
@Dequino
Merge branch 'pr/SparseUpdate' into pulp-trainlib-dev
983f336
@dnadalini
Merge pull request #48 from Dequino/pulp-trainlib-dev
4249fba 
Pulp trainlib dev
@dnadalini
Add forward step of transposed convolution 2d (FP32, no optimization)
6349d73
@dnadalini
Merge branch 'pr/SparseUpdate' of github.com:pulp-platform/pulp-train…
da982d3 
…lib into pr/SparseUpdate
@dnadalini
Fix pulp_train.h
de5d70f
@dnadalini
Fix forward step of transposed conv2d (FP32, no opt)
4b44e2b
@dnadalini
Add (partially tested) weight grad of transposed conv2d (FP32, no opt)
9ff2b60
@dnadalini
Add input grad (no padding) for FP32 transposed convolution 2d (no opt)
d32f938
@dnadalini
Add first working (but unoptimized) transposed convolution 2D kernels…
482de08 
… (FP32, all, with biases, stride and padding)
@dnadalini
Optimize the weight grad step of FP32 naive transposed conv2d
0cac308
@dnadalini
Parallelize and optimize the forward and backward grad steps of the t…
5ddab65 
…ransposed conv2d (FP32)
@dnadalini
Add parallelization and code optimizations for all 3 steps of transpo…
f36fff1 
…sed convolution 2d (FP32)
@dnadalini
Add fp16 transposed convolution (not optimized more than fp32) and re…
1e49fd0 
…lated test
@dnadalini
Add bilinear and nearest neighbour interpolations (fp32, parallel)
95ea4a0
@dnadalini
Add fp16 version of interpolations and fix test to support fp16
cf1b8e7
@dnadalini
Fix minors in test_losses
662c9bd
@dnadalini
Fix minors in test_losses
7f3fbb8
@dnadalini
Fix test for activations and sizes of convolutions for testing
1cc482a
@dnadalini
Fix test for the activation functions
69d17dd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dnadalini @diaconuccalin @Dequino