{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":601924134,"defaultBranch":"main","name":"RWKV-LM-LoRA","ownerLogin":"Blealtan","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2023-02-15T05:41:54.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/6376623?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1685617720.6771462","currentOid":""},"activityList":{"items":[{"before":"b7f2b6c17c0c0a10d1b3bff2d624548641e75b24","after":"54dadc936e8f71af0f012bdfbb96cc584017cbcc","ref":"refs/heads/dev-infctx","pushedAt":"2023-07-05T06:19:33.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"[dev-infctx][batch 3] Trainer validation notebooks (#41)\n\n* Added trainer validation notebooks, for replication process, and updated .gitignore / readme\r\n\r\n* config tweaks","shortMessageHtmlLink":"[dev-infctx][batch 3] Trainer validation notebooks (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1787892408\" data-permission-text=\"Title is private\" data-url=\"https://github.com/Blealtan/RWKV-LM-LoRA/issues/41\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/Blealtan/RWKV-LM-LoRA/pull/41/hovercard\" href=\"https://github.com/Blealtan/RWKV-LM-LoRA/pull/41\">#41</a>)"}},{"before":"2ed1e1fc61ca49d9f86697dca479da22fc2b7e61","after":"b7f2b6c17c0c0a10d1b3bff2d624548641e75b24","ref":"refs/heads/dev-infctx","pushedAt":"2023-07-05T06:14:38.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Major overhaul of the datamodule, into its own class. Also extended the lightning trainer class for fabric instance (#39)","shortMessageHtmlLink":"Major overhaul of the datamodule, into its own class. Also extended t…"}},{"before":"679264117be25b071616e8544bc89d74f52f8da3","after":"2ed1e1fc61ca49d9f86697dca479da22fc2b7e61","ref":"refs/heads/dev-infctx","pushedAt":"2023-07-05T06:12:01.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"[dev-infctx][batch 1] Added lr final support (#33)\n\n* Added support for lr_final\r\n\r\n* Introduction of bptt_learning\r\n\r\n* Revert \"Introduction of bptt_learning\"\r\n\r\nThis reverts commit e2cd3acc032e9e361262ab897b110bd8cb85109d.","shortMessageHtmlLink":"[dev-infctx][batch 1] Added lr final support (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1773195284\" data-permission-text=\"Title is private\" data-url=\"https://github.com/Blealtan/RWKV-LM-LoRA/issues/33\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/Blealtan/RWKV-LM-LoRA/pull/33/hovercard\" href=\"https://github.com/Blealtan/RWKV-LM-LoRA/pull/33\">#33</a>)"}},{"before":"1d609898e846c0d55d2dad45e9ce6bf5fd4f25d6","after":"679264117be25b071616e8544bc89d74f52f8da3","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-28T09:00:51.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Fix time checkpointing.\n\nThe previous implementation misses all chunks backward pass but the\nlast. Fixing this makes deepspeed stage 2 not working though.\n\nAlso making the states really checkpointed through expanding classes\ninto raw tensors.","shortMessageHtmlLink":"Fix time checkpointing."}},{"before":"b57690e039539238a2f3c9853055498ffe482f6e","after":"1d609898e846c0d55d2dad45e9ce6bf5fd4f25d6","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-24T05:08:31.196Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"[dev-infctx] better hf data support (#32)\n\n* Added support for weight_decay, and torch.set_float32_matmul_precision\r\n\r\n* Support for customizable Instruction/Input/Output formats, HF datasets, and token based filtering\r\n\r\n* Updated the config example\r\n\r\n* dataset preloading via preload_dataset.py\r\n\r\n* updating readme\r\n\r\n* Added torch.set_float32_matmul_precision (optimize cuda training)\r\n\r\n* Update config-example.yaml\r\n\r\n---------\r\n\r\nCo-authored-by: Blealtan <blealtan@outlook.com>","shortMessageHtmlLink":"[dev-infctx] better hf data support (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1770020241\" data-permission-text=\"Title is private\" data-url=\"https://github.com/Blealtan/RWKV-LM-LoRA/issues/32\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/Blealtan/RWKV-LM-LoRA/pull/32/hovercard\" href=\"https://github.com/Blealtan/RWKV-LM-LoRA/pull/32\">#32</a>)"}},{"before":"c0950274953114f5dfdd2470a18f54f357af9ebd","after":"b57690e039539238a2f3c9853055498ffe482f6e","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-21T09:57:00.329Z","pushType":"pr_merge","commitsCount":4,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Updated with env setup","shortMessageHtmlLink":"Updated with env setup"}},{"before":"486185cb28891f8ee647424d2ff1b3c271433dcc","after":"c0950274953114f5dfdd2470a18f54f357af9ebd","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-19T14:51:49.374Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Added a check for a valid wandb run","shortMessageHtmlLink":"Added a check for a valid wandb run"}},{"before":"d24f34a5849884a7c68207c0caad1ba3212199cd","after":"486185cb28891f8ee647424d2ff1b3c271433dcc","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-19T14:51:37.368Z","pushType":"pr_merge","commitsCount":8,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Added tag in config-example logging","shortMessageHtmlLink":"Added tag in config-example logging"}},{"before":"c94ec362ba7ca0baf27d0eab4d6cf33f7a01c930","after":"4987137c31dd49cbbf2b2db3977930bc6ce5b84e","ref":"refs/heads/main","pushedAt":"2023-06-19T12:13:11.994Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Hyperparameters sanity check.","shortMessageHtmlLink":"Hyperparameters sanity check."}},{"before":"ff7acf87a62b37b5cf40dedae743cd19da0c5553","after":"d24f34a5849884a7c68207c0caad1ba3212199cd","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-18T07:35:35.704Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Added the export_checkpoint script","shortMessageHtmlLink":"Added the export_checkpoint script"}},{"before":"d8e56015841efd40a3829cf7b0a4c0c9e877c0b9","after":"ff7acf87a62b37b5cf40dedae743cd19da0c5553","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-13T14:08:26.288Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"wandb logging of substep (gave in and ruin the steps)","shortMessageHtmlLink":"wandb logging of substep (gave in and ruin the steps)"}},{"before":"c5cb3efcc003ee8d8352363a6e1361f53febeb68","after":"c94ec362ba7ca0baf27d0eab4d6cf33f7a01c930","ref":"refs/heads/main","pushedAt":"2023-06-13T10:55:17.099Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Fix for World finetuning.","shortMessageHtmlLink":"Fix for World finetuning."}},{"before":"713f04608127c875084beb33039385a30965f592","after":"d8e56015841efd40a3829cf7b0a4c0c9e877c0b9","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-11T16:17:38.687Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Small fixes, better logging","shortMessageHtmlLink":"Small fixes, better logging"}},{"before":"b6fcc96372a86d1c3db60d113e7df7cf7bfaefa5","after":"713f04608127c875084beb33039385a30965f592","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-05T16:34:50.716Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Fixed and better ctxlen cutoff.","shortMessageHtmlLink":"Fixed and better ctxlen cutoff."}},{"before":"f60089a17c685e490b99ad209b760c8534fa7a6f","after":"b6fcc96372a86d1c3db60d113e7df7cf7bfaefa5","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-04T06:19:36.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Short doc.","shortMessageHtmlLink":"Short doc."}},{"before":"1036c8804c20b97be6782cc8b77e0f150f76838b","after":"f60089a17c685e490b99ad209b760c8534fa7a6f","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-03T12:36:44.693Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Fix NaN.","shortMessageHtmlLink":"Fix NaN."}},{"before":"c65ec37dd541f95c58be784b9c5fc0ef8a218936","after":"1036c8804c20b97be6782cc8b77e0f150f76838b","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-03T09:51:33.042Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Migrate to lightning cli (2.0) and massive cleanup","shortMessageHtmlLink":"Migrate to lightning cli (2.0) and massive cleanup"}},{"before":"a36a1c178837923eeedba95b3a8e158b8387805c","after":"c65ec37dd541f95c58be784b9c5fc0ef8a218936","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-02T15:49:05.487Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Add ctx_len_cutoff, remove Bo's dedicated options","shortMessageHtmlLink":"Add ctx_len_cutoff, remove Bo's dedicated options"}},{"before":"8be0f25665cb4c78c4ba0486b5299acd83c0f59c","after":"a36a1c178837923eeedba95b3a8e158b8387805c","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-02T14:01:02.676Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Long context training viable now.","shortMessageHtmlLink":"Long context training viable now."}},{"before":"c7e201ae8562482f58a6e455fab3748107554d8c","after":"8be0f25665cb4c78c4ba0486b5299acd83c0f59c","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-02T09:43:30.882Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Remove obsolete codes.","shortMessageHtmlLink":"Remove obsolete codes."}},{"before":"97b8de6d65afec7743fca36f21d61d4efe106945","after":"c7e201ae8562482f58a6e455fab3748107554d8c","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-01T13:43:41.560Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Initial implementation of time checkpointing","shortMessageHtmlLink":"Initial implementation of time checkpointing"}},{"before":null,"after":"97b8de6d65afec7743fca36f21d61d4efe106945","ref":"refs/heads/dev-infctx","pushedAt":"2023-06-01T11:08:40.677Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Better wandb and initial implementation of validation","shortMessageHtmlLink":"Better wandb and initial implementation of validation"}},{"before":"7ac25dc81bede1c9f40e7dd91a541034d9235366","after":"c5cb3efcc003ee8d8352363a6e1361f53febeb68","ref":"refs/heads/main","pushedAt":"2023-04-21T12:33:05.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Fix lora training for time related parameters","shortMessageHtmlLink":"Fix lora training for time related parameters"}},{"before":"748f00a2d996410e641ac5cdea7637b037bbfc25","after":"7ac25dc81bede1c9f40e7dd91a541034d9235366","ref":"refs/heads/main","pushedAt":"2023-04-08T12:03:59.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Fix merge script output.","shortMessageHtmlLink":"Fix merge script output."}},{"before":"ac281860f29a6b6deb88a3adf1f8d1ea50712675","after":"748f00a2d996410e641ac5cdea7637b037bbfc25","ref":"refs/heads/main","pushedAt":"2023-04-07T14:33:45.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Fix path.","shortMessageHtmlLink":"Fix path."}},{"before":"df5689bc88fc2f3334fbbc0117369817b0558b2b","after":"ac281860f29a6b6deb88a3adf1f8d1ea50712675","ref":"refs/heads/main","pushedAt":"2023-04-07T14:32:49.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Add lora merge script.","shortMessageHtmlLink":"Add lora merge script."}},{"before":"5e0262b96a216326e60663506f320af6443b13ef","after":"df5689bc88fc2f3334fbbc0117369817b0558b2b","ref":"refs/heads/main","pushedAt":"2023-03-16T08:09:30.591Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Fix weight merging for chat.py inference","shortMessageHtmlLink":"Fix weight merging for chat.py inference"}},{"before":"a81677edd4dcc54149b0636537777468697a984e","after":"5e0262b96a216326e60663506f320af6443b13ef","ref":"refs/heads/main","pushedAt":"2023-03-15T09:50:10.730Z","pushType":"push","commitsCount":45,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Merge remote-tracking branch 'upstream/main'","shortMessageHtmlLink":"Merge remote-tracking branch 'upstream/main'"}},{"before":"d04dcd74cd9a93167b4e52864ad28408fea3e30e","after":"a81677edd4dcc54149b0636537777468697a984e","ref":"refs/heads/main","pushedAt":"2023-03-15T09:30:23.608Z","pushType":"push","commitsCount":1,"pusher":{"login":"Blealtan","name":"Blealtan","path":"/Blealtan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6376623?s=80&v=4"},"commit":{"message":"Fix lora weight init and gradient checkpoint","shortMessageHtmlLink":"Fix lora weight init and gradient checkpoint"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAADTu6nQgA","startCursor":null,"endCursor":null}},"title":"Activity · Blealtan/RWKV-LM-LoRA"}