Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Uptrain Callback Handler #10432

Merged
merged 16 commits into from
Feb 16, 2024

Conversation

Dominastorm
Copy link
Contributor

Description

Add the UpTrainCallbackHandler to perform evaluations on query, context and response in the llamindex pipeline.

Fixes # (issue)

Type of Change

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • Added new notebook (that tests end-to-end)
  • I stared at the code and made sure it makes sense

Suggested Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added Google Colab support for the newly added notebooks.
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I ran make format; make lint to appease the lint gods

@dosubot dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Feb 3, 2024
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@Dominastorm
Copy link
Contributor Author

Hey @hatianzhang,

Just wanted to give a friendly ping on this PR. It's been a little over a week since I submitted it, and I understand everyone's busy. I just wanted to make sure it didn't slip through the cracks.

If there's anything I can do to help move this forward or provide additional context, please let me know!

Thanks!

@hatianzhang
Copy link
Contributor

@Dominastorm thanks for your contribution.
We are now in code freeze so I am not going to merge your pr atm. Once our 0.10.0 going out. I will merge this one asap

@logan-markewich
Copy link
Collaborator

@Dominastorm apologies, with v0.10.0 the project structure has changed slightly

I can take a stab at porting this PR in a bit, if you don't get to it first :)

@Dominastorm
Copy link
Contributor Author

@logan-markewich thanks for the offer! I'll port it, no worries :)

@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Feb 14, 2024
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Feb 14, 2024
@Dominastorm
Copy link
Contributor Author

@logan-markewich, I have made the changes for porting. I tested it locally and it is working in the python notebook. However, I am unable to understand the test that is failing. If you could let me know what exactly it means, I can get started on fixing it. Thanks!

@logan-markewich
Copy link
Collaborator

@Dominastorm It seems I cant push to your PR, but I can tell you how to make CICD pass :)

Install pants
Run pants tailor ::
add/commit/push and BUILD files it created

@Dominastorm
Copy link
Contributor Author

Thanks @logan-markewich! The PR is ready for review now :)

Copy link
Contributor

@hatianzhang hatianzhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Feb 16, 2024
@hatianzhang hatianzhang merged commit a8544e8 into run-llama:main Feb 16, 2024
8 checks passed
Dominastorm added a commit to uptrain-ai/llama_index that referenced this pull request Feb 28, 2024
* Add UpTrain Callback Handler

* Add logging

* Add note for project_name prefix

* Minor fixes

* Add explanations

* Add showcase

* Add callback to uptrain.md

* update import

* linter

* Update .gitignore file to ignore .venv and __pycache__ directories

* Fix imports

* create llama-index-callback-uptrain

* Update UpTrainCallback.ipynb

* Update README.md

* Run pants tailor
Izukimat pushed a commit to Izukimat/llama_index that referenced this pull request Mar 29, 2024
* Add UpTrain Callback Handler

* Add logging

* Add note for project_name prefix

* Minor fixes

* Add explanations

* Add showcase

* Add callback to uptrain.md

* update import

* linter

* Update .gitignore file to ignore .venv and __pycache__ directories

* Fix imports

* create llama-index-callback-uptrain

* Update UpTrainCallback.ipynb

* Update README.md

* Run pants tailor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lgtm This PR has been approved by a maintainer size:XL This PR changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants