Skip to content

Latest commit

 

History

History
659 lines (598 loc) · 43.9 KB

RELEASE.md

File metadata and controls

659 lines (598 loc) · 43.9 KB
Raw

Learning Interpretability Tool Release Notes

Release 1.3

This release updates how the Learning Interpretability Tool (LIT) can be deployed on Google Cloud. You can now use LIT to interpret foundation models—including Gemini, Gemma, Llama, and Mistral—using LIT's prompt debugging workflows. LIT now provides public container images to make it easier to deploy on your hosting platform of choice, with an updated tutorial for deploying LIT with Cloud Run.

New Stuff

Non-breaking Changes, Bug Fixes, and Enhancements

  • Upgrade LIT to MobX v6. - c1f5055
  • Fix indexing issue in Sequence Salience module. - 58b1d2
  • Load multiple model wrappers with shared model. - ba4d975
  • Add the custom model and dataset loaders to prompt debugging notebook. - 338c6b
  • Convert hosted demos images to multi-stage builds. - 4bf1f8
  • Adding testing instructions to README. - f24b841
  • More LIT documentation updates. - 2e9d267

Release 1.2

This release covers clean-ups on various obsolete demos, as well as improved packaging and isolated dependencies on the GLUE, Penguin, Prompt Debugging with Sequence Salience and TyDi demos for easier launch.

New Stuff

Non-breaking Changes, Bug Fixes, and Enhancements

  • Refactor DataService reactions - 483082d
  • Add warm_start option to LitWidget - a5265a4
  • Pretty-printing of Model objects - 4fb3bde
  • Avoid equivalent shuffles in Scrambler - 0d8c0d9
  • Updated gunicorn config for demos running in Docker - b14e3b1
  • Disable embeddings for TyDi - 7ff377f
  • Cast embeddings to float32 before computing distances - 5456011
  • Update colab examples to include installation of the lit-nlp package - 48b029c

Release 1.1.1

This release covers various improvements for sequence salience, including new features in the UI module, support of more LLMs, and detailed tutorial and documentation on how to use the sequence salience module for prompt engineering.

New stuff

Non-breaking Changes, Bug Fixes, and Enhancements

Release 1.1

This release provides the capabilities to interpret and debug the behaviors of Generative AI models in LIT. Specifically, we added sequence salience, which explains the impact of the preceding tokens on the generated tokens produced by the GenAI models. Major changes include:

  • An LM salience module in the LIT UI that computes generations, tokenization, and sequence salience on-demand;
  • Computation of sequence salience at different granularities, from the smallest possible level of tokens, to more interpretable larger spans, such as words, sentences, lines, or paragraphs.
  • Support of OSS modeling frameworks, including KerasNLP and Hugging Face Transformers for sequence salience computation. This release would not have been possible without the work of our contributors. Many thanks to: Ryan Mullins, Ian Tenney, Bin Du, and Cibi Arjun.

New Stuff

Non-breaking Changes, Bug Fixes, and Enhancements

  • Improvements to display various fields and their default ranges - 8a3f366 e63b674 d274508
  • Allow only displaying the UI layouts provided by users - a219863
  • Internal dependency changes - f254fa8 724bdee 2138bd9
  • Fix issues with adding more than one example from counterfactual generators - d4302bd
  • Fix issues with loading SimpleSentimentModel - ac8ed59
  • Notebook widget improvements - cdf79eb
  • Docs updates

Release 1.0

This is a major release, covering many new features and API changes from the dev branch since the v0.5 release over 8 months ago. This release includes a variety of breaking changes meant to simplify various aspects of the LIT API and visual changes to improve usability. This release includes over 250 commits. Major changes include:

  • Refactored python code to remove _with_metadata methods from all component and model classes.
  • Refactored Model and BatchedModel python classes to remove predict_minibatch method.
  • Reworked UI and backend logic for dynamic loading of new datasets and models from the UI. This makes use of the new init_spec methods for datasets and model classes.
    • Added a blank demo with no models or datasets preloaded which allows for dynamic loading of models and datasets through the UI.
  • Refactored to upgrade metrics calculation from a type of interpreter to its own top-level concept.
  • Updated front-end layout code to default to a new layout that includes a full height side-panel on the left side to complement the existing top and bottom panels, providing for more customization of module layouts.
  • Added automatic metrics calculations for multilabel models.
  • Added target selector dropdown for saliency methods.
  • A visual redesign of the Salience Clustering module.
  • Improved searching capabilities in the Data Table module.
  • Improved the Data Table module's display of long strings through a "Show more" capability.
  • Updated to Python 3.10.
  • Updated to Node 18 and Typescript 5.0.
  • Improved documentation pages, now at https://pair-code.github.io/lit/documentation/

This release would not have been possible without the work of our new contributors in 2023. Many thanks to Minsuk Kahng, Nada Hussein, Oscar Wahltinez, Bin Du, and Cibi Arjun for your support and contributions to this project! A full list of contributors to this repo can be found at https://github.com/PAIR-code/lit/graphs/contributors.

Breaking Changes

New Stuff

Non-breaking Changes, Bug Fixes, and Enhancements

Release 0.5

This is a major release, covering many new features from the dev branch since the v0.4 release nearly 11 months ago. Most notably, we're renaming! It's still LIT, but now the L stands for "Learning" instead of "Language", to better reflect the scope of LIT and support for non-text modalities like images and tabular data. Additionally, we've made lots of improvements, including:

  • New modules including salience clustering, tabular feature attribution, and a new Dive module for data exploration (inspired by our prior work on Facets Dive).
  • New demos and tutorials for input salience comparison and tabular feature attribution.
  • Many UI improvements, with better consistency across modules and shared functionality for colors, slicing, and faceting of data.
  • Better performance on large datasets (up to 100k examples), as well as improvements to the type system and new validation routines (--validate) for models and datasets.
  • Download data as CSV directly from tables in the UI, and in notebook mode access selected examples directly from Python.
  • Update to Python 3.9 and TypeScript 4.7.

This release would not have been possible without the work of many new contributors in 2022. Many thanks to Crystal Qian, Shane Wong, Anjishnu Mukherjee, Aryan Chaurasia, Animesh Okhade, Daniel Levenson, Danila Sinopalnikov, Deepak Ramachandran, Rebecca Chen, Sebastian Ebert, and Yilei Yang for your support and contributions to this project!

Breaking Changes

New Stuff

Non-breaking Changes, Bug Fixes, and Enhancements

  • Added Dataset embeddings to Embeddings projector – 78e2e9c, 3c0929f, e7ac98b
  • Added a “sparse” mode to Classification Results – 20a8f31
  • Added “Show only generated” option to Data Table module – 4851c9d
  • Added threshold property for MulticlassPreds that allows for default threshold values other than 0.5 – 5e91b19
  • Added toggle for module duplication direction – 4e05a75
  • Clickable links in the Generated Images module – 8cf8119
  • Constructor parameters for salience interpreters – ab057b5
  • Image upload in Datapoint Editor – a23b146
  • Markdown support in LIT component descriptions – 0eaa00c
  • Selection updates based on interactions in Metrics module – c3b6a0c
  • Support for many. new types of inputs in the Datapoint editor, including GeneratedText, GeneratedTextCandidates, MultiSegmentAnnotation, Tokens, SparseMultilabel, and SparseMultilabelPreds
  • Various styling fixes and code cleanup efforts
  • Docs, FAQ, and README updates

Release 0.4.1

This is a bug fix release aimed at improving visual clarity and common workflows.

The UI has been slightly revamped, bugs have been fixed, and new capabilities have been added. Notable changes include:

  • Adds "open in new tab" feature to LIT Notebook widget
  • Adds support for SparseMultilabelPreds to LIME
  • Improves color consistency across the UI
  • Switching NumPy instead of SciKit Learn for PCA
  • Ensuring all built-in demos are compatible with the Docker
  • Updating the Dockerfile to support run-time DEMO_NAME and DEMO_PORT args
  • Fixed a rendering bug in the Confusion Matrix related column and row spans when "hide empty labels" is turned on

Release 0.4

This release adds a lot of new features. The website and documentation have been updated accordingly.

The UI has been slightly revamped, bugs have been fixed, and new capabilities have been added. Notable changes include:

  • Support for Google Cloud Vertex AI notebooks.
  • Preliminary support for tabular and image data, in addition to NLP models.
  • Addition of TCAV global interpretability method.
  • New counterfactual generators for ablating or flipping text tokens for minimal changes to flip predictions.
  • New counterfactual generator for tabular data for minimal changes to flip predictions.
  • Partial dependence plots for tabular input features.
  • Ability to set binary classification thresholds separately for different facets of the dataset
  • Controls to find optimal thresholds across facets given different fairness constraints, such as demographic parity or equal opportunity.

Release 0.3

This release adds the ability to use LIT directly in colab and jupyter notebooks. The website and documentation have been updated accordingly.

The UI has been slightly revamped, bugs have been fixed, and new capabilities have been added. Notable changes include:

  • Notebook mode added.
  • New annotated text visualization module added.
  • Allow saving/loading of generated datapoints, and dynamic adding of new datasets by path in the UI.
  • Added synchronized scrolling between duplicated modules when comparing datapoints or models.
  • Added a focus service for visually linking focus (i.e. hover) states between components.
  • Allow layouts to be specified on LIT creation in python.

Release 0.2

This release of LIT coincides with the EMNLP 2020 conference, where the LIT paper was presented, and the publication of the LIT website, including tutorials and hosted demos.

The UI has been slightly revamped, bugs have been fixed, and new capabilities have been added.

Release 0.1.1

This release of LIT adds a pip package for easy installation, cleans up some of the code and documentation, and adds more examples.

Release 0.1

This is the initial release of LIT.