Skip to content

GSoC 2024 ‐ AI‐Powered Summarization and “Interaction” with Academic Papers

Jump to bottom
Christoph edited this page Sep 9, 2024 · 3 revisions
Student Ruslan Popov
Organization JabRef e.V.
Primary repository JabRef/jabref
Project name AI-Powered Summarization and “Interaction” with Academic Papers
Project mentors @koppor, @ThiloteE, @Calixtus
Project page Google Summer of Code Project Page
Status Complete

Project

Description

During my Google Summer of Code (GSoC) project, I was working on enhancing JabRef with AI features to assist researchers in their work. Given the current popularity and enhanced usability of AI technologies, my mentors and I aimed to integrate Large Language Models (LLMs) that would analyze research papers.

To achieve this, the project introduced three core AI features:

  • Chatting with Library Entries. This feature allows users to interact with their library entries through a chat interface. Users can ask questions about specific papers, and the AI will provide relevant answers, making it easier to find and process information without manually searching through the text.
  • Generating Summaries. The summarization feature provides a concise overview of research papers. This helps researchers quickly understand the content of a paper before deciding whether to read it in full, saving time and effort.
  • Supporting Multiple AI Providers. To ensure flexibility and robustness, I integrated support for multiple AI providers. This allows users to choose the AI model that best suits their needs and preferences.

The implementation was primarily done in Java using JavaFX, with the integration of the LangChain4j library (a Java counterpart to the Python-based LangChain). The embedding model part was done with the Java based DeepJavaLibrary.

The AI features developed in this project have the potential to significantly improve the workflow of researchers by providing them with an AI assistant capable of answering questions and generating summaries. This not only saves time but also enhances the overall research process.

Challenges and important things I learned during the project

There were many challanges during the development of this project, and I developed a lot of skills and knowledge by overcoming them:

  • LangChain4j was not easily usable in the JabRef development due to JabRef making use of JDK's modularization features. However, with the 0.33 release of Langchain4j, the problem was solved. Nevertheless, I learned a lot about Java build systems, when we were trying to develop JabRef with older versions of LangChain4j.
  • At the beginning, it was not possible to debug JabRef on Windows, because the command line generated by Gradle grew too large. This issue is related to a previous one with split packages. Patching modules introduced a lot of CLI arguments, so Windows could not handle them properly. Moreover, the dependency on LangChain4j introduced new modules (which were added to the command line, too). Handling of long command lines with Gradle was "fixed" by gradle-modules-plugin#282.
  • In some cases, our application cannot shutdown, because LangChain4j was running some threads that were not properly closed. That forced us to dive deep into its source code and use jvm-openai as the underlying OpenAI client.
  • In the end, we also found out that the application size with LangChain4j was enormous. The reason for this problem was the usage of ONNX library in LangChain4j, it was supplying all possible implementations for various platforms in one package. Also, it included the debugging symbols for Windows implementation (.pdb file that weighed 200 MBs!). To solve this problem, we partially switched to djl library.
  • On the frontend side, there is a long-standing issue in JavaFX that text of labels should be selectable/copyable. This severely affects how we develop chat UI because, on one hand, users need a comfortable UI (which can be made with Label or Text element), but, on the other hand, users will want a copy feature (which is only available in TextField or TextArea components). We still have many issues with the chat UI and scrolling down the messages.

All those issues were unexpected at the beginning of the project, and they caused headaches at my and mentor's side. With having time lost at these unexpected Java ecosystem issues, we could not work on JabRef itself.

Initially, understanding the existing code base and learning best practices was challenging. However, with time and guidance from mentors, I became more proficient. Integrating AI features into the existing application required overcoming numerous bugs and architectural challenges, but most issues were successfully resolved!

Throughout the development process, I gained valuable insights into Software Engineering. I identified and fixed several small bugs in the original JabRef codebase and discovered several issues in other libraries. These experiences have significantly enhanced my problem-solving skills and, especially, understanding of integrating AIs in software applications.

Work done

Chatting with AI

  • Сonnect to LLM.
  • Store chat history.
  • Delete messages.
  • Process errors from AI.

Chatting with AI screenshot

Processing errors from AI

Chatting with a group

Generating summary

  • Generating summary in background: just as embeddings generation, with all information needed.
  • Handle both small and large documents.
  • Handle models with both small and big context window.
  • Show the summary and record the time of the genera ted summary and model that was used.
  • Add the ability to regenerate summary.

Summary

Waiting for summary in AI summary tab

Summary generation background task

Generating embeddings in background

  • Tell users that they need to wait before chatting with attached files.
  • Generate embeddings in the background: it also shows how much work was and should be done, with ETA estimation.

Waiting for embedding generation in AI chat Embeddings generation background task

Privacy notice

  • Tell users how AI features work.
  • Give users link to privacy policies of AI providers.
  • Add the ability to agree or disagree (if user doesn't click "I agree", nothing will happen, they can turn off "AI Chat" and "AI Summary" in preferences).

privacy-notice

entry-editor-settings

AI preferences

  • Choose chat model.
  • Supply an API token.

Basic AI preferences

Support for multiple AI providers

  • Choose any provider you like.
  • The available model list is updated according to selected AI provider.
  • The API token is saved separately for AI providers.
  • For Hugging Face there is no supplied list of models (because it's too large), instead users can enter model name manually.

List of AI providers

Expert settings

  • Users can customize how AI will respond to their requests.

expert-settings

Future work

Future features include:

  • Introduce AI-assistant search for papers in the context of a research group sharing their references and pdf files using a BibTeX file.
  • Support full offline mode (no external access to network).
  • Support for external RAG (all the workload for generating and storing embeddings, generating AI answer is offloaded to a separate server). This could be implemented using Microsoft Kernel Memory.
  • Integrate scholar.ai, or other services.

Pull requests

Project-related work

Improving JabRef in addition to the project aims

Pull requests to other repositories

Issues raised

JabRef

Other repositories

JPro-Platform

classpath-to-file-gradle-plugin

LangChain4j

arxiv.py

GemsFX

h2database

djl

Jan.ai Website & Documentation

Blog posts

Acknowledgements

I want to express my gratitude to those people and companies:

Clone this wiki locally