🚀 Roadmap 🚀

[PeriniM]

We are considering adding the following features to enhance the functionality and flexibility of the library:

✅ TODOs:

• 📝 Update with New Document

  • Goal: When a new or updated document is loaded, the library should recognize entities and relationships that are already present in the existing schema and integrate them with the new information. Functions for matching and identifying overlaps between the current schema and the one extracted from the new document should be implemented.
  • Possible Implementation:
    • Analyze the nodes (entities) and edges (relationships) of the current schema.
    • Deduplication algorithms (e.g., comparing identifiers or entity names).
    • Reconciliation between new entities/relationships and pre-existing ones.

• 📊 Update with Information from Domain Experts

  • Goal: Allow users to manually add specific information to the model, improving output quality with their expertise or adding information from non-structured sources (e.g., images or notes).
  • Possible Implementation:
    • Support input of textual data or other sources (such as images, charts).
    • If the added information is textual, use LLM to parse and integrate the information into the current graph or schema.
    • If the input is an image or other file types, support OCR or visual analysis to extract entities and relationships.

• 🔀 Merge Two Schemas

  • Goal: Merge two schemas from different documents or sources.
  • Possible Implementation:
    • Before merging, establish whether the two schemas refer to the same domain or overlapping topics.
    • Implement algorithms to compare entities in the two schemas and find similarities (e.g., using similarity metrics on entity and relationship names).
    • Choose a merge strategy to resolve conflicts (e.g., handling duplications or inconsistencies).

• 🗑️ Delete Functionality (on Schema, Entities, or Relationships)

  • Goal: Allow users to modify the schema by removing entities or relationships that are no longer relevant.
  • Possible Implementation:
    • Create a simple interface or API where the user can specify the entity or relationship to delete (e.g., via ID or description).
    • Provide users the ability to describe what they want to remove in natural language and use LLM to interpret and apply the request.

• 🔍 Conditional Generation Based on User Prompt

  • Goal: Allow users to specify exactly which parts of the document or schema they want to extract and represent.
  • Possible Implementation:
    • Users provide a specific prompt (e.g., “extract only financial-related entities”).
    • The LLM interprets the prompt and filters the output, keeping only what is relevant to the requested context.

• 🔄 Iterative Refinement of Schema Generation

  • Goal: Improve the schema progressively through an iterative process, where each subsequent step is validated and optimized based on feedback.
  • Possible Implementation:
    • After generating a schema, the system could re-check with the LLM for accuracy or ask for specific improvements, such as adding missing details or removing redundant information.
    • Each cycle could include conditional queries, for example: "Do you see any inconsistencies in this schema?"

• 📜 Change Tracking:

  • Goal: Implement a versioning system that tracks every modification made to the schema so that users can see how it has been updated over time.
  • Possible Implementation:
    • Maintain a log of updates, showing changes in entities and relationships with timestamps and user inputs.

• 🖼️ Interactive Schema Visualization:

  • Goal: Add a graphical visualization feature that shows entities and relationships interactively and in an updatable manner.
  • Possible Implementation:
    • Use libraries like pyecharts for basic visualization, with an interactive layer allowing users to explore the schema.

• 🔗 Integration with External Knowledge Bases:

  • Goal: The system should suggest missing relationships or additional information based on data from external knowledge bases such as Wikidata.
  • Possible Implementation:
    • Use APIs to query external knowledge bases and cross-reference data from the extracted schema, enriching entities and relationships.