Integrate NER Module with the Q&A Transformation Pipeline

[grayJiaaoLi]

User story

1. As a developer
2. I want/need to integrate the NER module within the Q&A transformation pipeline
3. So that the training data for LLM can be enhanced

Acceptance criteria

• Integrate the NER process with the data transformation pipeline

  • Ensure the automation of the whole transformation pipeline
  • Ensure the output data of the pipeline is still prepared for LLM training

• Log the errors and the failed data

• Document the limitation of the upgraded data transformation pipeline

  • Observed problems and bugs
  • Estimate how much the work may require

Definition of done (DoD)

• Added only after week 5
• The same for all features
• Here goes the project specific part

DoD general criteria

• Feature has been fully implemented
• Feature has been merged into the mainline
• All acceptance criteria were met
• Product owner approved features
• All tests are passing
• Developers agreed to release