RougeSimilarityFilter Should Consider Input Field for Filtering

[coolbeevip]

I'm encountering an issue while using the SelfInstructPipeline for data generation. My goal is to keep the instruction field constant while generating diverse input and output pairs. However, it appears that the RougeSimilarityFilter is only being applied to the values within the instruction field. This prevents me from generating new data and results in repetitive outputs.

Here's an example of my data samples:

{
    "instruction": "Extract time information from the following text",
    "input": "Did you go to the party last Friday? It was March 7th.",
    "output": "[\"last Friday\", \"March 7th\"]"
}
{
    "instruction": "Extract time information from the following text",
    "input": "I remember last winter, when we went skiing. It was December 25th.",
    "output": "[\"last winter\", \"December 25th\"]"
}
{
    "instruction": "Extract time information from the following text",
    "input": "We plan to hold a meeting on the 15th of next month.",
    "output": "[\"next month\", \"15th\"]"
}
{
    "instruction": "Extract time information from the following text",
    "input": "Yesterday was my birthday, which was August 1st.",
    "output": "[\"yesterday\", \"August 1st\"]"
}

I understand that disabling the RougeSimilarityFilter would bypass this issue. However, I would prefer for it to also consider the input field during the filtering process. This would allow for more varied data generation while still maintaining control over similarity.

Furthermore, I'd like to suggest the addition of a system field to define roles or contexts. For example:

{
    "system": "Extract time information from the following text",
    "instruction": "Did you go to the party last Friday? It was March 7th.",
    "input": "",
    "output": "[\"last Friday\", \"March 7th\"]"
}

I noticed that Alpaca does not define a system field (https://github.com/tatsu-lab/stanford_alpaca). Could you provide any insights or suggestions on how to address my issue with the RougeSimilarityFilter and whether the addition of a system field is a feasible or recommended approach?