at config page, system prompt cause jailbreak filter error with gpt-4o-mini

[cyberflying]

Describe the bug

frontend web app log:
2024-11-26T15:08:54.704231547Z ERROR:create_app:Exception in /api/conversation | Error code: 400 - {'error': {'message': "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766", 'type': None, 'param': 'prompt', 'code': 'content_filter', 'status': 400, 'innererror': {'code': 'ResponsibleAIPolicyViolation', 'content_filter_result': {'hate': {'filtered': False, 'severity': 'safe'}, 'jailbreak': {'filtered': True, 'detected': True}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}}}

Expected behavior

A clear and concise description of what you expected to happen.

How does this bug make you feel?

Share a gif from giphy to tells us how you'd feel

---

Debugging information

delete the system prompt, it does work.

[Vinay-Microsoft]

@cyberflying Can you please give the steps to reproduce this Issue or explain in detail what you were trying to do when you observed this Issue? Meanwhile we will try to analyze it more with the information that you have provided.

[cyberflying]

@Vinay-Microsoft , I just deployed this repo and tried to test it.

1. first I encounter error at the admin config page , it seems cannot get ${ORCHESTRATION_STRATEGY} in line 143 in file default.json, I change to "langchain",
2. I chat with the frontend web page, then get the error
   
3. check the web app> deployment center> logs, find the following error message:

2024-11-27T03:00:23.668312676Z ERROR:create_app:Exception in /api/conversation | Error code: 400 - {'error': {'message': "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766", 'type': None, 'param': 'prompt', 'code': 'content_filter', 'status': 400, 'innererror': {'code': 'ResponsibleAIPolicyViolation', 'content_filter_result': {'hate': {'filtered': False, 'severity': 'safe'}, 'jailbreak': {'filtered': True, 'detected': True}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}}}

4. if delete the system prompt, it did work yesterday. but just now, I delete system prompts and also get the same error
5. if change the gpt-4o-mini "Content filter" from "DefaultV2" to "Default", it does work