[BUG] Bad JSON format on audit logs

[disaster37]

Describe the bug

Hi,

I have enable the JSON log formatter on Opensearch. It works fine, expected on Audit logs (comonent: audit).
I try to decode the field message with Json filter on Logstash, but it failed. After look the JSON validity on field ``message, you can look there are not "}` at the end. So the JSON is not valide.

{
    "type": "json_logger",
    "timestamp": "2024-09-12T07:28:04,224Z",
    "level": "INFO",
    "component": "audit",
    "cluster.name": "logmanagement2-prd",
    "node.name": "opensearch-master-os-2",
    "message": "{\"audit_trace_task_parent_id\":\"874Nj1EFSEma4EGc8ucJfw:319488\",\"audit_cluster_name\":\"logmanagement2-prd\",\"audit_transport_headers\":{\"_opendistro_security_remote_address_header\":\"rO0ABXNyABpqYXZhLm5ldC5JbmV0U29ja2V0QWRkcmVzc0ZxlGFv+apFAwADSQAEcG9ydEwABGFkZHJ0ABZMamF2YS9uZXQvSW5ldEFkZHJlc3M7TAAIaG9zdG5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cgAWamF2YS5uZXQuU29ja2V0QWRkcmVzc0hh9mL0l51qAgAAeHAAAKsQc3IAFGphdmEubmV0LkluZXRBZGRyZXNzLZtXr5/j69sDAANJAAdhZGRyZXNzSQAGZmFtaWx5TAAIaG9zdE5hbWVxAH4AAnhwCioLLAAAAAJweHB4\",\"_opendistro_security_initial_action_class_header\":\"IndexRequest\",\"_opendistro_security_origin_header\":\"REST\",\"_opendistro_security_user_header\":\"rO0ABXNyACJjb20uYW1hem9uLmRsaWMuYXV0aC5sZGFwLkxkYXBVc2VyAAAAAAAAAAECAAFMABBvcmlnaW5hbFVzZXJuYW1ldAASTGphdmEvbGFuZy9TdHJpbmc7eHIAIW9yZy5vcGVuc2VhcmNoLnNlY3VyaXR5LnVzZXIuVXNlcrOovZPrl0fcAgAGWgAKaXNJbmplY3RlZEwACmF0dHJpYnV0ZXN0AA9MamF2YS91dGlsL01hcDtMAARuYW1lcQB+AAFMAA9yZXF1ZXN0ZWRUZW5hbnRxAH4AAUwABXJvbGVzdAAPTGphdmEvdXRpbC9TZXQ7TAANc2VjdXJpdHlSb2xlc3EAfgAEeHAAc3IAJWphdmEudXRpbC5Db2xsZWN0aW9ucyRTeW5jaHJvbml6ZWRNYXAbc/kJS0s5ewMAAkwAAW1xAH4AA0wABW11dGV4dAASTGphdmEvbGFuZy9PYmplY3Q7eHBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAwdwgAAABAAAAALXQAGGF0dHIubGRhcC5wcmltYXJ5R3JvdXBJRHQAAzUxM3QAG2F0dHIubGRhcC51bml4SG9tZURpcmVjdG9yeXQAFi9ob21lL3N1LmxhbmdvdXJlYXV4LXN0ABhhdHRyLmxkYXAubXNUU0V4cGlyZURhdGV0ABEyMDIyMDUxNDA5MTkzNi4wWnQAE2F0dHIubGRhcC5sYXN0TG9nb250ABIxMzM3MDQ1NTQzMDExMzUzOTF0ABRhdHRyLmxkYXAubG9nb25Db3VudHQABDE1MzB0ABVhdHRyLmxkYXAuYmFkUHdkQ291bnR0AAEwdAAUYXR0ci5sZGFwLm9iamVjdEdVSUR0ABwT77+9ISzvv71WLkrvv71977+91YIX77+977+9dAAcYXR0ci5sZGFwLnVzZXJBY2NvdW50Q29udHJvbHQAAzUxMnQAHWF0dHIubGRhcC5tc1RTTGljZW5zZVZlcnNpb24zdAAKQzUwLTYuMDItU3QAHWF0dHIubGRhcC5tc1RTTGljZW5zZVZlcnNpb24ydAABN3QAH2F0dHIubGRhcC5tUy1EUy1Db25zaXN0ZW5jeUd1aWR0ABwT77+9ISzvv71WLkrvv71977+91YIX77+977+9dAAVYXR0ci5sZGFwLndoZW5DcmVhdGVkdAARMjAxMTA0MTMxNTA5NTguMFp0ABZsZGFwLm9yaWdpbmFsLnVzZXJuYW1ldAAQc3UubGFuZ291cmVhdXgtc3QAFGF0dHIubGRhcC5sYXN0TG9nb2ZmdAABMHQAFWF0dHIubGRhcC5jb3VudHJ5Q29kZXQAATB0AB5hdHRyLmxkYXAuZXh0ZW5zaW9uQXR0cmlidXRlMTV0AApBSVhfQURNX04ydAAWYXR0ci5sZGFwLmluc3RhbmNlVHlwZXQAATR0ABhhdHRyLmxkYXAuc0FNQWNjb3VudE5hbWV0ABBzdS5sYW5nb3VyZWF1eC1zdAAYYXR0ci5sZGFwLm1zVFNNYW5hZ2luZ0xTdAAXMDAyNTItNzAwMDAtNDU2OTUtQVQzMzd0ABthdHRyLmxkYXAudXNlclByaW5jaXBhbE5hbWV0ABlzdS5sYW5nb3VyZWF1eC1zQEhNLkRNLkFEdAAVYXR0ci5sZGFwLndoZW5DaGFuZ2VkdAARMjAyNDA5MDkwNzQ2MTcuMFp0ABVhdHRyLmxkYXAub2JqZWN0Q2xhc3N0AAN0b3B0AA1hdHRyLmxkYXAudWlkdAAQc3UubGFuZ291cmVhdXgtc3QAE2F0dHIubGRhcC5naXZlbk5hbWV0AANzdS50AAdsZGFwLmRudABSQ049c3UubGFuZ291cmVhdXgtcyxPVT1PMzY1LE9VPVByb2R1Y3Rpb24sT1U9Q3B0ZSBTVSxPVT1TZXJ2aWNlcyxEQz1ITSxEQz1ETSxEQz1BRHQAGGF0dHIubGRhcC5zQU1BY2NvdW50VHlwZXQACTgwNTMwNjM2OHQADGF0dHIubGRhcC5jbnQAEHN1LmxhbmdvdXJlYXV4LXN0ABhhdHRyLmxkYXAuYWNjb3VudEV4cGlyZXN0ABM5MjIzMzcyMDM2ODU0Nzc1ODA3dAAVYXR0ci5sZGFwLmRlc2NyaXB0aW9udAAeQ29tcHRlIFNVIGdyb3VwZXMgQWRtaW4gU3lzYWlkdAAfYXR0ci5sZGFwLmRTQ29yZVByb3BhZ2F0aW9uRGF0YXQAETIwMjQwNzA5MTU0ODI2LjBadAATYXR0ci5sZGFwLmdpZE51bWJlcnQABjIwNDMwNHQAFWF0dHIubGRhcC5sb2Nrb3V0VGltZXQAATB0AA5hdHRyLmxkYXAubmFtZXQAEHN1LmxhbmdvdXJlYXV4LXN0ABRhdHRyLmxkYXAudVNOQ3JlYXRlZHQABTU0MzgydAAUYXR0ci5sZGFwLnVTTkNoYW5nZWR0AAozODY1NDg4MzYwdAAnYXR0ci5sZGFwLm1zRFMtU3VwcG9ydGVkRW5jcnlwdGlvblR5cGVzdAABMHQAE2F0dHIubGRhcC51aWROdW1iZXJ0AAYxMzE2ODJ0ABVhdHRyLmxkYXAuZGlzcGxheU5hbWV0ABBzdS5sYW5nb3VyZWF1eC1zdAATYXR0ci5sZGFwLm9iamVjdFNpZHQANAEFwIDAgMCAwIDAgAUVwIDAgMCA77+977+9W++/ve+/vQJX77+9cQQgX++/ve+/vcCAwIB0ABJhdHRyLmxkYXAuY29kZVBhZ2V0AAEwdAAUYXR0ci5sZGFwLnB3ZExhc3RTZXR0ABIxMzM2NzMxNzE4NjM3ODAzNjZ0AAxhdHRyLmxkYXAuc250AA1sYW5nb3VyZWF1eC1zdAAUYXR0ci5sZGFwLmxvZ2luU2hlbGx0AAwvdXNyL2Jpbi9rc2h0ABxhdHRyLmxkYXAubXNUU0xpY2Vuc2VWZXJzaW9udAAGMzkzMjE4dAAcYXR0ci5sZGFwLmxhc3RMb2dvblRpbWVzdGFtcHQAEjEzMzcwMzQxNTc3Mzk4NTc2OHhxAH4ACHhxAH4ALnQACmV0bF9vdXRpbHNzcgAlamF2YS51dGlsLkNvbGxlY3Rpb25zJFN5bmNocm9uaXplZFNldAbDwnkC7t88AgAAeHIALGphdmEudXRpbC5Db2xsZWN0aW9ucyRTeW5jaHJvbml6ZWRDb2xsZWN0aW9uKmH4TQmcmbUDAAJMAAFjdAAWTGphdmEvdXRpbC9Db2xsZWN0aW9uO0wABW11dGV4cQB+AAd4cHNyABFqYXZhLnV0aWwuSGFzaFNldLpEhZWWuLc0AwAAeHB3DAAAAIA/QAAAAAAAXXQAakNOPUhNXzM3Q1RJMDBfRklDMzdDTFUyX0RUUzM3Q1RJNV9XLE9VPURyb2l0cyBSw6lwZXJ0b2lyZXMsT1U9SE0sT1U9R3JvdXBlcyBVdGlsaXNhdGV1cnMsREM9SE0sREM9RE0sREM9QUR0AHdDTj1ITV9BRE1fQ1RJX1NSMDcyOTE3Q1RJMzcwMF9OMyxPVT1MaW51eCxPVT1Ecm9pdHMgQWRtaW5pc3RyYXRpb24gUHJvZCxPVT1ITSxPVT1Hcm91cGVzIFV0aWxpc2F0ZXVycyxEQz1ITSxEQz1ETSxEQz1BRHQATkNOPUhNX1JCQUNfVVNSX1dJTkRPV1NfQURNSU4sT1U9UkJBQyxPVT1Hcm91cGVzIFV0aWxpc2F0ZXVycyxEQz1ITSxEQz1ETSxEQz1BRHQAYUNOPUhNXzM3Q1RJMDBfRFRIUDAxX0FETUlOLE9VPURyb2l0cyBSw6lwZXJ0b2lyZXMsT1U9SE0sT1U9R3JvdXBlcyBVdGlsaXNhdGV1cnMsREM9SE0sREM9RE0sREM9QUR0AG1DTj1ITV9BRE1fQ1RJX0xJTlVYX04zLE9VPUxpbnV4LE9VPURyb2l0cyBBZG1pbmlzdHJhdGlvbiBQcm9kLE9VPUhNLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdAB3Q049SE1fQURNX0NUSV9TUjA3MjkyMENUSTM3MDBfTjMsT1U9TGludXgsT1U9RHJvaXRzIEFkbWluaXN0cmF0aW9uIFByb2QsT1U9SE0sT1U9R3JvdXBlcyBVdGlsaXNhdGV1cnMsREM9SE0sREM9RE0sREM9QUR0AHdDTj1ITV9BRE1fQ1RJX1NSMDcyNzU3Q1RJMzcwMF9OMyxPVT1MaW51eCxPVT1Ecm9pdHMgQWRtaW5pc3RyYXRpb24gUHJvZCxPVT1ITSxPVT1Hcm91cGVzIFV0aWxpc2F0ZXVycyxEQz1ITSxEQz1ETSxEQz1BRHQAZENOPUhNX0tBRktBX0FQSUNVUklPX0FETUlOX0hQRCxPVT1Ecm9pdHMgQXBwbGljYXRpZnMsT1U9SE0sT1U9R3JvdXBlcyBVdGlsaXNhdGV1cnMsREM9SE0sREM9RE0sREM9QUR0AGFDTj1ITV8zN0NUSTAwX0RUTEQwMV9BRE1JTixPVT1Ecm9pdHMgUsOpcGVydG9pcmVzLE9VPUhNLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdABhQ049SE1fU1VfT0NQX0hQUkRfT1VUSUxMQUdFLE9VPURyb2l0cyBBcHBsaWNhdGlmcyxPVT1ITSxPVT1Hcm91cGVzIFV0aWxpc2F0ZXVycyxEQz1ITSxEQz1ETSxEQz1BRHQAd0NOPUhNX0FETV9DVElfU1IwNjc3NDdDVEkzNzAwX04zLE9VPUxpbnV4LE9VPURyb2l0cyBBZG1pbmlzdHJhdGlvbiBQcm9kLE9VPUhNLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdABsQ049SE1fMzdDVEkwMF9GSUMzN0NMVThfRlRQX1RCWFBERjFfVyxPVT1Ecm9pdHMgUsOpcGVydG9pcmVzLE9VPUhNLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdAB3Q049SE1fQURNX0NUSV9TUjA3Mjc2MENUSTM3MDBfTjMsT1U9TGludXgsT1U9RHJvaXRzIEFkbWluaXN0cmF0aW9uIFByb2QsT1U9SE0sT1U9R3JvdXBlcyBVdGlsaXNhdGV1cnMsREM9SE0sREM9RE0sREM9QUR0AG1DTj1ITV9BRE1fQ1RJX0xJTlVYX04yLE9VPUxpbnV4LE9VPURyb2l0cyBBZG1pbmlzdHJhdGlvbiBQcm9kLE9VPUhNLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdABnQ049SE1fQURNX1NZU0FJRF9EUExfUkVDLE9VPURyb2l0cyBBZG1pbmlzdHJhdGlvbiBNaWNybyxPVT1ITSxPVT1Hcm91cGVzIFV0aWxpc2F0ZXVycyxEQz1ITSxEQz1ETSxEQz1BRHQAd0NOPUhNX0FETV9DVElfU1IwNzI5MTZDVEkzNzAwX04zLE9VPUxpbnV4LE9VPURyb2l0cyBBZG1pbmlzdHJhdGlvbiBQcm9kLE9VPUhNLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdABUQ049SE1fTzM2NV9MSUNfQUFEX1AxLE9VPURyb2l0cyBPMzY1LE9VPUhNLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdAB3Q049SE1fQURNX0NUSV9TUjA4MjUzMUNUSTM3MDBfTjMsT1U9TGludXgsT1U9RHJvaXRzIEFkbWluaXN0cmF0aW9uIFByb2QsT1U9SE0sT1U9R3JvdXBlcyBVdGlsaXNhdGV1cnMsREM9SE0sREM9RE0sREM9QUR0AHdDTj1ITV9BRE1fQ1RJX1NSMDcyOTI2Q1RJMzcwMF9OMyxPVT1MaW51eCxPVT1Ecm9pdHMgQWRtaW5pc3RyYXRpb24gUHJvZCxPVT1ITSxPVT1Hcm91cGVzIFV0aWxpc2F0ZXVycyxEQz1ITSxEQz1ETSxEQz1BRHQAd0NOPUhNX0FETV9DVElfU1IwNzI5MjFDVEkzNzAwX04zLE9VPUxpbnV4LE9VPURyb2l0cyBBZG1pbmlzdHJhdGlvbiBQcm9kLE9VPUhNLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdABfQ049SE1fU1RBUkJVUlNUX0hQRF9BQ0NFUyxPVT1Ecm9pdHMgQXBwbGljYXRpZnMsT1U9SE0sT1U9R3JvdXBlcyBVdGlsaXNhdGV1cnMsREM9SE0sREM9RE0sREM9QUR0AElDTj1ITV9SQkFDX1VTUl9MSU5VWF9OMyxPVT1SQkFDLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdAB3Q049SE1fQURNX0NUSV9TUjA3MjkxOUNUSTM3MDBfTjMsT1U9TGludXgsT1U9RHJvaXRzIEFkbWluaXN0cmF0aW9uIFByb2QsT1U9SE0sT1U9R3JvdXBlcyBVdGlsaXNhdGV1cnMsREM9SE0sREM9RE0sREM9QUR0AHdDTj1ITV9BRE1fQ1RJX1NSMDcyNzY1Q1RJMzcwMF9OMyxPVT1MaW51eCxPVT1Ecm9pdHMgQWRtaW5pc3RyYXRpb24gUHJvZCxPVT1ITSxPVT1Hcm91cGVzIFV0aWxpc2F0ZXVycyxEQz1ITSxEQz1ETSxEQz1BRHQAakNOPUhNXzM3Q1RJMDBfRklDMzdDTFUyX0RUUzM3RVRVOF9XLE9VPURyb2l0cyBSw6lwZXJ0b2lyZXMsT1U9SE0sT1U9R3JvdXBlcyBVdGlsaXNhdGV1cnMsREM9SE0sREM9RE0sREM9QUR0AGpDTj1ITV8zN0NUSTAwX0ZJQzM3Q0xVMl9EVFMzN0VUVTdfVyxPVT1Ecm9pdHMgUsOpcGVydG9pcmVzLE9VPUhNLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdABsQ049SE1fS0FGS0FfU0VSVklDRV9SRUdJU1RSWV9BRE1JTl9IUEQsT1U9RHJvaXRzIEFwcGxpY2F0aWZzLE9VPUhNLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdABoQ049SE1fQURNX1NZU0FJRF9EUExfUFJPRCxPVT1Ecm9pdHMgQWRtaW5pc3RyYXRpb24gTWljcm8sT1U9SE0sT1U9R3JvdXBlcyBVdGlsaXNhdGV1cnMsREM9SE0sREM9RE0sREM9QUR0AHdDTj1ITV9BRE1fQ1RJX1NSMDcyOTIyQ1RJMzcwMF9OMyxPVT1MaW51eCxPVT1Ecm9pdHMgQWRtaW5pc3RyYXRpb24gUHJvZCxPVT1ITSxPVT1Hcm91cGVzIFV0aWxpc2F0ZXVycyxEQz1ITSxEQz1ETSxEQz1BRHQAZUNOPUhNX1NVUF9DTFVfRklDX1BST0QsT1U9RHJvaXRzIEFkbWluaXN0cmF0aW9uIE1pY3JvLE9VPUhNLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdABkQ049SE1fS0FGS0FfQVBJQ1VSSU9fQURNSU5fUFJELE9VPURyb2l0cyBBcHBsaWNhdGlmcyxPVT1ITSxPVT1Hcm91cGVzIFV0aWxpc2F0ZXVycyxEQz1ITSxEQz1ETSxEQz1BRHQAaENOPUhNX1NUQVJCVVJTVF9ST0xFX0RTSVRfQURNSU5fTjMsT1U9RHJvaXRzIEFwcGxpY2F0aWZzLE9VPUhNLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdAB3Q049SE1fQURNX0NUSV9TUjA3Mjc1NUNUSTM3MDBfTjMsT1U9TGludXgsT1U9RHJvaXRzIEFkbWluaXN0cmF0aW9uIFByb2QsT1U9SE0sT1U9R3JvdXBlcyBVdGlsaXNhdGV1cnMsREM9SE0sREM9RE0sREM9QUR0AGdDTj1ITV9BRE1fU1lTQUlEX0RQTF9ERVYsT1U9RHJvaXRzIEFkbWluaXN0cmF0aW9uIE1pY3JvLE9VPUhNLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdABfQ049SE1fU1RBUkJVUlNUX1JFQ19BQ0NFUyxPVT1Ecm9pdHMgQXBwbGljYXRpZnMsT1U9SE0sT1U9R3JvdXBlcyBVdGlsaXNhdGV1cnMsREM9SE0sREM9RE0sREM9QUR0AGpDTj1ITV8zN0NUSTAwX0ZJQzM3Q0xVMl9EVFMzN0VUVTZfVyxPVT1Ecm9pdHMgUsOpcGVydG9pcmVzLE9VPUhNLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdABjQ049SE1fQURNX1NSVl9NT09fUFJELE9VPURyb2l0cyBBZG1pbmlzdHJhdGlvbiBQcm9kLE9VPUhNLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdABhQ049SE1fMzdDVEkwMF9EVEhEMDJfQURNSU4sT1U9RHJvaXRzIFLDqXBlcnRvaXJlcyxPVT1ITSxPVT1Hcm91cGVzIFV0aWxpc2F0ZXVycyxEQz1ITSxEQz1ETSxEQz1BRHQAd0NOPUhNX0FETV9DVElfU1IwNzI5MTRDVEkzNzAwX04zLE9VPUxpbnV4LE9VPURyb2l0cyBBZG1pbmlzdHJhdGlvbiBQcm9kLE9VPUhNLE9VPUdyb3VwZXMgVXRpbGlzYXRldXJzLERDPUhNLERDPURNLERDPUFEdABfQ049SE1fQVBQTElfUlVOREVDS19BRE1JTixPVT1Ecm9pdHMgQXBwbGljYXRpZnMsT1U9SE0sT1U9R3JvdXBlcyBVdGlsaXNhdGV1cnMsREM9SE0sREM9RE0sREM9QUR0AFpDTj1ITV9BRE1fRVRMX091dGlscyxPVT1Ecm9pdHMgQXBwbGljYXRpZnMsT1U9SE0sT1U9R3JvdXBlcyBVdGlsaXNhdGV1cnMsREM",
    "cluster.uuid": "ZVJvcMA3TmK00qqKawBTjg",
    "node.id": "6aPiODxfRtWdU073wMGoLA"
}

Related component

Other

To Reproduce

1. Start Opensearch 2.16.0
2. Log on it to generate audit logs

Expected behavior

The field message must contain valid JSON string when component is audit.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.