Xsiam jira integration (demisto#18711)

* create the pack

* files

* fixes

* before uploading

* fix the command

* fetch

* fetch true

* added url to wl

* revert

* demisto.incidents

* add delta

* changes

* fix the last fun null

* all ut

* add readme

* docker tag

* remove author image

* remove report pytest

* fix UT

* fix lint errors

* add docstring

* typo

* remove ut

* fix ut

* inside with

* add pydantic to white list

* Update .pylintrc

* xsiam requirements

* RN

* add XSIAM to known words

* fetch incidents to fetch events

* fetch true

* fetch keys

* revert changes

* changed the integration name

* cr fixes

* fix the ut

* add updateModuleHealth

* added integration description

* update docker image

* cr fixes

* ignore py lint error no name in module

* hotspot

* fix ut

* revert changes

* added the modeling rules

* cr fixes

* cr fixes

* RN

* server version

* isfetchevents

* Update JiraEventCollector.yml

* Update README.md

I haven't changed the parameters such as Incident type, Fetch incidents, Your Server URL, etc. I'm not sure whether you want to change this to Alert type, fetches alerts, etc

* lower case

* docs fixes

* more docs fixes

* readme fixes

* fix ut

* patch send_events

Co-authored-by: Richard Bluestone <[email protected]>

Packs/Jira/.secrets-ignore

@@ -1,3 +1,5 @@
 docs.atlassian.com
 https://test.atlassian.net
-https://test2.atlassian.net
+https://test2.atlassian.net
+[email protected]
+[email protected]

Packs/Jira/Integrations/JiraEventCollector/JiraEventCollector.py

@@ -0,0 +1,202 @@
+from enum import Enum
+import urllib3
+from CommonServerPython import *
+import demistomock as demisto
+from pydantic import BaseConfig, BaseModel, AnyUrl, Json, Field  # pylint: disable=no-name-in-module
+import requests
+from requests.auth import HTTPBasicAuth
+import dateparser
+from datetime import datetime
+
+urllib3.disable_warnings()
+
+DATETIME_FORMAT = "%Y-%m-%dT%H:%M:%S.%f"
+
+
+class Method(str, Enum):
+    GET = 'GET'
+    POST = 'POST'
+    PUT = 'PUT'
+    HEAD = 'HEAD'
+    PATCH = 'PATCH'
+    DELETE = 'DELETE'
+
+
+class Args(BaseModel):
+    from_day = demisto.params().get('first_fetch', '3 days')
+    default_from_day = datetime.now() - timedelta(days=3)
+    from_: str = Field(
+        datetime.strftime(dateparser.parse(from_day, settings={'TIMEZONE': 'UTC'}) or default_from_day, DATETIME_FORMAT),
+        alias='from'
+    )
+    limit: int = 1000
+    offset: int = 0
+
+
+class ReqParams(BaseModel):
+    from_day = demisto.params().get('first_fetch', '3 days')
+    default_from_day = datetime.now() - timedelta(days=3)
+    from_: str = Field(
+        datetime.strftime(dateparser.parse(from_day, settings={'TIMEZONE': 'UTC'}) or default_from_day, DATETIME_FORMAT),
+        alias='from'
+    )
+    limit: int = 1000
+    offset: int = 0
+
+
+class Request(BaseModel):
+    method: Method = Method.GET
+    url: AnyUrl
+    headers: Union[Json, dict] = {}
+    params: ReqParams
+    insecure: bool = Field(not demisto.params().get('insecure', False), alias='verify')
+    proxy: bool = Field(demisto.params().get('proxy', False), alias='proxies')
+    data: Optional[str]
+    auth: Optional[HTTPBasicAuth] = Field(
+        HTTPBasicAuth(
+            demisto.params().get('credentials', {}).get('identifier'),
+            demisto.params().get('credentials', {}).get('password')
+        )
+    )
+
+    class Config(BaseConfig):
+        arbitrary_types_allowed = True
+
+
+class Client:
+    def __init__(self, request: Request, session=requests.Session()):
+        self.request = request
+        self.session = session
+        self._set_proxy()
+        self._set_cert_verification()
+
+    def __del__(self):
+        try:
+            self.session.close()
+        except AttributeError as err:
+            demisto.debug(f'Ignore exceptions raised due to session not used by the client. {err}')
+
+    def call(self) -> requests.Response:
+        try:
+            response = self.session.request(**self.request.dict(by_alias=True))
+            response.raise_for_status()
+            return response
+        except Exception as exc:
+            msg = f'Something went wrong with the http call {exc}'
+            LOG(msg)
+            raise DemistoException(msg) from exc
+
+    def prepare_next_run(self, offset: int):
+        self.request.params.offset += offset
+
+    def _set_cert_verification(self):
+        if not self.request.insecure:
+            skip_cert_verification()
+
+    def _set_proxy(self):
+        if self.request.proxy:
+            ensure_proxy_has_http_prefix()
+        else:
+            skip_proxy()
+
+
+class GetEvents:
+    def __init__(self, client: Client) -> None:
+        self.client = client
+
+    def call(self) -> list:
+        resp = self.client.call()
+        return resp.json().get('records', [])
+
+    def _iter_events(self):
+        events = self.call()
+
+        while events:
+            yield events
+
+            self.client.prepare_next_run(self.client.request.params.limit)
+            events = self.call()
+
+    def run(self, max_fetch: int = 100) -> List[dict]:
+        stored = []
+        last_run = demisto.getLastRun()
+
+        for logs in self._iter_events():
+            stored.extend(logs)
+
+            if len(stored) > max_fetch:
+                last_run['offset'] = last_run.get('offset', 0) + max_fetch
+                demisto.setLastRun(last_run)
+                return stored[:max_fetch]
+
+        last_run['offset'] = 0
+        demisto.setLastRun(last_run)
+        return stored
+
+    @staticmethod
+    def set_next_run(log: dict) -> dict:
+        """
+            Handles and saves the values required for next fetch.
+
+            There are 3 values:
+                * from: From which time to fetch
+                * next_run: Time of creation of the last event fetched
+                * offset: The size of the offset (how many events to skip)
+
+            Since the rest API returns the events in desc order (the last event returns first), We need to save the last
+            event time creation in some variable (next_run) for the next fetches, in addition we need to save in another
+            variable (offset) the number of how many events we already fetched to skip them in the next fetch to avoid
+            duplicates, in addition we need to save the time (from) from when to fetch if there is still some incident
+            to fetch with offset
+        """
+        last_run = demisto.getLastRun()
+
+        if not last_run.get('next_time'):
+            last_time = log.get('created', '').removesuffix('+0000')
+            next_time = last_time[:-1] + str(int(last_time[-1]) + 1)
+
+            if last_run.get('offset'):
+                last_run['next_time'] = next_time
+            else:
+                last_run['from'] = next_time
+
+        else:
+            if not last_run.get('offset'):
+                last_run['from'] = last_run.pop('next_time')
+
+        return last_run
+
+
+def main():
+    # Args is always stronger. Get last run even stronger
+    demisto_params = demisto.params() | demisto.args() | demisto.getLastRun()
+
+    demisto_params['params'] = ReqParams.parse_obj(demisto_params)
+
+    request = Request.parse_obj(demisto_params)
+    client = Client(request)
+    get_events = GetEvents(client)
+    command = demisto.command()
+
+    if command == 'test-module':
+        get_events.run(max_fetch=1)
+        demisto.results('ok')
+
+    elif command in ('fetch-events', 'jira-get-events'):
+        events = get_events.run(int(demisto_params.get('max_fetch', 100)))
+        send_events_to_xsiam(events, 'Jira', 'jira')
+
+        if events:
+            demisto.setLastRun(get_events.set_next_run(events[0]))
+            demisto.debug(f'Last run set to {demisto.getLastRun()}')
+            if command == 'jira-get-events':
+                command_results = CommandResults(
+                    readable_output=tableToMarkdown('Jira Audit Records', events, removeNull=True,
+                                                    headerTransform=pascalToSpace),
+                    raw_response=events,
+                )
+                return_results(command_results)
+
+
+if __name__ in ('__main__', '__builtin__', 'builtins'):
+    main()

Packs/Jira/Integrations/JiraEventCollector/JiraEventCollector.yml

@@ -0,0 +1,64 @@
+category: Analytics & SIEM
+commonfields:
+  id: Jira Event Collector
+  version: -1
+configuration:
+- defaultvalue: https://<your-domain>.atlassian.net/rest/api/3/auditing/record
+  display: Server URL
+  name: url
+  required: true
+  type: 0
+- display: User name
+  additionalinfo: The user name ([email protected]) and password
+  name: credentials
+  required: true
+  type: 9
+- defaultvalue: 3 days
+  display: First fetch time (<number> <time unit>, e.g., 12 hours, 1 day, 3 months). default is 3 days.
+  name: first_fetch
+  required: true
+  type: 0
+- defaultvalue: 100
+  display: The maximum number of events per fetch. Default is 100.
+  name: max_fetch
+  required: true
+  type: 0
+- display: Trust any certificate (not secure)
+  name: insecure
+  required: false
+  type: 8
+- display: Use system proxy settings
+  name: proxy
+  required: false
+  type: 8
+description: 'Jira logs event collector integration for XSIAM.'
+display: Jira Event Collector
+name: Jira Event Collector
+script:
+  commands:
+  - deprecated: false
+    description: 'Returns a list of audit records'
+    execution: false
+    name: jira-get-events
+    arguments:
+    - description: The maximum number of incidents per fetch. Default is 100.
+      isArray: true
+      name: max_fetch
+      required: false
+      secret: false
+    - description: First fetch time (<number> <time unit>, e.g., 12 hours, 1 day, 3 months). default is 3 days.
+      isArray: true
+      name: first_fetch
+      required: false
+      secret: false
+  runonce: false
+  script: '-'
+  isfetchevents: true
+  type: python
+  subtype: python3
+  dockerimage: demisto/fastapi:1.0.0.29406
+marketplaces:
+- marketplacev2
+fromversion: 6.5.0
+tests:
+- No tests (auto formatted)