Reporting (#179)

* Update type hints for monitoring alert creation

* Update index set and event stream titles and descriptions

* Refactor Office365 provision functions

* Update title and description formatting in build_index_set_config and build_event_stream_config

* Update index and event stream configuration

* Update index and event stream titles in SAP SIEM provision service

* Add data links to Grafana datasource

* Add dataLinks to GrafanaJsonData and create_grafana_datasource

* updated dependencies

* added reporting api/types

* added Report Creation page

* Add Grafana datasource URLs for data_vulnerability_cve and _id fields

* updated Report Creation page

* Add job functionality to scheduler

* Fix timestamp_utc fallback in create_alert_details function

* Add logging for agent details retrieval

* default to `timestamp` if timefield is None

* Update job time interval and add job metadata

* licensing test and github action docker build

* test reading auth secret

* updated Report Creation page

* license key checks

* Refactor code to enable browser testing using Playwright

* initial testing of reporting using playwright

* Add async support for agent synchronization. Remove the background task and run every 15 minutes

* Refactor sync_all_agents function to pass session parameter to sync_agents

* Add GenerateReportRequest to create_report function

* just use chromium browser

* Add GenerateReportResponse class and update create_report function signature

* migrage reporting to grafana connector

* updated Report Creation page

* license try catch

* added print media query

* updated connector form

* updated report page

* added playwrite test page

* added report template

* updated dependencies

* updated report page

* active response log analysis update

* sap siem analysis 10 minute window

* Update threshold and add time range parameter for SAP SIEM multiple logins analysis

* Update index names in sap_siem_multiple_logins.py

* same login failures multiple diff ips

* same user failed diff geo location

* Successful same user login from different locations

* brute force failed logins

* Refactor brute force failed logins route to handle multiple IPs

* brute_force_failed_logins_same_ip

* sap_siem_successful_login_after_failures

* change index name

* add sap siem to scheduler

* sap siem correction

for multiple IPs attempting to login with same user followed by a success. Added a correction to update the assets tab with the correct data

* pdf generation

* Update Jinja2 version to 3.1.2

* sort on the page number

* remove reportlab

* updated report page (added d&d)

* updated report panels drag & drop

* Update event stream configuration in graylog.py

* Update dashboard enum names

* Commented out Office365 related code

* Update Office365Dashboard enum value for SUMMARY

* Update index skipping logic in IndexConfigModel

* Update IndexConfigModel.is_valid_index method to allow non-"wazuh_" index names and exclude "deflector" index.

* Update Graylog schema for optional TLS and TCP keepalive

* updated report panels drag & drop

* updated report panels drag & drop

* modify wazuh agent config

* Refactor GenerateReportRequest schema in Grafana reporting module

* remodel report generation to fit new request schema

* use playwright for pdf generation

* Update custom_attributes field description in SingleCaseModel

* Add default value for custom_attributes in SingleCaseModel constructor

* Update custom_attributes field in SingleCaseModel

* test frontend auto build

* Add dependency installation step to Docker workflow

* Update dependencies in docker.yml and frontend/package-lock.json

* Fix template value for CUSTOMER_CODE in provision_custom_alert

* Fix incorrect client creation error message

* Add InfluxDB alerts fetching functionality to verification check. the ping and version will return true even if API token is not valid

* Update YouTube Tutorial link in README

* Fix cover-box typo and adjust screenshot styling

* Update Wazuh group configuration and replace placeholder with cluster name

* Fix placeholder replacement in wazuh_manager.py

* Add wazuh_worker_hostname field to default settings and schema

* Add wazuhWorkerHostname field to ProvisioningDefaultSettingsPayload and CustomerProvisioningDefaultSettings

* updated report panels drag & drop

* updated report template

* haproxy provisioning connector

* Add Grafana data link for O365 datasource

* Add feature enum and API endpoint to add a feature to a license

* Add feature check for reporting

* Update error message for disabled feature

* office365 fix

all office365 api keys now added to same office365 block

* delete wazuh_config.xml

* Add optional field for event definition configuration

* Refactor Config class in graylog/schema/events.py

* Add HAPROXY_PROVISIONING_URL to .env.example

* Add ExpressionItem class to handle complex expressions in Conditions

* Add SeriesItem model to Config class

* Add logging for Graylog alert definition provisioned response

* Add event_limit field to GraylogAlertProvisionConfig

* Add event limit to provision functions

* Commented out license check in create_report function

* Add panel width and height to RequestPanel model

* updated report api

* Update generate_panel_urls to include a theme parameter

* Add license-related API endpoints and models

* Update edr av malware ioc dashboard template

* Add ProvisionHaProxyRequest to customer_provisioning schema

* Add EDR_NETWORK_CONNECTIONS dashboard

* update grafana dashboard templates

* Add wazuh_agent_status field to WazuhAgent model and Agents table

* Update wazuh_agent_status default value

* updated report editor

* Add company name, timerange text, and logo to GenerateReportRequest

* Add theme field to GrafanaGenerateIframeLinksRequest and RequestPanel models

* Update report template and remove unnecessary files

* Update headless option in browser launch

* Add playwright dependencies installation step

* Update Grafana login handling and launch browser in headless mode

* Fix login issue in Grafana service

* precommit fixes

* RSA PUB KEY into build as env

* Add RSA_PUBLIC_KEY to build-args in docker.yml

* more precommit fixes

* Add PRODUCT_ID environment variable

* added report panel height settings

* Update branch name in Docker workflow

* Add HTTPException for feature not enabled

* precommit fixes

---------

Co-authored-by: Davide Di Modica <[email protected]>

.env.example

@@ -55,3 +55,5 @@ EVENT_SHIPPER_URL=graylog_host
 GELF_INPUT_PORT=gelf_port
 
 ALERT_CREATION_PROVISIONING_URL=http://1.1.1.1
+
+HAPROXY_PROVISIONING_URL=http://1.1.1.1

.flake8

@@ -1,6 +1,6 @@
 [flake8]
 #max-complexity = 18
-max-line-length = 170
+max-line-length = 180
 #select = B,C,E,F,W,T4,B9
 #ignore = E203, E266, E501, W503, F403, F401
 ignore = E402, W503, E231, W605, E266

.github/workflows/docker.yml

@@ -0,0 +1,58 @@
+name: Docker
+
+on:
+    push:
+        branches: [main]
+
+jobs:
+    build-backend:
+        runs-on: ubuntu-latest
+
+        steps:
+            - name: Check out code
+              uses: actions/checkout@v2
+
+            - name: Set up Docker Buildx
+              uses: docker/setup-buildx-action@v1
+
+            - name: Login to GitHub Container Registry
+              uses: docker/login-action@v1
+              with:
+                  registry: ghcr.io
+                  username: ${{ secrets.REPO_OWNER }}
+                  password: ${{ secrets.REPO_TOKEN }}
+
+            - name: Build and push Docker image
+              uses: docker/build-push-action@v2
+              with:
+                  context: ./backend
+                  push: true
+                  tags: ghcr.io/socfortress/copilot-backend:latest
+                  build-args: |
+                      CRYPTOLENS_AUTH=${{ secrets.CRYPTOLENS_AUTH }}
+                      RSA_PUBLIC_KEY=${{ secrets.RSA_PUBLIC_KEY }}
+                      PRODUCT_ID=${{ secrets.PRODUCT_ID }}
+
+    build-frontend:
+        runs-on: ubuntu-latest
+
+        steps:
+            - name: Check out code
+              uses: actions/checkout@v2
+
+            - name: Set up Docker Buildx
+              uses: docker/setup-buildx-action@v1
+
+            - name: Login to GitHub Container Registry
+              uses: docker/login-action@v1
+              with:
+                  registry: ghcr.io
+                  username: ${{ secrets.REPO_OWNER }}
+                  password: ${{ secrets.REPO_TOKEN }}
+
+            - name: Build and push Docker image
+              uses: docker/build-push-action@v2
+              with:
+                  context: ./frontend
+                  push: true
+                  tags: ghcr.io/socfortress/copilot-frontend:latest

.gitignore

@@ -55,3 +55,8 @@ package-lock.json
 firewall.spec
 
 frontend/cypress/screenshots/
+
+backend/report.pdf
+
+backend/report.html
+backend/app/integrations/office365/services/wazuh_config.xml

.vscode/extensions.json

@@ -7,12 +7,10 @@
 		"ecmel.vscode-html-css",
 		"george-alison.html-preview-vscodes",
 		"esbenp.prettier-vscode",
-		"octref.vetur",
 		"ms-vscode.remtoe-remote-wsl",
 		"dbaeumer.vscode-eslint",
 		"Vue.volar",
 		"Gruntfuggly.todo-tree",
-		"Vue.vscode-typescript-vue-plugin",
 		"usernamehw.errorlens",
 		"streetsidesoftware.code-spell-checker"
 	]

.vscode/settings.json

@@ -11,6 +11,7 @@
 		"forgotpassword",
 		"Healthcheck",
 		"healthchecks",
+		"Indicies",
 		"majesticons",
 		"mimecast",
 		"mynaui",
@@ -21,6 +22,7 @@
 		"Socfortress",
 		"sparkline",
 		"taze",
+		"timerange",
 		"uvicorn",
 		"venv",
 		"vuesjv",

README.md

@@ -102,7 +102,7 @@ Copilot shall be available on the host interface, port 443, protocol HTTPS - `ht
 By default, an `admin` account is created. The password is printed in stdout the very first time Copilot is started. It won't be printed anymore after that.
 `Admin user password` can be searched in the logs of the `copilot` docker to find the password. You will use the `plain` password to login to the web interface.
 
-🚀 **YouTube Tutorial:** [STILL NEED TO UPDATE](https://www.youtube.com/channel/UC4EUQtTxeC8wGrKRafI6pZg)
+🚀 **YouTube Tutorial:** [INSTALLING COPILOT](https://youtu.be/7dUHSMWWTuY?si=lRbn4tBHnyKmiTka)
 
 #### SSL
 

backend/Dockerfile

@@ -41,6 +41,13 @@ WORKDIR /opt/copilot/backend
 COPY requirements.txt ./
 RUN /opt/venv/bin/pip install --no-cache-dir -r requirements.txt
 
+# Install required browsers for playwright
+RUN playwright install-deps
+RUN playwright install
+
+# Install wkhtmltopdf
+RUN apt-get install -y wkhtmltopdf
+
 # Copy your application into the Docker image
 WORKDIR /opt/copilot/backend
 COPY . .
@@ -102,5 +109,14 @@ ENV GELF_INPUT_PORT=gelf_port
 
 ENV ALERT_CREATION_PROVISIONING_URL=http://1.1.1.1
 
+ARG CRYPTOLENS_AUTH
+ENV CRYPTOLENS_AUTH=$CRYPTOLENS_AUTH
+
+ARG RSA_PUBLIC_KEY
+ENV RSA_PUBLIC_KEY=$RSA_PUBLIC_KEY
+
+ARG PRODUCT_ID
+ENV PRODUCT_ID=$PRODUCT_ID
+
 # Run your application
 CMD ["sh", "-c", "ls -la && /opt/venv/bin/python copilot.py"]

backend/app/agents/routes/agents.py

@@ -1,5 +1,7 @@
+import asyncio
+
+# from fastapi import BackgroundTasks
 from fastapi import APIRouter
-from fastapi import BackgroundTasks
 from fastapi import Depends
 from fastapi import HTTPException
 from fastapi import Security
@@ -218,7 +220,7 @@ async def get_agent_by_hostname(
     ],
 )
 async def sync_all_agents(
-    backgroud_tasks: BackgroundTasks,
+    # backgroud_tasks: BackgroundTasks,
     session: AsyncSession = Depends(get_db),
 ) -> SyncedAgentsResponse:
     """
@@ -236,7 +238,9 @@ async def sync_all_agents(
 
     """
     logger.info("Syncing agents from Wazuh Manager")
-    backgroud_tasks.add_task(sync_agents, session)
+    # backgroud_tasks.add_task(sync_agents, session)
+    loop = asyncio.get_event_loop()
+    loop.create_task(sync_agents(session=session))
     return SyncedAgentsResponse(
         success=True,
         message="Agents synced started successfully",

backend/app/agents/wazuh/schema/agents.py

@@ -14,6 +14,7 @@ class WazuhAgent(BaseModel):
     agent_label: str = Field(..., alias="label")
     agent_last_seen: str = Field(..., alias="wazuh_last_seen")
     wazuh_agent_version: str = Field(..., alias="wazuh_agent_version")
+    wazuh_agent_status: Optional[str] = Field(None, alias="wazuh_agent_status")
 
     @property
     def agent_last_seen_as_datetime(self):

backend/app/agents/wazuh/services/agents.py

@@ -60,6 +60,7 @@ async def collect_wazuh_agents() -> WazuhAgentsList:
                     agent_label=agent_group,
                     agent_last_seen=last_keep_alive,
                     wazuh_agent_version=agent.get("version", "n/a"),
+                    wazuh_agent_status=agent.get("status", "n/a"),
                 )
                 wazuh_agents_list.append(wazuh_agent)
 

backend/app/connectors/dfir_iris/schema/cases.py

@@ -58,7 +58,14 @@ class SingleCaseModel(BaseModel):
     classification: Optional[Union[str, None]]
     classification_id: Optional[Union[int, None]]
     close_date: Optional[Union[str, None]]
-    custom_attributes: Optional[Union[str, None]]
+    # custom_attributes: Optional[Union[str, None]] = Field(
+    #     None,
+    #     description="The custom attributes of the case.",
+    # )
+    custom_attributes: Optional[Dict[str, Union[str, None]]] = Field(
+        None,
+        description="The custom attributes of the case.",
+    )
     customer_id: int
     customer_name: str
     initial_date: str
@@ -77,6 +84,11 @@ class SingleCaseModel(BaseModel):
     status_name: str
     customer_code: str
 
+    def __init__(self, **data):
+        if "custom_attributes" in data and not data["custom_attributes"]:
+            data["custom_attributes"] = {"default_key": "no custom attributes found"}  # Replace with your default entry
+        super().__init__(**data)
+
 
 class SingleCaseBody(BaseModel):
     case_id: int