update docs (#1105)

* update docs

---------

Co-authored-by: Davide Arcuri <[email protected]>

README.md

@@ -43,7 +43,6 @@ Orochi is an open source framework for collaborative forensic memory dump analys
 
 For people who prefer to install and try first and then read the guide:
 ```
-sudo sysctl -w vm.max_map_count=262144
 git clone https://github.com/LDO-CERT/orochi.git
 cd orochi
 sudo docker-compose pull
@@ -58,7 +57,7 @@ Browse http://127.0.0.1:8000 and access with admin//admin
 - distributes loads among nodes using [Dask](https://github.com/dask/dask)
 - uses [Django](https://github.com/django/django) as frontend
 - uses [Postgresql](https://github.com/postgres/postgres) to save users, analysis metadata such status and errors.
-- uses [MailHog](https://github.com/mailhog/MailHog) to manage the users registration emails
+- uses [Mailpit](https://github.com/axllent/mailpit) to manage the users registration emails
 - uses [Redis](https://github.com/redis/redis) for cache and websocket for notifications
 - all framework is provided as [docker-compose](https://github.com/docker/) images
 
@@ -124,14 +123,16 @@ Using Docker-compose you can start multiple dockers and link them together.
  ```
 
   ````
-CONTAINER ID   IMAGE                                     COMMAND                  CREATED       STATUS       PORTS                                                           NAMES
-40b14376265d   ghcr.io/ldo-cert/orochi_django:latest     "/entrypoint /start"     6 hours ago   Up 6 hours   0.0.0.0:8000->8000/tcp, :::8000->8000/tcp                       orochi_django
-016533025d9b   redis:6.2.5                               "docker-entrypoint.s…"   6 hours ago   Up 6 hours   0.0.0.0:6379->6379/tcp, :::6379->6379/tcp                       orochi_redis
-2cada5c22475   mailhog/mailhog:v1.0.1                    "MailHog"                6 hours ago   Up 6 hours   1025/tcp, 0.0.0.0:8025->8025/tcp, :::8025->8025/tcp             orochi_mailhog
-3e56e4f5b58e   ghcr.io/ldo-cert/orochi_postgres:latest   "docker-entrypoint.s…"   6 hours ago   Up 6 hours   0.0.0.0:5432->5432/tcp, :::5432->5432/tcp                       orochi_postgres
-0bb7f1a293ef   daskdev/dask:2021.10.0-py3.9              "tini -g -- /usr/bin…"   6 hours ago   Up 6 hours   0.0.0.0:8786-8787->8786-8787/tcp, :::8786-8787->8786-8787/tcp   orochi_scheduler
-10049fb631a4   ghcr.io/ldo-cert/orochi_worker:latest     "tini -g -- /usr/bin…"   6 hours ago   Up 6 hours                                                                   orochi_worker_2
-8e144a0c8972   ghcr.io/ldo-cert/orochi_worker:latest     "tini -g -- /usr/bin…"   6 hours ago   Up 6 hours                                                                   orochi_worker_1
+CONTAINER ID   IMAGE                                COMMAND                  CREATED        STATUS                 PORTS                                                                                            NAMES
+fdc1fa46c0d8   ghcr.io/ldo-cert/orochi_nginx:new    "/docker-entrypoint.…"   21 hours ago   Up 4 hours             0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp                         orochi_nginx
+db5b7f50ee5b   ghcr.io/ldo-cert/orochi_worker:new   "tini -g -- /usr/bin…"   21 hours ago   Up 4 hours                                                                                                              orochi-worker-1
+5f334d521d04   ghcr.io/ldo-cert/orochi_worker:new   "tini -g -- /usr/bin…"   21 hours ago   Up 4 hours                                                                                                              orochi-worker-2
+3768f5fa73d3   ghcr.io/ldo-cert/orochi_django:new   "/entrypoint /start"     21 hours ago   Up 4 hours             8000/tcp                                                                                         orochi_django_wsgi
+a3f79c5281cc   ghcr.io/ldo-cert/orochi_django:new   "/entrypoint daphne …"   21 hours ago   Up 4 hours             9000/tcp                                                                                         orochi_django_asgi
+6bb5d6107029   ghcr.io/ldo-cert/orochi_worker:new   "tini -g -- /usr/bin…"   21 hours ago   Up 4 hours             0.0.0.0:8786-8787->8786-8787/tcp, :::8786-8787->8786-8787/tcp                                    orochi_scheduler
+636c41f3fe9b   postgres:16.3                        "docker-entrypoint.s…"   22 hours ago   Up 4 hours             0.0.0.0:5432->5432/tcp, :::5432->5432/tcp                                                        orochi_postgres
+6d8d337667ad   redis:7.4.0                          "docker-entrypoint.s…"   22 hours ago   Up 4 hours             0.0.0.0:6379->6379/tcp, :::6379->6379/tcp                                                        orochi_redis
+596be665ef37   axllent/mailpit:latest               "/mailpit"               22 hours ago   Up 4 hours (healthy)   0.0.0.0:1025->1025/tcp, :::1025->1025/tcp, 0.0.0.0:8025->8025/tcp, :::8025->8025/tcp, 1110/tcp   orochi_mailpit
 
    ```
   ````
@@ -154,10 +155,10 @@ CONTAINER ID   IMAGE                                     COMMAND
 (\*) It is also possible to run plugins_sync and symbols_sync directly from the admin page in case new plugins or new symbols are available.
 
 - To create a **normal user account**, just go to Sign Up (http://127.0.0.1:8000) and fill out the form. Once you submit it, you'll see a "Verify Your E-mail Address" page. Go to your console to see a simulated email verification message. Copy the link into your browser. Now the user's email should be verified and ready to go.
-  In development, it is often nice to be able to see emails that are being sent from your application. For that reason local SMTP server [Mailhog](https://github.com/mailhog/MailHog) with a web interface is available as docker container.
-  Container mailhog will start automatically when you will run all docker containers.
+  In development, it is often nice to be able to see emails that are being sent from your application. For that reason local SMTP server [Mailpit](https://github.com/axllent/mailpit) with a web interface is available as docker container.
+  Container mailpit will start automatically when you will run all docker containers.
   Please check `cookiecutter-django Docker documentation` for more details how to start all containers.
-  With MailHog running, to view messages that are sent by your application, open your browser and go to `http://127.0.0.1:8025`
+  With Mailpit running, to view messages that are sent by your application, open your browser and go to `http://127.0.0.1:8025`
 
 - Other details in [cookiecutter-django Docker documentation](http://cookiecutter-django.readthedocs.io/en/latest/deployment-with-docker.html)
 
@@ -175,7 +176,7 @@ Applications links:
 
 - Orochi homepage: http://127.0.0.1:8000
 - Orochi admin: http://127.0.0.1:8000/admin
-- Mailhog: http://127.0.0.1:8025
+- Mailpit: http://127.0.0.1:8025
 - Dask: http://127.0.0.1:8787
 
 ### User guide

docs/API-Guide.md

@@ -4,17 +4,13 @@
 - [Concepts](#concepts)
 - [Login](#login)
 - [Swagger](#swagger)
-- [ReDoc](#redoc)
-- [Browsable API](#browsable-api)
+- [Demo notebook](#demo-notebook)
 
 ## Concepts
 
 The following libraries are used to implement API in Orochi: 
 
-- Django REST framework: a powerful and flexible toolkit for building Web APIs.
-- DRF - Yet another Swagger generator 2: Automated generation of real Swagger/OpenAPI 2.0 schemas from Django REST Framework code.
-- drf-nested-routers: this package provides routers and fields to create nested resources in the Django Rest Framework. Nested resources are needed for full REST URL structure, if one resource lives inside another.
-
+- [Django-ninja](https://django-ninja.dev/): a powerful and flexible toolkit for building Web APIs.
 
 ## Login
 
@@ -26,24 +22,10 @@ The GUI is available on port 8000, so if you are running dockers locally: http:/
 ## Swagger
 
 Swagger is a set of open-source tools built around the OpenAPI Specification that can help you design, build, document and consume REST APIs.
-Swagger is available at http://127.0.0.1:8000/swagger/
+Swagger is available at https://localhost/api/docs
 
 ![api-swagger](images/043_api_swagger.png)
 
-## ReDoc
-
-ReDoc is an open-source tool that generates API documentation from OpenAPI specifications. It’s one of the most powerful free docs tools in the industry, producing clean, customizable documentation with an attractive three-panel design.
-ReDoc is available at http://127.0.0.1:8000/redoc/
-
-![api-redoc](images/044_api_redoc.png)
-
-## Browsable API
-
-API may stand for Application Programming Interface, but humans have to be able to read the APIs, too; someone has to do the programming. Django REST Framework supports generating human-friendly HTML output for each resource when the HTML format is requested. These pages allow for easy browsing of resources, as well as forms for submitting data to the resources using POST, PUT, and DELETE.
-Runserver and check product list and product detail endpoints. You should see similar pages.
-Browsable API are available at http://127.0.0.1:8000/api/
-
-
-![api-browsable](images/045_api_browsable.png)
-
+## Demo notebook
 
+A python example on API usage is available [here](../examples/local_api.ipynb)

docs/Admin-Guide.md

@@ -19,7 +19,7 @@
 - [YARA](#yara)
     - [Update Rules](#update-rules)
     - [Generate Default Rule](#generate-default-rule)
-    - [Manage Rules](#manage-rules)    
+    - [Manage Rules](#manage-rules)
     - [Manage Ruleset](#manage-ruleset)
 
 
@@ -49,9 +49,7 @@ The admin page is available under /admin , so if you are running dockers locally
 
 In this section Admin can check the status of users registration, if emails are verified or not. If needed the Admin can validate manually the email of the users.
 
-![admin-email](images/025_admin_email.png)
-
-![admin-email-edit](images/026_admin_email_edit.png)
+![admin-email](animations/admin_email.gif)
 
 
 ## Users

docs/Users-Guide.md

@@ -14,7 +14,6 @@
 - [Deleting Dumps](#deleting-dumps)
 - [YARA](#yara)
 - [HEX Viewer](#hex-viewer)
-- [OROCHI Stats](#orochi-stats)
 
 ## Concepts
 
@@ -30,13 +29,12 @@ Use sign-up page to register your user and confirm email address through MailHog
 ![sign-in](images/001_sign_in.png)
 ![sign-up](images/002_sign_up.png)
 ![mailog](images/003_mailhog_mail.png)
-![confirm-email](images/004_confirm_email.png)
+
 ## Plugins
 
 Each user can choose which plugins will be executed automatically after uploading a memory dump. If none is selected, user will be able to choose and run any plugin after upload memory dump.
 
-![plugin-list](images/006_user_plugin_list.png)
-![plugin-selection](images/007_user_plugin_filter&selection.png)
+![plugin-selection](animations/plugins.gif)
 
 ## Upload Dump
 
@@ -49,24 +47,16 @@ Password protected zip files are supported as well.
 
 VmWare Snapshots are also supported, when needed both vmem and vmss, just upload zip file containing both.
 
-![home-page](images/005_home_page.png)
-![upload-dump](images/008_upload_dump.png)
-
-
-Large memory dumps can be placed manually in /media/uploads folder and then loaded in he system through API
-
-![upload-dump-swagger](images/060_upload_local_dump_swagger.png)
-
+![home-page](animations/upload.gif)
 
-or also via management command
+Large memory dumps can be placed manually in /media/uploads folder and then selected via "Local folder" dropdown in the upload dialog or via management command
 
 ![upload-dump-swagger](images/061_upload_local_dump_manage.png)
 
 
-
 When upload is completed it is possible to view the details of dump by pressing "i" button close to the memory dump name.
-Useful data shown of the uploaded file are md5, sha256, size, filepath where it is stored and the index name in ElasticSearch.
-![dump-info](images/068_dump_info.png)
+Useful data shown of the uploaded file are md5, sha256, size, filepath where it is stored and the index name.
+![dump-info](animations/dump_info.gif)
 
 
 ## Executing Plugins
@@ -75,18 +65,10 @@ A list of plugins will be shown after selecting the dump, then it is possible to
 - see the result of plugin if it was set to be run automatically
 - run the plugin if it was not set to be run automatically
 - re-run the plugin if need to pass some additional parameter (like dump flag, or string file)
+- Websocket is used to send notifications about plugins execution status
 
-
-
-![plugin-cmdline](images/010_plugin_result_cmdline.png)
-![plugin-pstree](images/011_plugin_result_pstree.png)
-![plugin-pslist-dump](images/012_plugin_pslist_dump.png)
-![plugin-rerun](images/013_rerun_plugin.png)
-![plugin-rerun-result](images/014_rerun_plugin_result.png)
-
-A websocket is used to send notifications about plugins execution status
-
-![plugin-notification](images/015_plugin_notifications.png)
+![plugin-main](animations/main.gif)
+![plugin-pstree](animations/pstree.gif)
 
 If the plugin ends with an error, a log button will be shown with the relative error.
 
@@ -103,16 +85,15 @@ By default docker-compose will create for you 2 different worker on the same mac
 It is possible to perform a full text search through plugin result  thanks to DataTable.
 The search works also through multiple dumps, if selected.
 
-![result-search](images/017_plugin_result_search.png)
+![result-search](animations/search.gif)
 
 ## Comparing plugin results
 
 When 2 dumps are selected it is possible to chose a plugin (that was run on both dumps) and visualize simultaneously the results.
 In this case the color chosen during upload of dump is useful to identify different dumps.
 There is also a function that performs json diff.
 
-![result-compare-tab](images/018_results_tab_compare.png)
-![result-compare-json](images/019_results_json_compare.png)
+![result-compare](animations/diff.gif)
 
 ## Sharing Dumps
 
@@ -126,19 +107,14 @@ User that uploads the dump is the owner of dump; when dump is shared, other user
 
 It is possible to bookmark the result of current view, so when find something interesting you can filter out and then add to bookmarks
 
-![bookmarks](images/051_bookmarks.png)
-
 After press the bookmark button it is possible to set a name for the bookmark, choose an icon picked from [MTG](https://magic.wizards.com/) sets and star it if want this bookmark appear in starred menu.
 
-![bookmarks](images/052_bookmarks_save.png)
-
 Then is possible go to bookmarks from admin button and if bookmark is starred it is shown directly in the menu.
 
-![bookmarks](images/053_bookmarks_starred.png)
 
 Otherwise opening bookmark the menu will show all bookmarks. The bookmark can be a query against multiple dumps.
 
-![bookmarks](images/054_bookmarks_list.png)
+![bookmarks](animations/bookmarks.gif)
 
 ## Export to MISP
 
@@ -156,8 +132,7 @@ This is the result in MISP: files and AV signatures will be created as objects c
 ## Deleting Dumps
 
 Deleting dump function will delete dump and all results of plugins.
-![dump-delete](images/021_dump_delete.png)
-
+![dump-delete](animations/delete.gif)
 
 
 ## YARA
@@ -171,7 +146,7 @@ Through this page is possible to view all rules previously imported and enabled
 
 At this page user can search for rules and build the custom yara compiled file to be passed to Volatility yara plugin.
 
-Thanks to ElasticSearch the fulltext search inside yara rules files is supported.
+Thanks to postgres the fulltext search inside yara rules files is supported.
 
 The user created the yara file can choose if keep it private or make public and available also for other users.
 Is it possible to have different yara compiled files, the only one set as default will be used by Volatility yara plugin.
@@ -186,20 +161,9 @@ At this point it will be possible to use the Volatility yara plugin and view the
 OROCHI support remote HEX View of dumps.
 It is possible to browse the memory dump by pressing "*"  button close to the memory dump.
 
-![hex-view-button](images/069_hex_view_button.png)
 
 
 At this point in the page will appear the HEX Viewer that shows the memory address, the hex values and the ascii values.
 It is possible browse manually the entire dump, go to a specific offset anb also search for a specific text.
 
-
-![hex-viewer](images/070_hex_viewer.png)
-
-
-## OROCHI Stats
-
-Thanks to Kibana it is possible create some dashboard to show stats about dumps,plugins, etc.
-
-![kibana-timeline](images/071_kibana-timeline.png)
-![kibana-os-images](images/072_kibana-os-images.png)
-![kibana-plugins](images/073_kibana-plugins.png)
+![hex-view](animations/hex.gif)