Skip to content

Commit

Permalink
Merge pull request #187 from cisagov/v523_merge
Browse files Browse the repository at this point in the history 
* Version bumps
    * Arkime [v3.3.1](https://github.com/arkime/arkime/blob/54fb9cb1ee007aa51bda0712e466fca525e1db71/CHANGELOG#L25-L30)
    * Zeek [v4.2.0](https://github.com/zeek/zeek/releases/tag/v4.2.0)

* Improvements
    * Added script and better documentation for putting Malcolm in "read-only" mode
    * Improved `Files` dashboard

* Bug fixes
    * Fixed an issue where Logstash wasn't parsing the `ftime` from `files.log` correctly (a field added by the Spicy ZIP analyzer)
    * Fixed idaholab#73 (path for tcpdump changed) for Hedgehog Linux
    * Fixed idaholab#72 (better file directory/name parsing and normalization in Logstash)
  • Loading branch information
@mmguero
mmguero authored Jan 31, 2022
2 parents 2c62e87 + a660431 commit ba503df
Show file tree
Hide file tree
Showing 23 changed files with 982 additions and 644 deletions.
2 changes: 1 addition & 1 deletion Dockerfiles/arkime.Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ FROM debian:bullseye-slim AS build

ENV DEBIAN_FRONTEND noninteractive

ENV ARKIME_VERSION "3.3.0"
ENV ARKIME_VERSION "3.3.1"
ENV ARKIMEDIR "/opt/arkime"
ENV ARKIME_URL "https://github.com/arkime/arkime.git"
ENV ARKIME_LOCALELASTICSEARCH no
Expand Down
1 change: 1 addition & 0 deletions Dockerfiles/dashboards-helper.Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ ADD dashboards/malcolm_template.json /data/malcolm_template.json
ADD shared/bin/docker-uid-gid-setup.sh /usr/local/bin/
ADD shared/bin/opensearch_status.sh /data/
ADD shared/bin/opensearch_index_size_prune.py /data/
ADD shared/bin/opensearch_read_only.py /data/

RUN apk --no-cache add bash python3 py3-pip curl procps psmisc npm shadow jq && \
npm install -g http-server && \
Expand Down
2 changes: 1 addition & 1 deletion Dockerfiles/zeek.Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ ENV PUSER_PRIV_DROP true

# for download and install
ARG ZEEK_LTS=
ARG ZEEK_VERSION=4.1.1-0
ARG ZEEK_VERSION=4.2.0-0
ARG SPICY_VERSION=1.3.0

ENV ZEEK_LTS $ZEEK_LTS
Expand Down
80 changes: 45 additions & 35 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -172,22 +172,22 @@ You can then observe that the images have been retrieved by running `docker imag
```
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
malcolmnetsec/api 5.2.2 xxxxxxxxxxxx 2 days ago 155MB
malcolmnetsec/arkime 5.2.2 xxxxxxxxxxxx 2 days ago 811MB
malcolmnetsec/dashboards 5.2.2 xxxxxxxxxxxx 2 days ago 970MB
malcolmnetsec/dashboards-helper 5.2.2 xxxxxxxxxxxx 2 days ago 154MB
malcolmnetsec/filebeat-oss 5.2.2 xxxxxxxxxxxx 2 days ago 621MB
malcolmnetsec/file-monitor 5.2.2 xxxxxxxxxxxx 2 days ago 586MB
malcolmnetsec/file-upload 5.2.2 xxxxxxxxxxxx 2 days ago 259MB
malcolmnetsec/freq 5.2.2 xxxxxxxxxxxx 2 days ago 132MB
malcolmnetsec/htadmin 5.2.2 xxxxxxxxxxxx 2 days ago 242MB
malcolmnetsec/logstash-oss 5.2.2 xxxxxxxxxxxx 2 days ago 1.27GB
malcolmnetsec/name-map-ui 5.2.2 xxxxxxxxxxxx 2 days ago 142MB
malcolmnetsec/nginx-proxy 5.2.2 xxxxxxxxxxxx 2 days ago 117MB
malcolmnetsec/opensearch 5.2.2 xxxxxxxxxxxx 2 days ago 1.18GB
malcolmnetsec/pcap-capture 5.2.2 xxxxxxxxxxxx 2 days ago 122MB
malcolmnetsec/pcap-monitor 5.2.2 xxxxxxxxxxxx 2 days ago 214MB
malcolmnetsec/zeek 5.2.2 xxxxxxxxxxxx 2 days ago 938MB
malcolmnetsec/api 5.2.3 xxxxxxxxxxxx 2 days ago 155MB
malcolmnetsec/arkime 5.2.3 xxxxxxxxxxxx 2 days ago 811MB
malcolmnetsec/dashboards 5.2.3 xxxxxxxxxxxx 2 days ago 970MB
malcolmnetsec/dashboards-helper 5.2.3 xxxxxxxxxxxx 2 days ago 154MB
malcolmnetsec/filebeat-oss 5.2.3 xxxxxxxxxxxx 2 days ago 621MB
malcolmnetsec/file-monitor 5.2.3 xxxxxxxxxxxx 2 days ago 586MB
malcolmnetsec/file-upload 5.2.3 xxxxxxxxxxxx 2 days ago 259MB
malcolmnetsec/freq 5.2.3 xxxxxxxxxxxx 2 days ago 132MB
malcolmnetsec/htadmin 5.2.3 xxxxxxxxxxxx 2 days ago 242MB
malcolmnetsec/logstash-oss 5.2.3 xxxxxxxxxxxx 2 days ago 1.27GB
malcolmnetsec/name-map-ui 5.2.3 xxxxxxxxxxxx 2 days ago 142MB
malcolmnetsec/nginx-proxy 5.2.3 xxxxxxxxxxxx 2 days ago 117MB
malcolmnetsec/opensearch 5.2.3 xxxxxxxxxxxx 2 days ago 1.18GB
malcolmnetsec/pcap-capture 5.2.3 xxxxxxxxxxxx 2 days ago 122MB
malcolmnetsec/pcap-monitor 5.2.3 xxxxxxxxxxxx 2 days ago 214MB
malcolmnetsec/zeek 5.2.3 xxxxxxxxxxxx 2 days ago 938MB
```

#### Import from pre-packaged tarballs
Expand Down Expand Up @@ -856,13 +856,23 @@ Run `./scripts/wipe` to stop the Malcolm instance and wipe its OpenSearch databa

### <a name="ReadOnlyUI"></a>Temporary read-only interface

To temporarily set the Malcolm user interaces into a read-only configuration, run the following command from the Malcolm installation directory:
To temporarily set the Malcolm user interaces into a read-only configuration, run the following commands from the Malcolm installation directory.

First, to configure [Nginx] to disable access to the upload and other interfaces for changing Malcolm settings, and to deny HTTP methods other than `GET` and `POST`:

```
docker-compose exec nginx-proxy bash -c "cp /etc/nginx/nginx_readonly.conf /etc/nginx/nginx.conf && nginx -s reload"
```

This command must be re-run every time you restart Malcolm.
Second, to set the existing OpenSearch data store to read-only:

```
docker-compose exec dashboards-helper /data/opensearch_read_only.py -i _cluster
```

These commands must be re-run every time you restart Malcolm.

Note that after you run these commands you may see an increase of error messages in the Malcolm containers' output as various background processes will fail due to the read-only nature of the indices. Additionally, some features such as Arkime's [Hunt](#ArkimeHunt) and [building your own visualizations and dashboards](#BuildDashboard) in OpenSearch Dashboards will not function correctly in read-only mode.

## <a name="Upload"></a>Capture file and log archive upload

Expand Down Expand Up @@ -3270,7 +3280,7 @@ Building the ISO may take 30 minutes or more depending on your system. As the bu

```
…
Finished, created "/malcolm-build/malcolm-iso/malcolm-5.2.2.iso"
Finished, created "/malcolm-build/malcolm-iso/malcolm-5.2.3.iso"
…
```

Expand Down Expand Up @@ -3657,22 +3667,22 @@ Pulling zeek ... done

user@host:~/Malcolm$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
malcolmnetsec/api 5.2.2 xxxxxxxxxxxx 2 days ago 155MB
malcolmnetsec/arkime 5.2.2 xxxxxxxxxxxx 2 days ago 811MB
malcolmnetsec/dashboards 5.2.2 xxxxxxxxxxxx 2 days ago 970MB
malcolmnetsec/dashboards-helper 5.2.2 xxxxxxxxxxxx 2 days ago 154MB
malcolmnetsec/filebeat-oss 5.2.2 xxxxxxxxxxxx 2 days ago 621MB
malcolmnetsec/file-monitor 5.2.2 xxxxxxxxxxxx 2 days ago 586MB
malcolmnetsec/file-upload 5.2.2 xxxxxxxxxxxx 2 days ago 259MB
malcolmnetsec/freq 5.2.2 xxxxxxxxxxxx 2 days ago 132MB
malcolmnetsec/htadmin 5.2.2 xxxxxxxxxxxx 2 days ago 242MB
malcolmnetsec/logstash-oss 5.2.2 xxxxxxxxxxxx 2 days ago 1.27GB
malcolmnetsec/name-map-ui 5.2.2 xxxxxxxxxxxx 2 days ago 142MB
malcolmnetsec/nginx-proxy 5.2.2 xxxxxxxxxxxx 2 days ago 117MB
malcolmnetsec/opensearch 5.2.2 xxxxxxxxxxxx 2 days ago 1.18GB
malcolmnetsec/pcap-capture 5.2.2 xxxxxxxxxxxx 2 days ago 122MB
malcolmnetsec/pcap-monitor 5.2.2 xxxxxxxxxxxx 2 days ago 214MB
malcolmnetsec/zeek 5.2.2 xxxxxxxxxxxx 2 days ago 938MB
malcolmnetsec/api 5.2.3 xxxxxxxxxxxx 2 days ago 155MB
malcolmnetsec/arkime 5.2.3 xxxxxxxxxxxx 2 days ago 811MB
malcolmnetsec/dashboards 5.2.3 xxxxxxxxxxxx 2 days ago 970MB
malcolmnetsec/dashboards-helper 5.2.3 xxxxxxxxxxxx 2 days ago 154MB
malcolmnetsec/filebeat-oss 5.2.3 xxxxxxxxxxxx 2 days ago 621MB
malcolmnetsec/file-monitor 5.2.3 xxxxxxxxxxxx 2 days ago 586MB
malcolmnetsec/file-upload 5.2.3 xxxxxxxxxxxx 2 days ago 259MB
malcolmnetsec/freq 5.2.3 xxxxxxxxxxxx 2 days ago 132MB
malcolmnetsec/htadmin 5.2.3 xxxxxxxxxxxx 2 days ago 242MB
malcolmnetsec/logstash-oss 5.2.3 xxxxxxxxxxxx 2 days ago 1.27GB
malcolmnetsec/name-map-ui 5.2.3 xxxxxxxxxxxx 2 days ago 142MB
malcolmnetsec/nginx-proxy 5.2.3 xxxxxxxxxxxx 2 days ago 117MB
malcolmnetsec/opensearch 5.2.3 xxxxxxxxxxxx 2 days ago 1.18GB
malcolmnetsec/pcap-capture 5.2.3 xxxxxxxxxxxx 2 days ago 122MB
malcolmnetsec/pcap-monitor 5.2.3 xxxxxxxxxxxx 2 days ago 214MB
malcolmnetsec/zeek 5.2.3 xxxxxxxxxxxx 2 days ago 938MB
```

Finally, we can start Malcolm. When Malcolm starts it will stream informational and debug messages to the console. If you wish, you can safely close the console or use `Ctrl+C` to stop these messages; Malcolm will continue running in the background.
Expand Down
3 changes: 2 additions & 1 deletion arkime/etc/config.ini
View file
Original file line number Diff line number Diff line change
Expand Up @@ -422,6 +422,7 @@ zeek.files.depth=db:zeek.files.depth;group:zeek_files;kind:integer;friendly:Sour
zeek.files.analyzers=db:zeek.files.analyzers;group:zeek_files;kind:termfield;friendly:Analyzer;help:Analyzer
zeek.files.mime_type=db:zeek.files.mime_type;group:zeek_files;kind:termfield;friendly:File Magic;help:File Magic
zeek.files.filename=db:zeek.files.filename;group:zeek_files;kind:termfield;friendly:Filename;help:Filename
zeek.files.ftime=db:zeek.files.ftime;group:zeek_files;kind:termfield;friendly:File Timestamp;help:File Timestamp
zeek.files.duration=db:zeek.files.duration;group:zeek_files;kind:termfield;friendly:Analysis Duration;help:Analysis Duration
zeek.files.local_orig=db:zeek.files.local_orig;group:zeek_files;kind:termfield;friendly:Local Originator;help:Local Originator
zeek.files.is_orig=db:zeek.files.is_orig;group:zeek_files;kind:termfield;friendly:Originator is Transmitter;help:Originator is Transmitter
Expand Down Expand Up @@ -1377,7 +1378,7 @@ zeek_ecat_foe_info=require:zeek.ecat_foe_info;title:Zeek ecat_foe_info.log;field
zeek_ecat_soe_info=require:zeek.ecat_soe_info;title:Zeek ecat_soe_info.log;fields:zeek.ecat_soe_info.opcode,zeek.ecat_soe_info.incomplete,zeek.ecat_soe_info.error,zeek.ecat_soe_info.drive_num,zeek.ecat_soe_info.element,zeek.ecat_soe_info.index
zeek_ecat_arp_info=require:zeek.ecat_arp_info;title:Zeek ecat_arp_info.log;fields:zeek.ecat_arp_info.arp_type,zeek.ecat_arp_info.orig_proto_addr,zeek.ecat_arp_info.orig_hw_addr,zeek.ecat_arp_info.resp_proto_addr,zeek.ecat_arp_info.resp_hw_addr
zeek_enip=require:zeek.enip;title:Zeek enip.log;fields:zeek.enip.enip_command,zeek.enip.enip_command_code,zeek.enip.length,zeek.enip.session_handle,zeek.enip.enip_status,zeek.enip.sender_context,zeek.enip.options
zeek_files=require:zeek.files;title:Zeek files.log;fields:zeek.files.tx_hosts,zeek.files.rx_hosts,zeek.files.conn_uids,zeek.files.source,zeek.files.depth,zeek.files.analyzers,zeek.files.mime_type,zeek.files.filename,zeek.files.duration,zeek.files.local_orig,zeek.files.is_orig,zeek.files.seen_bytes,zeek.files.total_bytes,zeek.files.missing_bytes,zeek.files.overflow_bytes,zeek.files.timedout,zeek.files.parent_fuid,zeek.files.md5,zeek.files.sha1,zeek.files.sha256,zeek.files.extracted,zeek.files.extracted_cutoff,zeek.files.extracted_size
zeek_files=require:zeek.files;title:Zeek files.log;fields:zeek.files.tx_hosts,zeek.files.rx_hosts,zeek.files.conn_uids,zeek.files.source,zeek.files.depth,zeek.files.analyzers,zeek.files.mime_type,zeek.files.filename,zeek.files.ftime,zeek.files.duration,zeek.files.local_orig,zeek.files.is_orig,zeek.files.seen_bytes,zeek.files.total_bytes,zeek.files.missing_bytes,zeek.files.overflow_bytes,zeek.files.timedout,zeek.files.parent_fuid,zeek.files.md5,zeek.files.sha1,zeek.files.sha256,zeek.files.extracted,zeek.files.extracted_cutoff,zeek.files.extracted_size
zeek_ftp=require:zeek.ftp;title:Zeek ftp.log;fields:zeek.ftp.command,zeek.ftp.arg,zeek.ftp.mime_type,zeek.ftp.file_size,zeek.ftp.reply_code,zeek.ftp.reply_msg,zeek.ftp.data_channel_passive,zeek.ftp.data_channel_orig_h,zeek.ftp.data_channel_resp_h,zeek.ftp.data_channel_resp_p
zeek_gquic=require:zeek.gquic;title:Zeek gquic.log;fields:zeek.gquic.version,zeek.gquic.server_name,zeek.gquic.user_agent,zeek.gquic.tag_count,zeek.gquic.cyu,zeek.gquic.cyutags
zeek_http=require:zeek.http;title:Zeek http.log;fields:zeek.http.trans_depth,zeek.http.method,zeek.http.host,zeek.http.uri,zeek.http.origin,zeek.http.post_password_plain,zeek.http.post_username,zeek.http.referrer,zeek.http.version,zeek.http.user_agent,zeek.http.request_body_len,zeek.http.response_body_len,zeek.http.status_code,zeek.http.status_msg,zeek.http.info_code,zeek.http.info_msg,zeek.http.tags,zeek.http.proxied,zeek.http.orig_fuids,zeek.http.orig_filenames,zeek.http.orig_mime_types,zeek.http.resp_fuids,zeek.http.resp_filenames,zeek.http.resp_mime_types
Expand Down
76 changes: 0 additions & 76 deletions arkime/patch/viewer_330_large_or_xor_packet_fix_f13e9366.patch
View file

This file was deleted.

1 change: 1 addition & 0 deletions arkime/wise/source.zeeklogs.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -406,6 +406,7 @@ class MalcolmSource extends WISESource {
"zeek.files.extracted_cutoff",
"zeek.files.extracted_size",
"zeek.files.filename",
"zeek.files.ftime",
"zeek.files.is_orig",
"zeek.files.local_orig",
"zeek.files.md5",
Expand Down
Loading

0 comments on commit ba503df

Please sign in to comment.