Skip to content

Commit

Permalink
Merge branch 'maintenance'
Browse files Browse the repository at this point in the history
  • Loading branch information
Sebastian Wagner committed Apr 20, 2018
2 parents 429e8fe + 0148860 commit 70f8098
Show file tree
Hide file tree
Showing 63 changed files with 1,287 additions and 864 deletions.
52 changes: 48 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,50 @@ CHANGELOG
==========


1.0.4 Bugfix release (2018-04-20)
---------------------------------
- make code style compatible to pycodestyle 2.4.0
- fixed permissions of some files (they were executable but shouldn't be)

### Core
- lib/harmonization:
* FQDN validation now handles None correctly (raised an Exception).
* Fixed several sanitize() methods, the generic sanitation method were called by is_valid, not the sanitize methods (#1219).

### Harmonization

### Bots
* Use the new pypi website at https://pypi.org/ everywhere.

#### Parsers
- Shadowserver parser:
* The fields `url` and `http_url` now handle HTTP URL paths and HTTP requests for all feeds (#1204).
* The conversion function `validate_fqdn` now handles empty strings correctly.
* Feed 'drone (hadoop)':
* Correct validation of field `cc_dns`, will now only be added as `destination.fqdn` if correct FQDN, otherwise ignored. Previously this field could be saved in extra containing an IP address.
* Adding more mappings for added columns.
* A lot of newly added fields and fixed conversions.
* Add newly added columns of `Ssl-Scan` feed to parser
- Spamhaus CERT parser:
* fix parsing and classification for bot names 'openrelay', 'iotrdp', 'sshauth', 'telnetauth', 'iotcmd', 'iotuser', 'wpscanner', 'w_wplogin', 'iotscan'
see the NEWS file - Postgresql section - for all changes.
- CleanMX phishing parser: handle FQDNs in IP column (#1162).

#### Experts
- `bots.experts.ripencc_abuse_contact`: Add existing parameter `mode` to BOTS file.

### Tools
- intelmqctl check: Fixed and extended message for 'run_mode' check.
- `intelmqctl start` botnet. When using `--type json`, no non-json information about wrong bots are output because that would confuse eg. intelmq-manager

### Tests
- lib/bot: No dumps will be written during tests (#934).
- lib/test: Expand regular expression on python version to match pre-releases (debian testing).

### Packaging
* Static data is now included in source tarballs, development files are excluded


1.0.3 Bugfix release (2018-02-05)
---------------------------------
### Contrib
Expand Down Expand Up @@ -319,7 +363,7 @@ Changes between 0.9 and 1.0.0.dev6
-`classification.taxonomy` is now lower case only

### Known issues
- Harmonization: hashes are not normalized and classified, see also issue #394 and pull #634
- Harmonization: hashes are not normalized and classified, see also issue #394 and pull #634

### Contrib
- ansible and vagrant scripts added
Expand Down Expand Up @@ -358,6 +402,6 @@ Changes between 0.9 and 1.0.0.dev6
2015/06/03 (aaron)
------------------

* fixed the license to AGPL in setup.py
* moved back the documentation from the wiki repo to `docs/`. See #205.
* added python-zmq as a setup requirement in UserGuide . See #206
* fixed the license to AGPL in setup.py
* moved back the documentation from the wiki repo to `docs/`. See #205.
* added python-zmq as a setup requirement in UserGuide . See #206
10 changes: 8 additions & 2 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
exclude .gitignore
exclude .travis.yml
graft contrib
graft docs
graft intelmq/bots
graft intelmq/etc
graft intelmq/tests
include COPYRIGHT
include LICENSE
include CHANGELOG.md
recursive-exclude intelmq/bin intelmq_gen_feeds_docs.py intelmq_gen_harm_docs.py rewrite_config_files.py
exclude .*
global-exclude *~
global-exclude *.py[co]
global-exclude __pycache__
57 changes: 48 additions & 9 deletions NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,45 @@ NEWS

See the changelog for a full list of changes.

1.0.4 Bugfix release (2018-04-20)
---------------------------------

### Postgres databases
Use the following statement carefully to upgrade your database.
Adapt your feedname in the query to the one used in your setup.
```SQL
UPDATE events
SET "classification.taxonomy" = 'intrusion attempts', "classification.type" = 'brute-force', "classification.identifier" = 'rdp', "protocol.application" = 'rdp', "malware.name" = NULL
WHERE "malware.name" = 'iotrdp' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'vulnerable', "classification.type" = 'vulnerable service', "classification.identifier" = 'openrelay', "protocol.application" = 'smtp', "malware.name" = NULL
WHERE "malware.name" = 'openrelay' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "protocol.application" = 'portmapper'
WHERE "classification.identifier" = 'openportmapper' AND "feed.name" = 'Open-Portmapper';
UPDATE events
SET "protocol.application" = 'netbios-nameservice'
WHERE "classification.identifier" = 'opennetbios' AND "feed.name" = 'Open-NetBIOS-Nameservice';
UPDATE events
SET "protocol.application" = 'ipsec'
WHERE "classification.identifier" = 'openike' AND "feed.name" = 'Vulnerable-ISAKMP';
UPDATE events
SET "classification.taxonomy" = 'intrusion attempts', "classification.type" = 'brute-force', "classification.identifier" = 'ssh', "malware.name" = NULL, "protocol.application" = 'ssh'
WHERE "malware.name" = 'sshauth' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'intrusion attempts', "classification.type" = 'brute-force', "classification.identifier" = 'telnet', "malware.name" = NULL, "protocol.application" = 'ssh'
WHERE ("malware.name" = 'telnetauth' OR "malware.name" = 'iotcmd' OR "malware.name" = 'iotuser') AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'information gathering', "classification.type" = 'scanner', "classification.identifier" = 'wordpress-vulnerabilities', "malware.name" = NULL, "event_description.text" = 'scanning for wordpress vulnerabilities', "protocol.application" = 'http'
WHERE "malware.name" = 'wpscanner' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'information gathering', "classification.type" = 'scanner', "classification.identifier" = 'wordpress-login', "malware.name" = NULL, "event_description.text" = 'scanning for wordpress login pages', "protocol.application" = 'http'
WHERE "malware.name" = 'w_wplogin' AND "feed.name" = 'Spamhaus CERT';
UPDATE events
SET "classification.taxonomy" = 'intrusion attempts', "classification.type" = 'scanner', "classification.identifier" = 'scanner-generic', "malware.name" = NULL, "event_description.text" = 'infected IoT device scanning for other vulnerable IoT devices'
WHERE "malware.name" = 'iotscan' AND "feed.name" = 'Spamhaus CERT';
```

1.0.3 Bugfix release (2018-02-05)
---------------------------------
### Configuration
Expand All @@ -12,24 +51,24 @@ See the changelog for a full list of changes.
| n6 classification | Previous classification | | | Current classification | | | Notes |
|-|-|-|-|-|-|-|-|
| | taxonomy | type | identifier | taxonomy | type | identifier |
| dns-query | Other | other | ignore me | Other | other | dns-query |
| proxy | Vulnerable | proxy | open proxy | Other | proxy | openproxy |
| dns-query | other | other | ignore me | other | other | dns-query |
| proxy | vulnerable | proxy | open proxy | other | proxy | openproxy |
| sandbox-url | ignore | ignore | ignore me | malicious code | malware | sandboxurl | As this previous taxonomy did not exist, these events have been rejected |
| other | Vulnerable | unknow | unknown | Other | other | other |
| other | vulnerable | unknow | unknown | other | other | other |

### Postgres databases
Use the following statement carefully to upgrade your database.
Adapt your feedname in the query to the one used in your setup.
```SQL
UPDATE events
SET "classification.identifier" = "dns-query"
WHERE "feed.name" = 'n6' AND "classification.taxonomy" = "Other" AND "classification.type" = "other" AND "classification.identifier" = "ignore me";
SET "classification.identifier" = 'dns-query'
WHERE "feed.name" = 'n6' AND "classification.taxonomy" = 'other' AND "classification.type" = 'other' AND "classification.identifier" = 'ignore me';
UPDATE events
SET "classification.taxonomy" = "malicious code" AND "classification.type" = "malware" AND "classification.identifier" = "sandboxurl"
WHERE "feed.name" = 'n6' AND "classification.taxonomy" = "Vulnerable" AND "classification.type" = "ignore" AND "classification.identifier" = "ignore me";
SET "classification.taxonomy" = 'malicious code' AND "classification.type" = 'malware' AND "classification.identifier" = 'sandboxurl'
WHERE "feed.name" = 'n6' AND "classification.taxonomy" = 'vulnerable' AND "classification.type" = 'ignore' AND "classification.identifier" = 'ignore me';
UPDATE events
SET "classification.taxonomy" = "Other" AND "classification.type" = "other" AND "classification.identifier" = "other"
WHERE "feed.name" = 'n6' AND "classification.taxonomy" = "Vulnerable" AND "classification.type" = "unknow" AND "classification.identifier" = "unknow";
SET "classification.taxonomy" = 'other' AND "classification.type" = 'other' AND "classification.identifier" = 'other'
WHERE "feed.name" = 'n6' AND "classification.taxonomy" = 'vulnerable' AND "classification.type" = 'unknow' AND "classification.identifier" = 'unknow';
```

1.0.2 Bugfix release
Expand Down
6 changes: 6 additions & 0 deletions debian/changelog
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
intelmq (1.0.4-1) stable; urgency=low

* Update to version 1.0.4

-- Wagner Sebastian <[email protected]> Fri, 20 Apr 2018 15:25:55 +0200

intelmq (1.0.3-1) stable; urgency=low

* Update to version 1.0.3
Expand Down
2 changes: 1 addition & 1 deletion docs/Bots.md
Original file line number Diff line number Diff line change
Expand Up @@ -711,11 +711,11 @@ Sources:

#### Configuration Parameters:

* `mode`: either `append` (default) or `replace`
* `query_ripe_db_asn`: Query for IPs at `http://rest.db.ripe.net/abuse-contact/%s.json`, default `true`
* `query_ripe_db_ip`: Query for ASNs at `http://rest.db.ripe.net/abuse-contact/as%s.json`, default `true`
* `query_ripe_stat_asn`: Query for ASNs at `https://stat.ripe.net/data/abuse-contact-finder/data.json?resource=%s`, default `true`
* `query_ripe_stat_ip`: Query for IPs at `https://stat.ripe.net/data/abuse-contact-finder/data.json?resource=%s`, default `true`
* `mode`: either `append` (default) or `replace`

* * *

Expand Down
Loading

0 comments on commit 70f8098

Please sign in to comment.