Skip to content
This repository has been archived by the owner on Jun 17, 2023. It is now read-only.

FAQ

Jump to bottom
meetzoo edited this page Nov 13, 2018 · 43 revisions

Asking for help

if what you're looking for doesn't appear in the FAQ, here's what info we'll probably need when you log an issue (be sure to fill out the issue template completely with logs, etc) or ask the list, be sure to post the relevant information:

  • SEARCH THE MAILING LIST, there's a lot of good info in there.
  • steps to reproduce the problem
  • release version of your OS, and of CIF
  • your obfuscated config
  • recent logs as a result of the problem
  • use something like Github Gists to paste the relevant information
  • BE SURE TO OBFUSCATE SENSITIVE DATA

Install Failed

  • did you install using an official deploykit release (eg: a tagged release, NOT from the master branch)?
  • Are you using the latest version of Ubuntu16 LTS (16.04.3) ?
  • Are you using a vanilla install to test with (no extra checked or un-checked boxes? customizations, puppet, chef, salt, etc..) ?
  • Are you testing with the SQLite install?
  • did you check the logs? (journalctl -u cif-* -n 2000)
  • before you log an issue, make sure to include a snippet of your easybutton run and your logs to help scope what went wrong and where..

Proxies

Everything under the hood uses the Python Requests. This means you need to set the correct environmental var's for both csirtg-smrt and the cif command to pick up your proxy. In the case of csirtg-smrt it means adjusting the /etc/smrt.env so the csirtg-smrt service picks up these variables.

CIF_LOGLEVEL=INFO
...

HTTPS_PROXY=localhost:8443

Be sure to execute a systemctl daemon-reload && systemctl restart csirtg-smrt so the values are picked up correctly.

Query Failed

After running the easy-button, if the queries are failing, make sure you're logged in as the cif user, not root or your normal user. The easybutton installs ~/.cif.yml to /home/cif/.cif.yml which contains the necessarily tokens for accessing your CIF instance.

Also- if you're not getting results right after the install, that's OK. csirtg-smrt takes a few minutes to run, and the hunters take a few minutes to process the data. Give your system some time to get through a few data passes before logging an issue. Then try a query like:

$ cif --itype ipv4 --tags scanner --confidence 2

Searching Logs

  1. Make sure you enable the following vars variables in /etc/cif.env

    CIF_ROUTER_TRACE=1 
CIF_STORE_TRACE=1 
CIF_HTTPD_TRACE=1 
CIF_HUNTER_TRACE=1
CIF_GATHERER_TRACE=1

  2. Restart cif-router and cif-httpd and then check the logs

    $ sudo systemctl restart cif-router
$ sudo systemctl restart cif-httpd
$ sudo journalctl -fu cif-* -n 2000

Router Dies

  1. Did you enable debugging and check the logs?
  2. Did you try adding more memory (eg: try 16+GB) to see if the problem clears up?
  3. Did you try a cron that restarts cif-router and cif-httpd every week (sometimes this clears up possible memory leaks on smaller systems).
  4. Are you using SQLite and need to switch to ElasticSearch?

Server Busy

  1. See Router Dies above.

Ansible Site

Ansible is your friend- the easybutton is just an example of how to build an all-in-one demo box. Production environments require a bit more TLC. Check out:

What is a hunter?

gathers enrich the data (geo, cc, asn), hunters “pile on more data” (eg: take a url and generate an fqdn, ip, etc.. all with lower confidence values of course).

as you query- you’ll see more and more results as they come in. easy to test and see with the openphish feed, where it’s taking lots of urls and turning them into ip addresses over time. (hint: use the —itype|—provider|—tags combo to see the differing datasets as they get generated).

ref:

Hunters off by default

In the latest default install we're leaving those blank on purpose. We've had numerous issues over the years with new users who didn't quite understand what hunters (formally cif-worker) did, and why the default install wreaked havoc on their system (network traffic, cpu utilization, as hunters churned through messages). There's still some FAQ work to be done around this specific question and the best ways to turn them back on by default..

To turn hunters back on, modify /etc/cif.env or /etc/default/cif to include the following lines:

# this will enable 2 hunter threads, scale up slowly to 4 or 6 as you add larger URL or FQDN feeds
CIF_HUNTER_THREADS=2

# by default, hunters will NOT do things that access the network (dns resolution, etc). enabling this
# will tell the more advanced hunters to start reversing data that requires a network lookup
# start with just enabling the basic hunters above, then slowly add this and keep an eye on your outbound
# port 53 (DNS) traffic.
CIF_HUNTERS_ADVANCED=1

and then restart cif-router.

also see: https://github.com/csirtgadgets/bearded-avenger-deploymentkit/issues/7

Hunters Spam

In v3, 'hunters' (aka: cif-workers in v2) are turned off by default, there-for data you may have been used to seeing in v2 may no longer show up in v3. To resolve this, see the 'Hunters off by default' FAQ above to turn this back on. Be sure to give your CIFv3 instance a few days to resolve some data before logging an issue.

If you like their data and want more of it, while we don't endorse them specifically, it may be worth reaching out and building a relationship with them.

ref: https://groups.google.com/forum/#!topic/ci-framework/nbp2KIj_F6k

Google Feed

This happens when you're doing something like this:

$ cif --itype fqdn --tags malware

and missing the --feed flag. This flag applies a whitelist to the feed before it's sent back to your client. Retry with this flag.

DNS Warnings

  • this can be an EC2-like instance, but be ware of the network activity coming from the box, it could be flagged as malicious, check with your provider's policies
  • with post processing, these boxes make a lot of threaded DNS resolution requests, make sure you understand your operating environment and work with your network team to address high volume dns queries

DNS References

Ubuntu LTS Desktop

Is the "Desktop" version of Ubuntu LTS supported? No. Use the server version.

https://groups.google.com/forum/#!topic/ci-framework/2A8Hhv9WG-g

Elasticsearch

Elasticsearch has a lot of moving parts- generally troubleshooting starts with checking the cluster health, and figuring out that you probably need to add more memory (up to ~32GB), or past that, new nodes:

  1. https://www.google.com/search?q=check+cluster+health
  2. https://discuss.elastic.co/t/how-to-fix-cluster-health-yellow-with-elastic-search/23484
  3. https://www.google.com/search?q=elasticsearch+add+node+to+cluster

Generally- a single node install should have the correct configuration to add new nodes, but in certain instances you may need to rebuild your cluster from scratch.

Developmentkit

see: https://github.com/csirtgadgets/bearded-avenger-deploymentkit/wiki/Development-Guide

What open source feeds come out of the box?

See the current rules directory:

https://github.com/csirtgadgets/bearded-avenger/tree/master/rules/default

Elasticsearch Version

Currently the 5.6.x series.

Elasticsearch Tokens

$ curl -XGET localhost:9200/tokens/_search?pretty

Smrt debugging

To enable csirtg-smrt debugging in service mode, you need to modify the /etc/systemd/system/csirtg-smrt.service file and reload systemd:

$ vi /etc/systemd/system/csirtg-smrt.service

ExecStart=../csirtg-smrt --remember --service ... -d

$ systemctl daemon-reload
$ systemctl restart csirtg-smrt
$ journalctl -fu csirtg-smrt*

ASN Resolution

MaxMind stopped providing some of their free databases.

https://groups.google.com/forum/#!topic/ci-framework/rjnR2QH7Y6g
https://blog.maxmind.com/2018/01/02/discontinuation-of-the-geolite-legacy-databases/

Unauthorized

Usually this is because either there is a token mix-match, or no token found.

  • run $ cif-tokens as the cif user, make sure the tokens are there and they match your tokens
  • check env | grep CIF variables, are your tokens there and do they match?
  • check /home/cif/.cif.yml, do you see token: aaabbbccc...
Clone this wiki locally