Skip to content

Commit

Permalink
setting up sphinx docs
Browse files Browse the repository at this point in the history
  • Loading branch information
P-T-I committed Dec 1, 2020
1 parent 0d5f429 commit 946cd12
Show file tree
Hide file tree
Showing 32 changed files with 11,176 additions and 184 deletions.
120 changes: 3 additions & 117 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,123 +25,9 @@ This document gives you basic information how to start with cve-search. For more
information please refer to the documentation in the **_/doc_** folder of this
project.

Requirements
------------

* Python 3.6 or later
* MongoDB 2.2 or later
* redis server
* Pip3
* click
* feedformater (for RSS and Atom dump_last) http://code.google.com/p/feedformatter/
* Flask
* Flask-Login
* Flask-PyMongo
* irc
* itsdangerous
* Jinja2
* lxml
* passlib
* PyMongo
* Python-dateutil
* Pytz
* Redis
* requests
* requirements-parser
* six
* sleekxmpp
* Tornado
* Werkzeug
* Whoosh http://packages.python.org/Whoosh/ (If you're planning to use the Full-text indexer)
* xlrd
* tqdm
* ijson
* jsonpickle
* nltk
* dicttoxml

The requirements can be installed with pip:

sudo pip3 install -r requirements.txt

Installation of MongoDB
-----------------------

First, you'll need to have a Python 3 installation (3.3 or higher).
Then you need to install MongoDB (2.2) from source (this should also work
with any standard packages from your favorite distribution). Don't forget
to install the headers for development while installing MongoDB.
You can go to http://docs.mongodb.org/manual/installation/ for to get the
packages for your distribution, or http://www.mongodb.org/downloads for
the source code.


Populating the database
-----------------------

For the initial run, you need to populate the CVE database by running:

./sbin/db_mgmt_cpe_dictionary.py -p
./sbin/db_mgmt_json.py -p
./sbin/db_updater.py -c # This will take >45minutes on a decent machine, please be patient

It will fetch all the existing JSON files from the Common Vulnerabilities
and Exposures feed and the Common Platform Enumeration. The initial
Common Platform Enumeration (CPE) import might take some time depending
of your configuration.

If you want to add the cross-references from NIST, Red Hat and other vendors thanks to [VIA4CVE](https://github.com/cve-search/VIA4CVE):

./sbin/db_mgmt_ref.py

NB: If you want to import your own JSON from VIA4CVE, you have to replace URL in sources.ini the VIA4 attribute with `file:///PATH/TO/VIA4CVE/VIA4CVE-feed.json`.

A more detailed documentation can be found in the Documentations folder of the project.

Databases and collections
-------------------------

The MongoDB database is called cvedb and there are 11 collections:

* cves (Common Vulnerabilities and Exposure items) - source NVD NIST (JSON)
* cpe (Common Platform Enumeration items) - source NVD NIST
* cwe (Common Weakness Enumeration items) - source NVD NIST
* capec (Common Attack Pattern Enumeration and Classification) - source NVD NIST
* ranking (ranking rules per group) - local cve-search
* [MITRE Reference Key/Maps](https://cve.mitre.org/data/refs/) - source MITRE reference Key/Maps
* info (metadata of each collection like last-modified) - local cve-search
* via4 [VIA4CVE](https://github.com/cve-search/VIA4CVE) cross-references.

The Redis database has 3 databases:

* 10: The cpe (Common Platform Enumeration) cache - source MongoDB cvedb collection cpe
* 11: The notification database - source cve-search
* 12: The [CVE reference database](https://cve.mitre.org/data/refs/) is a cross-reference database to CVE ids against various vendors ID - source NVD NIST/MITRE

The reference database has 3 additional sources:

* [MITRE Reference Key/Maps](https://cve.mitre.org/data/refs/).
* Red Hat RPM to CVE database.
* Red Hat RHSA Oval database.

Updating the database
---------------------

An updater script helps to start the db_mgmt_*

./sbin/db_updater.py -v

You can run it in a crontab, logging is done in syslog by default.

Repopulating the database
-------------------------

To easily drop and re-populate all the databases

./sbin/db_updater.py -v -f

This will drop all the existing external sources and reimport everything. This operation can take some time
and it's usually only required when new attributes parsing are added in cve-search.
Getting started
---------------
Check the [documentation](https://cve-search.github.io/cve-search/) to get you started

Usage
-----
Expand Down
2 changes: 1 addition & 1 deletion docs/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ CHANGELOG = $(WGETCMD) --no-check-certificate -O $(CHANGELOGFILE) $(CHANGELOGUR
vars = $@

html:
$(CHANGELOG)
$(CHANGELOG)
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

# Catch-all target: route all unknown targets to Sphinx using the new
Expand Down
Binary file modified docs/build/doctrees/changelog/changelog.doctree
Binary file not shown.
Binary file added docs/build/doctrees/docker/docker.doctree
Binary file not shown.
Binary file modified docs/build/doctrees/environment.pickle
Binary file not shown.
Binary file modified docs/build/doctrees/getting_started/database.doctree
Binary file not shown.
Binary file modified docs/build/doctrees/getting_started/installation.doctree
Binary file not shown.
Binary file modified docs/build/doctrees/index.doctree
Binary file not shown.
Binary file added docs/build/doctrees/software/software.doctree
Binary file not shown.
Loading

0 comments on commit 946cd12

Please sign in to comment.