Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support upgrades without database loss #2

Closed
frakman1 opened this issue Oct 20, 2021 · 9 comments
Closed

Support upgrades without database loss #2

frakman1 opened this issue Oct 20, 2021 · 9 comments

Comments

@frakman1
Copy link

Be aware that all state of FACT is only in the container so if you pull a new FACT version everything (most importantly your database) will be lost.

Why can't the database be on a mounted volume (-v option in docker run) so that the container can be upgraded without losing the databse? This is how most docker web-apps handle databases. For example, I can upgrade my Gitlab docker container without losing all my repos/user info.
@jstucke
Copy link
Collaborator

jstucke commented Oct 21, 2021

It should be possible to mount your own database (with -v) and mount your own main.cfg and mongo.conf (where the database path and passwords are configured). The text only refers to the default configuration where everything is contained inside the container (and nothing needs to be manually configured). Maybe it is a bit ambiguous in this regard.

@jstucke
Copy link
Collaborator

jstucke commented Oct 21, 2021

It is probably necessary to touch [YOUR_FACT_WT_MONGODB_PATH]/REINITIALIZE_DB (as it says in the readme.md) so that the new database is correctly initialized when you start the container, though.

@frakman1
Copy link
Author

frakman1 commented Oct 21, 2021 •
edited
Loading

What is YOUR_FACT_WT_MONGODB_PATH? (What does the WT stand for?) is it in /media/ or /media/data/ or inside one of the three folders (fact_auth_data fact_fw_data fact_wt_mongodb)? Although the touch command is mentioned in the README.md, it doesn't say what that path is or what it means. It seems you want the user to create a new file but it's not clear to me where.

Sorry if this is obvious but I have no experience with mongodb.

UPDATE:
OK, I am guessing it is: media/data/fact_wt_mongodb/REINITIALIZE_DB

@jstucke
Copy link
Collaborator

jstucke commented Oct 21, 2021

OK, I am guessing it is: media/data/fact_wt_mongodb/REINITIALIZE_DB

Yes, that should be the correct path. I will update the readme to make it clearer.

@frakman1
Copy link
Author

frakman1 commented Oct 21, 2021 •
edited
Loading

I am currently facing permission issues between the container and host. I keep getting permission errors for the media sub-folders despite using --group-add $(getent group docker | cut -d: -f3)
I got past this using the brute force chmod -R 777 for the /tmp/... folder but I'm not getting any luck with the media folder.

PermissionError: [Errno 13] Permission denied: '/media/data'

I'm currently running as:

docker run -it --name fact --group-add $(getent group docker | cut -d: -f3) -v /home/frak/git/tch/fact/media:/media -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/fact-docker-tmp:/tmp/fact-docker-tmp -p 0.0.0.0:5000:5000 frakman1/fact:latest start

with themedia folder in the current working directory. I even tried starting with a clean slate with fresh, empty folder and also did a chmod -R 777 on media but still got permission errors.

Full error message:

[2021-10-21 14:31:23][start_all_installed_fact_components][INFO]: starting db
Traceback (most recent call last):
  File "/opt/FACT_core/src/../start_fact_db", line 44, in <module>
    FactDb().main()
  File "/opt/FACT_core/src/../start_fact_db", line 35, in __init__
    self.mongo_server = MongoMgr(config=config)
  File "/opt/FACT_core/src/storage/MongoMgr.py", line 28, in __init__
    os.makedirs(self.mongo_db_file_path, exist_ok=True)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/media/data'
[2021-10-21 14:31:25][start_all_installed_fact_components][INFO]: starting frontend
[2021-10-21 14:31:25][start_all_installed_fact_components][INFO]: starting backend
^C[2021-10-21 14:31:35][start_all_installed_fact_components][INFO]: shutting down...
[2021-10-21 14:31:36][fact_base][INFO]: Received signal 10. Shutting down FACT Backend...
Traceback (most recent call last):
  File "/opt/FACT_core/src/../start_fact_frontend", line 94, in <module>
    FactFrontend().main()
  File "/opt/FACT_core/src/../start_fact_frontend", line 73, in __init__
    super().__init__()
  File "/opt/FACT_core/src/fact_base.py", line 31, in __init__
    self.work_load_stat = WorkLoadStatistic(config=self.config, component=self.COMPONENT)
  File "/opt/FACT_core/src/statistic/work_load.py", line 21, in __init__
    self.db = StatisticDbUpdater(config=self.config)
  File "/opt/FACT_core/src/storage/db_interface_statistic.py", line 15, in __init__
    super().__init__(config=config)
  File "/opt/FACT_core/src/storage/mongo_interface.py", line 25, in __init__
    self._authenticate()
  File "/opt/FACT_core/src/storage/mongo_interface.py", line 40, in _authenticate
    self.client.admin.authenticate(user, pw, mechanism='SCRAM-SHA-1')
  File "/usr/local/lib/python3.8/dist-packages/pymongo/database.py", line 1578, in authenticate
    self.client._cache_credentials(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 809, in _cache_credentials
    server = self._get_topology().select_server(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/topology.py", line 244, in select_server
    return random.choice(self.select_servers(selector,
  File "/usr/local/lib/python3.8/dist-packages/pymongo/topology.py", line 202, in select_servers
    server_descriptions = self._select_servers_loop(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/topology.py", line 218, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost:27018: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 617179bd21b4b87c3ac60a38, topology_type: Single, servers: [<ServerDescription ('localhost', 27018) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27018: [Errno 111] Connection refused')>]>
Traceback (most recent call last):
  File "/opt/FACT_core/src/../start_fact_backend", line 91, in <module>
    FactBackend().main()
  File "/opt/FACT_core/src/../start_fact_backend", line 39, in __init__
    super().__init__()
  File "/opt/FACT_core/src/fact_base.py", line 31, in __init__
    self.work_load_stat = WorkLoadStatistic(config=self.config, component=self.COMPONENT)
  File "/opt/FACT_core/src/statistic/work_load.py", line 21, in __init__
    self.db = StatisticDbUpdater(config=self.config)
  File "/opt/FACT_core/src/storage/db_interface_statistic.py", line 15, in __init__
    super().__init__(config=config)
  File "/opt/FACT_core/src/storage/mongo_interface.py", line 25, in __init__
    self._authenticate()
  File "/opt/FACT_core/src/storage/mongo_interface.py", line 40, in _authenticate
    self.client.admin.authenticate(user, pw, mechanism='SCRAM-SHA-1')
  File "/usr/local/lib/python3.8/dist-packages/pymongo/database.py", line 1578, in authenticate
    self.client._cache_credentials(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 809, in _cache_credentials
    server = self._get_topology().select_server(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/topology.py", line 244, in select_server
    return random.choice(self.select_servers(selector,
  File "/usr/local/lib/python3.8/dist-packages/pymongo/topology.py", line 202, in select_servers
    server_descriptions = self._select_servers_loop(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/topology.py", line 218, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost:27018: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 617179be6bb05944cc064016, topology_type: Single, servers: [<ServerDescription ('localhost', 27018) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27018: [Errno 111] Connection refused')>]>

@frakman1
Copy link
Author

frakman1 commented Oct 21, 2021 •
edited
Loading

When I added -e PUID=1000 -e PGID=1000 -e UMASK=0000 to docker run to use my username/group as obtained from /etc/passwd, I got a different error:

$ docker run -it --name fact --group-add $(getent group docker | cut -d: -f3) -e PUID=1000 -e PGID=1000 -e UMASK=0000  -v /home/frak/git/tch/fact/media:/media -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/fact-docker-tmp:/tmp/fact-docker-tmp -p 0.0.0.0:5000:5000 frakman1/fact:latest start
[2021-10-21 14:42:20][start_all_installed_fact_components][INFO]: starting db
[2021-10-21 14:42:20][MongoMgr][INFO]: Starting local mongo database
[2021-10-21 14:42:20][MongoMgr][INFO]: Starting DB: mongod --auth --config /opt/FACT_core/src/config/mongod.conf --fork --logpath /var/log/fact/mongo.log
[2021-10-21 14:42:21][process][WARNING]: Error: Authentication not successful: Authentication failed., full error: {'ok': 0.0, 'errmsg': 'Authentication failed.', 'code': 18, 'codeName': 'AuthenticationFailed'}
[2021-10-21 14:42:21][process][WARNING]: Error: Authentication not successful: Authentication failed., full error: {'ok': 0.0, 'errmsg': 'Authentication failed.', 'code': 18, 'codeName': 'AuthenticationFailed'}
[2021-10-21 14:42:21][process][CRITICAL]: SHUTTING DOWN SYSTEM
[2021-10-21 14:42:21][process][CRITICAL]: SHUTTING DOWN SYSTEM
[2021-10-21 14:42:22][start_all_installed_fact_components][INFO]: starting frontend
[2021-10-21 14:42:22][start_all_installed_fact_components][INFO]: starting backend
[2021-10-21 14:42:22][process][WARNING]: Error: Authentication not successful: Authentication failed., full error: {'ok': 0.0, 'errmsg': 'Authentication failed.', 'code': 18, 'codeName': 'AuthenticationFailed'}
[2021-10-21 14:42:22][process][CRITICAL]: SHUTTING DOWN SYSTEM
[2021-10-21 14:42:22][process][WARNING]: Error: Authentication not successful: Authentication failed., full error: {'ok': 0.0, 'errmsg': 'Authentication failed.', 'code': 18, 'codeName': 'AuthenticationFailed'}
[2021-10-21 14:42:22][process][CRITICAL]: SHUTTING DOWN SYSTEM
[2021-10-21 14:42:24][start_all_installed_fact_components][CRITICAL]: Backend did not start. Shutting down...

I saw this in /var/log/fact/mongo.log:

2021-10-21T14:42:21.568+0000 I INDEX    [LogicalSessionCacheRefresh] 	 building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2021-10-21T14:42:21.571+0000 I INDEX    [LogicalSessionCacheRefresh] build index done.  scanned 0 total records. 0 secs
2021-10-21T14:42:22.668+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:36336 #3 (1 connection now open)
2021-10-21T14:42:22.669+0000 I NETWORK  [conn3] received client metadata from 127.0.0.1:36336 conn3: { driver: { name: "PyMongo", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-84-generic" }, platform: "CPython 3.8.10.final.0" }
2021-10-21T14:42:22.671+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:36338 #4 (2 connections now open)
2021-10-21T14:42:22.671+0000 I NETWORK  [conn4] received client metadata from 127.0.0.1:36338 conn4: { driver: { name: "PyMongo", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-84-generic" }, platform: "CPython 3.8.10.final.0" }
2021-10-21T14:42:22.672+0000 I ACCESS   [conn4] SCRAM-SHA-1 authentication failed for fact_admin on admin from client 127.0.0.1:36338 ; UserNotFound: Could not find user fact_admin@admin
2021-10-21T14:42:22.678+0000 I NETWORK  [conn4] end connection 127.0.0.1:36338 (1 connection now open)
2021-10-21T14:42:22.678+0000 I NETWORK  [conn3] end connection 127.0.0.1:36336 (0 connections now open)
2021-10-21T14:42:22.913+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:36340 #5 (1 connection now open)
2021-10-21T14:42:22.914+0000 I NETWORK  [conn5] received client metadata from 127.0.0.1:36340 conn5: { driver: { name: "PyMongo", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-84-generic" }, platform: "CPython 3.8.10.final.0" }
2021-10-21T14:42:22.915+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:36342 #6 (2 connections now open)
2021-10-21T14:42:22.916+0000 I NETWORK  [conn6] received client metadata from 127.0.0.1:36342 conn6: { driver: { name: "PyMongo", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-84-generic" }, platform: "CPython 3.8.10.final.0" }
2021-10-21T14:42:22.917+0000 I ACCESS   [conn6] SCRAM-SHA-1 authentication failed for fact_admin on admin from client 127.0.0.1:36342 ; UserNotFound: Could not find user fact_admin@admin
2021-10-21T14:42:22.923+0000 I NETWORK  [conn6] end connection 127.0.0.1:36342 (1 connection now open)
2021-10-21T14:42:22.923+0000 I NETWORK  [conn5] end connection 127.0.0.1:36340 (0 connections now open)

@frakman1
Copy link
Author

frakman1 commented Oct 21, 2021 •
edited
Loading

I also tried mapping to the host's /media folder with no luck.
It would be great if there was an official, working docker image.

I tried a publicly available MIPS based-squashfs openwrt image file but also saw that there was no cwe_checker output and no binwalk tab output (entropy image missing) so I doubt it's something to do with my ARM based binaries.

I opened a separate ticket for the binwalk issue here

@frakman1
Copy link
Author

frakman1 commented Oct 21, 2021 •
edited
Loading

OK, the following comment in another issue resolved the database and general permissions issue :
#3 (comment)

Thank you for that!

@frakman1
Copy link
Author

frakman1 commented Oct 21, 2021 •
edited
Loading

Have you considered creating a Discussions forum for this repository?
It would be a good way for people to ask questions about getting FACT to work rather than create issue tickets which is not really the intent of the Issues feature.

Discussions are a bult-in feature of Github and is available under a repository's Settings page:

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@frakman1 @jstucke