Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Proxmox support #837

Open
wants to merge 42 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
77f6be4
Created base proxmox os plugin
otnxSl Mar 12, 2024
3c45dff
Implemented proxmox version retrieval
otnxSl Mar 12, 2024
76cb753
Implemented ability to parse, setup and mount pmxcfs from database.
otnxSl Mar 13, 2024
84df21e
Added helper functions of vm listing
otnxSl Mar 19, 2024
ca59ceb
W.I.P. VM listing function
otnxSl Mar 19, 2024
350b5e0
Refractoring of helper functions and completing vm listing function
otnxSl Mar 20, 2024
264e588
Made child plugin for loading vm
otnxSl Mar 20, 2024
40bdb32
Modified proxmox plugin to facilitate child loading
otnxSl Mar 20, 2024
b36b1b9
Created function that adds lvm devices to target fs
otnxSl Apr 10, 2024
b58fa36
Added missing deps
otnxSl Apr 12, 2024
7755b18
Refractored logic for proxmox loader
otnxSl Apr 12, 2024
c7a613d
Created proxmox loader (w.i.p.)
otnxSl Apr 12, 2024
c75ede5
Added disk mapping functionality in loader
otnxSl May 1, 2024
d2ccd26
Fixed Bug causing lvm volumes to be added twice per taget volume
otnxSl May 1, 2024
f4a6766
Re-implemented Breadth-first search logic into pmxcfs creation
otnxSl May 15, 2024
32bf4e3
Temporary changes to test proof-of-concept
otnxSl May 15, 2024
3c6b10e
Implemented file & directory fs mounting with metadata
otnxSl Jun 12, 2024
0f929d5
Merge branch 'feature/proxmox-compatability' into feature/proxmox-imp…
otnxSl Jun 22, 2024
9d72586
Fixed bug causing pmxcfs root directories to be empty
otnxSl Jul 9, 2024
28e6e6e
Updated code to work with properly implemented pmxcfs
otnxSl Jul 9, 2024
d47aff6
Cleaned up code a bit
otnxSl Jul 9, 2024
6358c46
Fixed KeyError when loading Windows targets over SMB (#726)
Paradoxis Jul 9, 2024
dbe5869
Add glob/dump function for config tree (#728)
cecinestpasunepipe Jul 9, 2024
9640951
Fix edge case where unix history path is a directory (#727)
JSCU-CNI Jul 9, 2024
480317f
Bump dissect.ctruct dependency to version 4 (#731)
pyrco Jul 9, 2024
823dc77
Correctly detect Windows 11 builds (#714)
JSCU-CNI Jul 9, 2024
f7abd55
Fix EOF read error for char arrays in a BEEF0004 shellbag (#730)
Miauwkeru Jul 9, 2024
55fd035
Add username and password options to MQTT loader (#732)
cecinestpasunepipe Jul 9, 2024
2c88703
Make ESXi Plugin work without crypto and fix vm_inventory (#697)
Matthijsy Jul 9, 2024
d8df205
Fix visual bugs in cyber (#738)
Schamper Jul 9, 2024
fa17a5e
Improve type hint in Defender plugin (#739)
Schamper Jul 9, 2024
7b95394
Fix issue with MPLogs (#742)
cecinestpasunepipe Jul 9, 2024
1bd787f
Use target logger in etc-plugin (#741)
cecinestpasunepipe Jul 9, 2024
629dfcb
Fix TargetPath instances for configutil.parse (#743)
Miauwkeru Jul 9, 2024
737d9e3
Merge branch 'fox-it:main' into feature/proxmox-implementation
otnxSl Jul 9, 2024
63dccd6
Refactor
Schamper Aug 22, 2024
c57f4b1
Merge branch 'main' into feature/proxmox-implementation
Schamper Aug 26, 2024
66dcefb
Use a layer instead of a mount for /dev
Schamper Aug 26, 2024
c73987b
Add unit test
Schamper Aug 26, 2024
346e0ec
Add Proxmox unit test file
Schamper Aug 26, 2024
4c98d9b
Update dissect/target/plugins/os/unix/linux/debian/proxmox/_os.py
Schamper Sep 13, 2024
7a5bd65
Merge branch 'main' into add-proxmox-support
Horofic Sep 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions dissect/target/loader.py
Original file line number Diff line number Diff line change
Expand Up @@ -206,4 +206,5 @@ def open(item: Union[str, Path], *args, **kwargs) -> Loader:
register("smb", "SmbLoader")
register("cb", "CbLoader")
register("cyber", "CyberLoader")
register("proxmox", "ProxmoxLoader")
register("multiraw", "MultiRawLoader") # Should be last
68 changes: 68 additions & 0 deletions dissect/target/loaders/proxmox.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
from __future__ import annotations

import re
from pathlib import Path

from dissect.target import container
from dissect.target.loader import Loader
from dissect.target.target import Target

RE_VOLUME_ID = re.compile(r"(?:file=)?([^:]+):([^,]+)")


class ProxmoxLoader(Loader):
"""Loader for Proxmox VM configuration files.

Proxmox uses volume identifiers in the format of ``storage_id:volume_id``. The ``storage_id`` maps to a
storage configuration in ``/etc/pve/storage.cfg``. The ``volume_id`` is the name of the volume within
that configuration.

This loader currently does not support parsing the storage configuration, so it will attempt to open the
volume directly from the same directory as the configuration file, or from ``/dev/pve/`` (default LVM config).
If the volume is not found, it will log a warning.
"""

def __init__(self, path: Path, **kwargs):
path = path.resolve()
super().__init__(path)
self.base_dir = path.parent

Check warning on line 28 in dissect/target/loaders/proxmox.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/loaders/proxmox.py#L26-L28

Added lines #L26 - L28 were not covered by tests

@staticmethod
def detect(path: Path) -> bool:
if path.suffix.lower() != ".conf":
return False

Check warning on line 33 in dissect/target/loaders/proxmox.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/loaders/proxmox.py#L32-L33

Added lines #L32 - L33 were not covered by tests

with path.open("rb") as fh:
lines = fh.read(512).split(b"\n")
needles = [b"cpu:", b"memory:", b"name:"]
return all(any(needle in line for line in lines) for needle in needles)

Check warning on line 38 in dissect/target/loaders/proxmox.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/loaders/proxmox.py#L35-L38

Added lines #L35 - L38 were not covered by tests

def map(self, target: Target) -> None:
with self.path.open("rt") as fh:
for line in fh:
if not (line := line.strip()):
continue

Check warning on line 44 in dissect/target/loaders/proxmox.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/loaders/proxmox.py#L41-L44

Added lines #L41 - L44 were not covered by tests

key, value = line.split(":", 1)
value = value.strip()

Check warning on line 47 in dissect/target/loaders/proxmox.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/loaders/proxmox.py#L46-L47

Added lines #L46 - L47 were not covered by tests

if key.startswith(("scsi", "sata", "ide", "virtio")) and key[-1].isdigit():

Check warning on line 49 in dissect/target/loaders/proxmox.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/loaders/proxmox.py#L49

Added line #L49 was not covered by tests
# https://pve.proxmox.com/wiki/Storage
if match := RE_VOLUME_ID.match(value):
storage_id, volume_id = match.groups()

Check warning on line 52 in dissect/target/loaders/proxmox.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/loaders/proxmox.py#L51-L52

Added lines #L51 - L52 were not covered by tests

# TODO: parse the storage information from /etc/pve/storage.cfg
# For now, let's try a few assumptions
disk_path = None
if (path := self.base_dir.joinpath(volume_id)).exists():
disk_path = path
elif (path := self.base_dir.joinpath("/dev/pve/").joinpath(volume_id)).exists():
disk_path = path

Check warning on line 60 in dissect/target/loaders/proxmox.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/loaders/proxmox.py#L56-L60

Added lines #L56 - L60 were not covered by tests

if disk_path:
try:
target.disks.add(container.open(disk_path))
except Exception:
target.log.exception("Failed to open disk: %s", disk_path)

Check warning on line 66 in dissect/target/loaders/proxmox.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/loaders/proxmox.py#L62-L66

Added lines #L62 - L66 were not covered by tests
else:
target.log.warning("Unable to find disk: %s:%s", storage_id, volume_id)

Check warning on line 68 in dissect/target/loaders/proxmox.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/loaders/proxmox.py#L68

Added line #L68 was not covered by tests
15 changes: 8 additions & 7 deletions dissect/target/plugin.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,17 +57,18 @@


class OperatingSystem(StrEnum):
LINUX = "linux"
WINDOWS = "windows"
ESXI = "esxi"
ANDROID = "android"
BSD = "bsd"
CITRIX = "citrix-netscaler"
ESXI = "esxi"
FORTIOS = "fortios"
IOS = "ios"
LINUX = "linux"
OSX = "osx"
PROXMOX = "proxmox"
UNIX = "unix"
ANDROID = "android"
VYOS = "vyos"
IOS = "ios"
FORTIOS = "fortios"
CITRIX = "citrix-netscaler"
WINDOWS = "windows"


def export(*args, **kwargs) -> Callable:
Expand Down
23 changes: 23 additions & 0 deletions dissect/target/plugins/child/proxmox.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
from typing import Iterator

from dissect.target.exceptions import UnsupportedPluginError
from dissect.target.helpers.record import ChildTargetRecord
from dissect.target.plugin import ChildTargetPlugin


class ProxmoxChildTargetPlugin(ChildTargetPlugin):
"""Child target plugin that yields from the VM listing."""

__type__ = "proxmox"

def check_compatible(self) -> None:
if self.target.os != "proxmox":
raise UnsupportedPluginError("Not a Proxmox operating system")

def list_children(self) -> Iterator[ChildTargetRecord]:
for vm in self.target.vmlist():
yield ChildTargetRecord(

Check warning on line 19 in dissect/target/plugins/child/proxmox.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/plugins/child/proxmox.py#L18-L19

Added lines #L18 - L19 were not covered by tests
type=self.__type__,
path=vm.path,
_target=self.target,
)
12 changes: 12 additions & 0 deletions dissect/target/plugins/os/unix/_os.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
def __init__(self, target: Target):
super().__init__(target)
self._add_mounts()
self._add_devices()
self._hostname_dict = self._parse_hostname_string()
self._hosts_dict = self._parse_hosts_string()
self._os_release = self._parse_os_release()
Expand Down Expand Up @@ -244,6 +245,17 @@
self.target.log.debug("Mounting %s (%s) at %s", fs, fs.volume, mount_point)
self.target.fs.mount(mount_point, fs)

def _add_devices(self) -> None:
"""Add some virtual block devices to the target.

Currently only adds LVM devices.
"""
vfs = self.target.fs.append_layer()

for volume in self.target.volumes:
if volume.vs and volume.vs.__type__ == "lvm":
vfs.map_file_fh(f"/dev/{volume.raw.vg.name}/{volume.raw.name}", volume)

Check warning on line 257 in dissect/target/plugins/os/unix/_os.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/plugins/os/unix/_os.py#L256-L257

Added lines #L256 - L257 were not covered by tests

def _parse_os_release(self, glob: Optional[str] = None) -> dict[str, str]:
"""Parse files containing Unix version information.

Expand Down
Empty file.
141 changes: 141 additions & 0 deletions dissect/target/plugins/os/unix/linux/debian/proxmox/_os.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
from __future__ import annotations

import stat
from io import BytesIO
from typing import BinaryIO

from dissect.sql import sqlite3
from dissect.util.stream import BufferedStream

from dissect.target.filesystem import (
Filesystem,
VirtualDirectory,
VirtualFile,
VirtualFilesystem,
)
from dissect.target.helpers import fsutil
from dissect.target.plugins.os.unix._os import OperatingSystem, export
from dissect.target.plugins.os.unix.linux.debian._os import DebianPlugin
from dissect.target.target import Target


class ProxmoxPlugin(DebianPlugin):
@classmethod
def detect(cls, target: Target) -> Filesystem | None:
for fs in target.filesystems:
if fs.exists("/etc/pve") or fs.exists("/var/lib/pve"):
return fs

return None
Horofic marked this conversation as resolved.
Show resolved Hide resolved

@classmethod
def create(cls, target: Target, sysvol: Filesystem) -> ProxmoxPlugin:
obj = super().create(target, sysvol)

if (config_db := target.fs.path("/var/lib/pve-cluster/config.db")).exists():
with config_db.open("rb") as fh:
vfs = _create_pmxcfs(fh, obj.hostname)

target.fs.mount("/etc/pve", vfs)

return obj

@export(property=True)
def version(self) -> str:
"""Returns Proxmox VE version with underlying OS release."""

for pkg in self.target.dpkg.status():
if pkg.name == "proxmox-ve":
distro_name = self._os_release.get("PRETTY_NAME", "")
return f"{pkg.name} {pkg.version} ({distro_name})"

Check warning on line 50 in dissect/target/plugins/os/unix/linux/debian/proxmox/_os.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/plugins/os/unix/linux/debian/proxmox/_os.py#L47-L50

Added lines #L47 - L50 were not covered by tests

@export(property=True)
def os(self) -> str:
return OperatingSystem.PROXMOX.value


DT_DIR = 4
DT_REG = 8


def _create_pmxcfs(fh: BinaryIO, hostname: str | None = None) -> VirtualFilesystem:
# https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs)
db = sqlite3.SQLite3(fh)

entries = {row.inode: row for row in db.table("tree")}

vfs = VirtualFilesystem()
for entry in entries.values():
if entry.type == DT_DIR:
cls = ProxmoxConfigDirectoryEntry
elif entry.type == DT_REG:
cls = ProxmoxConfigFileEntry
else:
raise ValueError(f"Unknown pmxcfs file type: {entry.type}")

Check warning on line 74 in dissect/target/plugins/os/unix/linux/debian/proxmox/_os.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/plugins/os/unix/linux/debian/proxmox/_os.py#L74

Added line #L74 was not covered by tests

parts = []
current = entry
while current.parent != 0:
parts.append(current.name)
current = entries[current.parent]
parts.append(current.name)

path = "/".join(parts[::-1])
vfs.map_file_entry(path, cls(vfs, path, entry))

if hostname:
node_root = vfs.path(f"nodes/{hostname}")
vfs.symlink(str(node_root), "local")
vfs.symlink(str(node_root / "lxc"), "lxc")
vfs.symlink(str(node_root / "openvz"), "openvz")
vfs.symlink(str(node_root / "qemu-server"), "qemu-server")

# TODO: .version, .members, .vmlist, maybe .clusterlog and .rrd?

return vfs


class ProxmoxConfigFileEntry(VirtualFile):
def open(self) -> BinaryIO:
return BufferedStream(BytesIO(self.entry.data or b""))

Check warning on line 100 in dissect/target/plugins/os/unix/linux/debian/proxmox/_os.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/plugins/os/unix/linux/debian/proxmox/_os.py#L100

Added line #L100 was not covered by tests

def lstat(self) -> fsutil.stat_result:
# ['mode', 'addr', 'dev', 'nlink', 'uid', 'gid', 'size', 'atime', 'mtime', 'ctime']
return fsutil.stat_result(

Check warning on line 104 in dissect/target/plugins/os/unix/linux/debian/proxmox/_os.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/plugins/os/unix/linux/debian/proxmox/_os.py#L104

Added line #L104 was not covered by tests
[
stat.S_IFREG | 0o640,
self.entry.inode,
id(self.fs),
1,
0,
0,
len(self.entry.data) if self.entry.data else 0,
0,
self.entry.mtime,
0,
]
)


class ProxmoxConfigDirectoryEntry(VirtualDirectory):
def __init__(self, fs: VirtualFilesystem, path: str, entry: sqlite3.Row):
super().__init__(fs, path)
self.entry = entry

def lstat(self) -> fsutil.stat_result:
"""Return the stat information of the given path, without resolving links."""
# ['mode', 'addr', 'dev', 'nlink', 'uid', 'gid', 'size', 'atime', 'mtime', 'ctime']
return fsutil.stat_result(

Check warning on line 128 in dissect/target/plugins/os/unix/linux/debian/proxmox/_os.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/plugins/os/unix/linux/debian/proxmox/_os.py#L128

Added line #L128 was not covered by tests
[
stat.S_IFDIR | 0o755,
self.entry.inode,
id(self.fs),
1,
0,
0,
0,
0,
self.entry.mtime,
0,
]
)
29 changes: 29 additions & 0 deletions dissect/target/plugins/os/unix/linux/debian/proxmox/vm.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
from typing import Iterator

from dissect.target.exceptions import UnsupportedPluginError
from dissect.target.helpers.record import TargetRecordDescriptor
from dissect.target.plugin import Plugin, export

VirtualMachineRecord = TargetRecordDescriptor(
"proxmox/vm",
[
("string", "path"),
],
)


class VirtualMachinePlugin(Plugin):
"""Plugin to list Proxmox virtual machines."""

def check_compatible(self) -> None:
if self.target.os != "proxmox":
raise UnsupportedPluginError("Not a Proxmox operating system")

Check warning on line 20 in dissect/target/plugins/os/unix/linux/debian/proxmox/vm.py

View check run for this annotation

Codecov / codecov/patch

dissect/target/plugins/os/unix/linux/debian/proxmox/vm.py#L20

Added line #L20 was not covered by tests

@export(record=VirtualMachineRecord)
def vmlist(self) -> Iterator[VirtualMachineRecord]:
"""List Proxmox virtual machines on this node."""
for config in self.target.fs.path("/etc/pve/qemu-server").iterdir():
yield VirtualMachineRecord(
path=config,
_target=self.target,
)
Git LFS file not shown
Empty file.
Loading