Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Downstairs not coming up in crucible zone #1514

Open
askfongjojo opened this issue Oct 21, 2024 · 2 comments
Open

Downstairs not coming up in crucible zone #1514

askfongjojo opened this issue Oct 21, 2024 · 2 comments

Comments

@askfongjojo
Copy link

The issue was seen on one crucible zone on BRM42220089 of rack3. There was a problem with mupdate during that time and we're not sure if it's related at all. Other sleds and crucibles on that particular sled do not have this particular issue AFAICT. https://github.com/oxidecomputer/colo/issues/88#issuecomment-2423210321 is where we noted the problem, also cloned below:

There is something strange with some of the stuck VM's downstairs, e.g. this propolis is complaining about connection issues (which is not related to the problematic sled 9 at all):

20:28:09.664Z WARN propolis-server (vm_state_driver): ds_connection connect to [fd00:1122:3344:10c::3]:19001 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }
     = downstairs
    client = 0
    session_id = 2d5aa8d9-5a6f-49c8-96c6-8270cd6d46f4
20:28:09.664Z WARN propolis-server (vm_state_driver): client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }))
     = downstairs
    client = 0
    session_id = 2d5aa8d9-5a6f-49c8-96c6-8270cd6d46f4

The crucible zone in question has something very odd - it doesn't have any downstairs running, and its /data/crucible.json file is gone!

BRM42220089 # zlogin oxz_crucible_9a5a2fcf-44a0-4468-979a-a71686cef627
[Connected to zone 'oxz_crucible_9a5a2fcf-44a0-4468-979a-a71686cef627' pts/3]
The illumos Project     helios-2.0.22941        October 2024
root@oxz_crucible_9a5a2fcf:~# ipadm | grep 1122
oxControlService7/omicron6 static ok    fd00:1122:3344:10c::3/64
root@oxz_crucible_9a5a2fcf:~# svcs -xv
root@oxz_crucible_9a5a2fcf:~# svcs *crucible*
STATE          STIME    FMRI
disabled       17:48:59 svc:/oxide/crucible/downstairs:default
online         17:49:02 svc:/oxide/crucible/agent:default
root@oxz_crucible_9a5a2fcf:~# tail /var/svc/log/oxide-crucible-agent\:default.log 
root@oxz_crucible_9a5a2fcf:~# ls /data/regions   
root@oxz_crucible_9a5a2fcf:~# ls /data
regions
root@oxz_crucible_9a5a2fcf:~# cat /var/svc/log/oxide-crucible-agent\:default.log

The latest crucible-agent log is empty (as shown above). The most recent non-empty logs rotated out also didn't show anything out of the ordinary:

BRM42220089 # ls -ltr /pool/ext/*/crypt/debug/oxz_crucible_9a5a2fcf-44a0-4468-979a-a71686cef627/oxide-crucible-agent* | tail -3
-rw-r--r--   1 root     root           0 Oct 18 02:31 /pool/ext/74ac4da9-cdae-4c08-8431-11211184aa09/crypt/debug/oxz_crucible_9a5a2fcf-44a0-4468-979a-a71686cef627/oxide-crucible-agent:default.log.1729204200
-rw-r--r--   1 root     root        1547 Oct 18 13:48 /pool/ext/74ac4da9-cdae-4c08-8431-11211184aa09/crypt/debug/oxz_crucible_9a5a2fcf-44a0-4468-979a-a71686cef627/oxide-crucible-agent:default.log.1729258738
-rw-r--r--   1 root     root        1548 Oct 18 18:03 /pool/ext/74ac4da9-cdae-4c08-8431-11211184aa09/crypt/debug/oxz_crucible_9a5a2fcf-44a0-4468-979a-a71686cef627/oxide-crucible-agent:default.log.1729273743

BRM42220089 # cat /pool/ext/74ac4da9-cdae-4c08-8431-11211184aa09/crypt/debug/oxz_crucible_9a5a2fcf-44a0-4468-979a-a71686cef627/oxide-crucible-agent:default.log.1729273743 | looker
[ Oct 18 17:48:59 Enabled. ]
[ Oct 18 17:48:59 Rereading configuration. ]
[ Oct 18 17:49:00 Rereading configuration. ]
[ Oct 18 17:49:02 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_dbfdc981-1b81-4d7d-9449-9530890b199a/crucible -l [fd00:1122:3344:10c::3]:32345 -P 19000 -p downstairs -s snapshot"). ]
note: configured to log to "/dev/stdout"
17:49:02.544Z INFO crucible-agent: dataset: "oxp_dbfdc981-1b81-4d7d-9449-9530890b199a/crucible"
17:49:02.562Z INFO crucible-agent: listen IP: [fd00:1122:3344:10c::3]:32345
17:49:02.562Z INFO crucible-agent: SMF instance name downstairs_prefix: "downstairs"
17:49:02.765Z INFO crucible-agent (datafile): Using conf_path:"/data/crucible.json"
17:49:03.047Z INFO crucible-agent (dropshot): listening
    local_addr = [fd00:1122:3344:10c::3]:32345

BRM42220089 # cat /pool/ext/74ac4da9-cdae-4c08-8431-11211184aa09/crypt/debug/oxz_crucible_9a5a2fcf-44a0-4468-979a-a71686cef627/oxide-crucible-agent:default.log.1729258738 | looker
[ Oct 18 13:38:56 Enabled. ]
[ Oct 18 13:38:56 Rereading configuration. ]
[ Oct 18 13:38:57 Rereading configuration. ]
[ Oct 18 13:38:58 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_dbfdc981-1b81-4d7d-9449-9530890b199a/crucible -l [fd00:1122:3344:10c::3]:32345 -P 19000 -p downstairs -s snapshot"). ]
note: configured to log to "/dev/stdout"
13:38:58.215Z INFO crucible-agent: dataset: "oxp_dbfdc981-1b81-4d7d-9449-9530890b199a/crucible"
13:38:58.218Z INFO crucible-agent: listen IP: [fd00:1122:3344:10c::3]:32345
13:38:58.218Z INFO crucible-agent: SMF instance name downstairs_prefix: "downstairs"
13:38:58.244Z INFO crucible-agent (datafile): Using conf_path:"/data/crucible.json"
13:38:58.309Z INFO crucible-agent (dropshot): listening
    local_addr = [fd00:1122:3344:10c::3]:32345
@leftwo
Copy link
Contributor

leftwo commented Oct 21, 2024

For the sled where the crucible zone is having, could we see

zpool status
zfs list

Also, inside the crucible zone:

df -h
ls -ltra /data

@askfongjojo
Copy link
Author

@leftwo - Sorry about the delay in response. Here is the data from the sled and crucible zone:

BRM42220089 # zpool status
  pool: oxi_15357603-6eb5-45b4-9f3b-54fa1e73f03e
 state: ONLINE
  scan: none requested
config:

        NAME                                      STATE     READ WRITE CKSUM
        oxi_15357603-6eb5-45b4-9f3b-54fa1e73f03e  ONLINE       0     0     0
          c8t00A075013280BD08d0s5                 ONLINE       0     0     0

errors: No known data errors

  pool: oxi_25a4cacb-2148-4ad0-821a-aa26c158b557
 state: ONLINE
  scan: none requested
config:

        NAME                                      STATE     READ WRITE CKSUM
        oxi_25a4cacb-2148-4ad0-821a-aa26c158b557  ONLINE       0     0     0
          c1t00A0750132747180d0s5                 ONLINE       0     0     0

errors: No known data errors

  pool: oxp_25856a84-6707-4b94-81d1-b43d5bc990d7
 state: ONLINE
  scan: none requested
config:

        NAME                                      STATE     READ WRITE CKSUM
        oxp_25856a84-6707-4b94-81d1-b43d5bc990d7  ONLINE       0     0     0
          c6t0014EE81000BC4E0d0s0                 ONLINE       0     0     0

errors: No known data errors

  pool: oxp_55eb093d-6b6f-418c-9767-09afe4c51fff
 state: ONLINE
  scan: none requested
config:

        NAME                                      STATE     READ WRITE CKSUM
        oxp_55eb093d-6b6f-418c-9767-09afe4c51fff  ONLINE       0     0     0
          c3t0014EE81000BC3C7d0s0                 ONLINE       0     0     0

errors: No known data errors

  pool: oxp_74ac4da9-cdae-4c08-8431-11211184aa09
 state: ONLINE
  scan: none requested
config:

        NAME                                      STATE     READ WRITE CKSUM
        oxp_74ac4da9-cdae-4c08-8431-11211184aa09  ONLINE       0     0     0
          c2t0014EE81000BC4E2d0s0                 ONLINE       0     0     0

errors: No known data errors

  pool: oxp_7c121177-3210-4457-9b42-3657add6e166
 state: ONLINE
  scan: none requested
config:

        NAME                                      STATE     READ WRITE CKSUM
        oxp_7c121177-3210-4457-9b42-3657add6e166  ONLINE       0     0     0
          c4t0014EE81000BC36Ed0s0                 ONLINE       0     0     0

errors: No known data errors

  pool: oxp_842bdd28-196e-4b18-83db-68bd81176a44
 state: ONLINE
  scan: none requested
config:

        NAME                                      STATE     READ WRITE CKSUM
        oxp_842bdd28-196e-4b18-83db-68bd81176a44  ONLINE       0     0     0
          c12t0014EE81000BC34Ed0s0                ONLINE       0     0     0

errors: No known data errors

  pool: oxp_92ba1667-a6f7-4913-9b00-14825384c7bf
 state: ONLINE
  scan: none requested
config:

        NAME                                      STATE     READ WRITE CKSUM
        oxp_92ba1667-a6f7-4913-9b00-14825384c7bf  ONLINE       0     0     0
          c7t0014EE81000BC215d0s0                 ONLINE       0     0     0

errors: No known data errors

  pool: oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3
 state: ONLINE
  scan: none requested
config:

        NAME                                      STATE     READ WRITE CKSUM
        oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3  ONLINE       0     0     0
          c5t0014EE81000BC355d0s0                 ONLINE       0     0     0

errors: No known data errors

  pool: oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b
 state: ONLINE
  scan: none requested
config:

        NAME                                      STATE     READ WRITE CKSUM
        oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b  ONLINE       0     0     0
          c11t0014EE81000BC4D8d0s0                ONLINE       0     0     0

errors: No known data errors

  pool: oxp_ab62b941-5f84-42c7-929d-295b20efffe7
 state: ONLINE
  scan: none requested
config:

        NAME                                      STATE     READ WRITE CKSUM
        oxp_ab62b941-5f84-42c7-929d-295b20efffe7  ONLINE       0     0     0
          c9t0014EE81000BC449d0s0                 ONLINE       0     0     0

errors: No known data errors

  pool: oxp_dbfdc981-1b81-4d7d-9449-9530890b199a
 state: ONLINE
  scan: none requested
config:

        NAME                                      STATE     READ WRITE CKSUM
        oxp_dbfdc981-1b81-4d7d-9449-9530890b199a  ONLINE       0     0     0
          c10t0014EE81000BC3D7d0s0                ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME                  STATE     READ WRITE CKSUM
        rpool                 ONLINE       0     0     0
          /dev/ramdisk/rpool  ONLINE       0     0     0

errors: No known data errors

BRM42220089 # zfs list
NAME                                                                                                           USED  AVAIL     REFER  MOUNTPOINT
oxi_15357603-6eb5-45b4-9f3b-54fa1e73f03e                                                                       907M   724G       96K  /oxi_15357603-6eb5-45b4-9f3b-54fa1e73f03e
oxi_15357603-6eb5-45b4-9f3b-54fa1e73f03e/backing                                                                96K   724G       96K  /pool/int/15357603-6eb5-45b4-9f3b-54fa1e73f03e/backing
oxi_15357603-6eb5-45b4-9f3b-54fa1e73f03e/cluster                                                               164K   724G      164K  /pool/int/15357603-6eb5-45b4-9f3b-54fa1e73f03e/cluster
oxi_15357603-6eb5-45b4-9f3b-54fa1e73f03e/config                                                                124K   724G      124K  /pool/int/15357603-6eb5-45b4-9f3b-54fa1e73f03e/config
oxi_15357603-6eb5-45b4-9f3b-54fa1e73f03e/crash                                                                  96K   724G       96K  /pool/int/15357603-6eb5-45b4-9f3b-54fa1e73f03e/crash
oxi_15357603-6eb5-45b4-9f3b-54fa1e73f03e/debug                                                                58.3M  99.9G     58.3M  /pool/int/15357603-6eb5-45b4-9f3b-54fa1e73f03e/debug
oxi_15357603-6eb5-45b4-9f3b-54fa1e73f03e/install                                                               844M   724G      844M  /pool/int/15357603-6eb5-45b4-9f3b-54fa1e73f03e/install
oxi_25a4cacb-2148-4ad0-821a-aa26c158b557                                                                       910M   724G       96K  /oxi_25a4cacb-2148-4ad0-821a-aa26c158b557
oxi_25a4cacb-2148-4ad0-821a-aa26c158b557/backing                                                              1.49M   724G       96K  /pool/int/25a4cacb-2148-4ad0-821a-aa26c158b557/backing
oxi_25a4cacb-2148-4ad0-821a-aa26c158b557/backing/fmd                                                          1.40M   499M     1.40M  /var/fm/fmd
oxi_25a4cacb-2148-4ad0-821a-aa26c158b557/cluster                                                               164K   724G      164K  /pool/int/25a4cacb-2148-4ad0-821a-aa26c158b557/cluster
oxi_25a4cacb-2148-4ad0-821a-aa26c158b557/config                                                                124K   724G      124K  /pool/int/25a4cacb-2148-4ad0-821a-aa26c158b557/config
oxi_25a4cacb-2148-4ad0-821a-aa26c158b557/crash                                                                 120K   724G      120K  /pool/int/25a4cacb-2148-4ad0-821a-aa26c158b557/crash
oxi_25a4cacb-2148-4ad0-821a-aa26c158b557/debug                                                                58.4M  99.9G     58.4M  /pool/int/25a4cacb-2148-4ad0-821a-aa26c158b557/debug
oxi_25a4cacb-2148-4ad0-821a-aa26c158b557/install                                                               845M   724G      845M  /pool/int/25a4cacb-2148-4ad0-821a-aa26c158b557/install
oxi_25a4cacb-2148-4ad0-821a-aa26c158b557/swap                                                                   88K   724G       88K  -
oxp_25856a84-6707-4b94-81d1-b43d5bc990d7                                                                      59.4G  2.76T       24K  /oxp_25856a84-6707-4b94-81d1-b43d5bc990d7
oxp_25856a84-6707-4b94-81d1-b43d5bc990d7/crucible                                                             50.0G  2.76T     73.5K  /data
oxp_25856a84-6707-4b94-81d1-b43d5bc990d7/crucible/regions                                                     50.0G  2.76T       27K  /data/regions
oxp_25856a84-6707-4b94-81d1-b43d5bc990d7/crucible/regions/502ab050-6212-40bb-a77c-4b43b061258c                 137M  89.9G      137M  /data/regions/502ab050-6212-40bb-a77c-4b43b061258c
oxp_25856a84-6707-4b94-81d1-b43d5bc990d7/crucible/regions/60bb712a-798b-43e2-87e2-dae6a68f94fa                3.69G  26.3G     3.69G  /data/regions/60bb712a-798b-43e2-87e2-dae6a68f94fa
oxp_25856a84-6707-4b94-81d1-b43d5bc990d7/crypt                                                                9.30G  2.76T     9.02G  /pool/ext/25856a84-6707-4b94-81d1-b43d5bc990d7/crypt
oxp_25856a84-6707-4b94-81d1-b43d5bc990d7/crypt/debug                                                            99K   100G       99K  /pool/ext/25856a84-6707-4b94-81d1-b43d5bc990d7/crypt/debug
oxp_25856a84-6707-4b94-81d1-b43d5bc990d7/crypt/zone                                                            286M  2.76T      142K  /pool/ext/25856a84-6707-4b94-81d1-b43d5bc990d7/crypt/zone
oxp_25856a84-6707-4b94-81d1-b43d5bc990d7/crypt/zone/oxz_crucible_de376149-aa45-4660-9ae6-15e8ba4a4233          286M  2.76T      286M  /pool/ext/25856a84-6707-4b94-81d1-b43d5bc990d7/crypt/zone/oxz_crucible_de376149-aa45-4660-9ae6-15e8ba4a4233
oxp_25856a84-6707-4b94-81d1-b43d5bc990d7/crypt/zone/oxz_propolis-server_16fbf666-de9c-4db4-b49d-defbda01decc   135K  2.76T      135K  /pool/ext/25856a84-6707-4b94-81d1-b43d5bc990d7/crypt/zone/oxz_propolis-server_16fbf666-de9c-4db4-b49d-defbda01decc
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff                                                                       752G  2.08T       24K  /oxp_55eb093d-6b6f-418c-9767-09afe4c51fff
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crucible                                                              746G  2.08T       90K  /data
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crucible/regions                                                      746G  2.08T       34K  /data/regions
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crucible/regions/27133363-ee9e-43c7-8ed8-50924ae9b14b                2.20G  9.80G     2.20G  /data/regions/27133363-ee9e-43c7-8ed8-50924ae9b14b
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crucible/regions/30960089-3e2e-40cf-8cc7-6d18ec146150                42.5G   438G     42.5G  /data/regions/30960089-3e2e-40cf-8cc7-6d18ec146150
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crucible/regions/316d8f2e-7372-4852-a2f2-dd05a64edda2                3.28G  26.7G     3.28G  /data/regions/316d8f2e-7372-4852-a2f2-dd05a64edda2
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crucible/regions/3b7425e9-fea4-4e61-a82d-b8409020255f                9.86G  50.1G     9.86G  /data/regions/3b7425e9-fea4-4e61-a82d-b8409020255f
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crucible/regions/8d7dfb28-0b8b-44cc-9d70-ee5009ea380d                11.4G   289G     11.4G  /data/regions/8d7dfb28-0b8b-44cc-9d70-ee5009ea380d
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crucible/regions/945cd894-309e-472d-bc0a-38d47d1d31cf                4.10M  6.00G     4.10M  /data/regions/945cd894-309e-472d-bc0a-38d47d1d31cf
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crucible/regions/b34f6f6e-badb-4e32-b746-7ae54a4e258e                16.9G   751G     16.9G  /data/regions/b34f6f6e-badb-4e32-b746-7ae54a4e258e
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crucible/regions/b5e77217-f45a-4886-934d-06d5de4dcdcb                1.40G  58.6G     1.40G  /data/regions/b5e77217-f45a-4886-934d-06d5de4dcdcb
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crucible/regions/c214caa7-4d35-4c08-84e5-2a3704a61238                1.52G  28.5G     1.52G  /data/regions/c214caa7-4d35-4c08-84e5-2a3704a61238
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crucible/regions/cc6f4989-e8e5-47ca-bb39-491714ac1c31                4.10M  6.00G     4.10M  /data/regions/cc6f4989-e8e5-47ca-bb39-491714ac1c31
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crucible/regions/cd5ca14b-c1a7-4f0a-a562-8e758465ac29                1.60G  7.40G     1.60G  /data/regions/cd5ca14b-c1a7-4f0a-a562-8e758465ac29
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crucible/regions/d1666577-25cf-40ae-a9d9-100c77d4efd8                73.7M  29.9G     73.7M  /data/regions/d1666577-25cf-40ae-a9d9-100c77d4efd8
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crypt                                                                5.36G  2.08T     5.08G  /pool/ext/55eb093d-6b6f-418c-9767-09afe4c51fff/crypt
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crypt/debug                                                            99K   100G       99K  /pool/ext/55eb093d-6b6f-418c-9767-09afe4c51fff/crypt/debug
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crypt/zone                                                            286M  2.08T      100K  /pool/ext/55eb093d-6b6f-418c-9767-09afe4c51fff/crypt/zone
oxp_55eb093d-6b6f-418c-9767-09afe4c51fff/crypt/zone/oxz_crucible_2dfbd4c6-afbf-4c8c-bf40-764f02727852          285M  2.08T      285M  /pool/ext/55eb093d-6b6f-418c-9767-09afe4c51fff/crypt/zone/oxz_crucible_2dfbd4c6-afbf-4c8c-bf40-764f02727852
oxp_74ac4da9-cdae-4c08-8431-11211184aa09                                                                      77.3G  2.74T       24K  /oxp_74ac4da9-cdae-4c08-8431-11211184aa09
oxp_74ac4da9-cdae-4c08-8431-11211184aa09/crucible                                                             75.0G  2.74T     88.5K  /data
oxp_74ac4da9-cdae-4c08-8431-11211184aa09/crucible/regions                                                     75.0G  2.74T       29K  /data/regions
oxp_74ac4da9-cdae-4c08-8431-11211184aa09/crucible/regions/027f4ff4-4cb9-496b-8c4a-4d91ffa2de4d                2.24G  9.76G     2.24G  /data/regions/027f4ff4-4cb9-496b-8c4a-4d91ffa2de4d
oxp_74ac4da9-cdae-4c08-8431-11211184aa09/crucible/regions/49111829-a669-45ef-8923-3bd35ab6dd51                2.18G  3.82G     2.18G  /data/regions/49111829-a669-45ef-8923-3bd35ab6dd51
oxp_74ac4da9-cdae-4c08-8431-11211184aa09/crucible/regions/57a108eb-106c-4f8a-8156-f484d2742a04                 102M   150G      102M  /data/regions/57a108eb-106c-4f8a-8156-f484d2742a04
oxp_74ac4da9-cdae-4c08-8431-11211184aa09/crucible/regions/aadabf56-dd5c-468b-ba30-443326884653                2.18G  3.82G     2.18G  /data/regions/aadabf56-dd5c-468b-ba30-443326884653
oxp_74ac4da9-cdae-4c08-8431-11211184aa09/crucible/regions/c5e514d0-56dc-4076-a998-df88e5ffd90f                 809M  5.21G      809M  /data/regions/c5e514d0-56dc-4076-a998-df88e5ffd90f
oxp_74ac4da9-cdae-4c08-8431-11211184aa09/crypt                                                                2.27G  2.74T     1.99G  /pool/ext/74ac4da9-cdae-4c08-8431-11211184aa09/crypt
oxp_74ac4da9-cdae-4c08-8431-11211184aa09/crypt/debug                                                            99K   100G       99K  /pool/ext/74ac4da9-cdae-4c08-8431-11211184aa09/crypt/debug
oxp_74ac4da9-cdae-4c08-8431-11211184aa09/crypt/zone                                                            286M  2.74T      100K  /pool/ext/74ac4da9-cdae-4c08-8431-11211184aa09/crypt/zone
oxp_74ac4da9-cdae-4c08-8431-11211184aa09/crypt/zone/oxz_crucible_65293ce4-2e63-4336-9207-3c61f58667f9          286M  2.74T      286M  /pool/ext/74ac4da9-cdae-4c08-8431-11211184aa09/crypt/zone/oxz_crucible_65293ce4-2e63-4336-9207-3c61f58667f9
oxp_7c121177-3210-4457-9b42-3657add6e166                                                                       904G  1.93T       24K  /oxp_7c121177-3210-4457-9b42-3657add6e166
oxp_7c121177-3210-4457-9b42-3657add6e166/crucible                                                              894G  1.93T       85K  /data
oxp_7c121177-3210-4457-9b42-3657add6e166/crucible/regions                                                      894G  1.93T       33K  /data/regions
oxp_7c121177-3210-4457-9b42-3657add6e166/crucible/regions/18564dbd-c5e8-4996-9d13-e13ae02008ae                1.64G  7.36G     1.64G  /data/regions/18564dbd-c5e8-4996-9d13-e13ae02008ae
oxp_7c121177-3210-4457-9b42-3657add6e166/crucible/regions/237a348e-1593-4449-85cd-ac125b5fe56e                 118G   182G      118G  /data/regions/237a348e-1593-4449-85cd-ac125b5fe56e
oxp_7c121177-3210-4457-9b42-3657add6e166/crucible/regions/26065d99-9882-4fab-9a5b-14c766bff0bf                8.17M  12.0G     8.17M  /data/regions/26065d99-9882-4fab-9a5b-14c766bff0bf
oxp_7c121177-3210-4457-9b42-3657add6e166/crucible/regions/65705768-b1a6-492e-af4d-befa2890f6be                9.25G   141G     9.25G  /data/regions/65705768-b1a6-492e-af4d-befa2890f6be
oxp_7c121177-3210-4457-9b42-3657add6e166/crucible/regions/75a9a3db-ef81-46ce-bbeb-810035b0e7f8                10.8G  34.2G     10.8G  /data/regions/75a9a3db-ef81-46ce-bbeb-810035b0e7f8
oxp_7c121177-3210-4457-9b42-3657add6e166/crucible/regions/81340d33-1051-4239-b288-fc1200540f74                1.58G  28.4G     1.58G  /data/regions/81340d33-1051-4239-b288-fc1200540f74
oxp_7c121177-3210-4457-9b42-3657add6e166/crucible/regions/820deb62-3e8c-4e09-a0bc-0818eb15fd08                2.06M  3.00G     2.06M  /data/regions/820deb62-3e8c-4e09-a0bc-0818eb15fd08
oxp_7c121177-3210-4457-9b42-3657add6e166/crucible/regions/a9b22219-e7da-47b1-85b3-72cf5dbd4a01                20.4M  30.0G     20.4M  /data/regions/a9b22219-e7da-47b1-85b3-72cf5dbd4a01
oxp_7c121177-3210-4457-9b42-3657add6e166/crucible/regions/b37ebfd7-79ce-44c0-9e80-943dc1bb06b5                6.14M  8.99G     6.14M  /data/regions/b37ebfd7-79ce-44c0-9e80-943dc1bb06b5
oxp_7c121177-3210-4457-9b42-3657add6e166/crucible/regions/c01e6677-e9de-4358-9588-591a5f5fd904                4.61G  7.39G     4.61G  /data/regions/c01e6677-e9de-4358-9588-591a5f5fd904
oxp_7c121177-3210-4457-9b42-3657add6e166/crucible/regions/cf6b8357-7bcf-426f-af19-b0205f89a0a6                10.8G  34.2G     10.8G  /data/regions/cf6b8357-7bcf-426f-af19-b0205f89a0a6
oxp_7c121177-3210-4457-9b42-3657add6e166/crucible/regions/f0f60c10-7a24-48bc-ae53-d439497d881a                 595G   905G      595G  /data/regions/f0f60c10-7a24-48bc-ae53-d439497d881a
oxp_7c121177-3210-4457-9b42-3657add6e166/crypt                                                                10.2G  1.93T     9.43G  /pool/ext/7c121177-3210-4457-9b42-3657add6e166/crypt
oxp_7c121177-3210-4457-9b42-3657add6e166/crypt/debug                                                            99K   100G       99K  /pool/ext/7c121177-3210-4457-9b42-3657add6e166/crypt/debug
oxp_7c121177-3210-4457-9b42-3657add6e166/crypt/zone                                                            820M  1.93T      175K  /pool/ext/7c121177-3210-4457-9b42-3657add6e166/crypt/zone
oxp_7c121177-3210-4457-9b42-3657add6e166/crypt/zone/oxz_crucible_431768b8-26ba-4ab4-b616-9e183bb79b8b          286M  1.93T      286M  /pool/ext/7c121177-3210-4457-9b42-3657add6e166/crypt/zone/oxz_crucible_431768b8-26ba-4ab4-b616-9e183bb79b8b
oxp_7c121177-3210-4457-9b42-3657add6e166/crypt/zone/oxz_ntp_8c73baf7-1a58-4e2c-b4d1-966c89a18d03               227M  1.93T      227M  /pool/ext/7c121177-3210-4457-9b42-3657add6e166/crypt/zone/oxz_ntp_8c73baf7-1a58-4e2c-b4d1-966c89a18d03
oxp_7c121177-3210-4457-9b42-3657add6e166/crypt/zone/oxz_propolis-server_3c694e24-4e92-474f-a216-3ca660f90fb8   307M  1.93T      307M  /pool/ext/7c121177-3210-4457-9b42-3657add6e166/crypt/zone/oxz_propolis-server_3c694e24-4e92-474f-a216-3ca660f90fb8
oxp_842bdd28-196e-4b18-83db-68bd81176a44                                                                       563G  2.27T       24K  /oxp_842bdd28-196e-4b18-83db-68bd81176a44
oxp_842bdd28-196e-4b18-83db-68bd81176a44/crucible                                                              539G  2.27T     77.5K  /data
oxp_842bdd28-196e-4b18-83db-68bd81176a44/crucible/regions                                                      539G  2.27T       33K  /data/regions
oxp_842bdd28-196e-4b18-83db-68bd81176a44/crucible/regions/2936e4df-0436-4294-a0c2-0b2b553d4a0a                1.63G  7.37G     1.63G  /data/regions/2936e4df-0436-4294-a0c2-0b2b553d4a0a
oxp_842bdd28-196e-4b18-83db-68bd81176a44/crucible/regions/5120d0a2-47a6-47d0-94ca-a92dfd912bef                3.48G  56.5G     3.48G  /data/regions/5120d0a2-47a6-47d0-94ca-a92dfd912bef
oxp_842bdd28-196e-4b18-83db-68bd81176a44/crucible/regions/675b51e5-2462-413a-b704-9beadb5a47d6                 114M  5.89G      114M  /data/regions/675b51e5-2462-413a-b704-9beadb5a47d6
oxp_842bdd28-196e-4b18-83db-68bd81176a44/crucible/regions/689d706b-ef61-42bd-85d6-0b6fdba94cf0                10.8G  34.2G     10.8G  /data/regions/689d706b-ef61-42bd-85d6-0b6fdba94cf0
oxp_842bdd28-196e-4b18-83db-68bd81176a44/crucible/regions/7683607b-51ab-420f-8cdf-cc6c3035fcad                6.10G   594G     6.10G  /data/regions/7683607b-51ab-420f-8cdf-cc6c3035fcad
oxp_842bdd28-196e-4b18-83db-68bd81176a44/crucible/regions/8afca143-2e7a-4d63-ad58-108aeb22a085                22.0G  98.0G     21.7G  /data/regions/8afca143-2e7a-4d63-ad58-108aeb22a085
oxp_842bdd28-196e-4b18-83db-68bd81176a44/crucible/regions/a7a61ff0-f401-44b6-9e3e-1bc10728fba6                5.70G  84.3G     5.63G  /data/regions/a7a61ff0-f401-44b6-9e3e-1bc10728fba6
oxp_842bdd28-196e-4b18-83db-68bd81176a44/crucible/regions/b96f866a-b9cb-43e7-a939-f2c56453b70d                11.5G  63.5G     11.5G  /data/regions/b96f866a-b9cb-43e7-a939-f2c56453b70d
oxp_842bdd28-196e-4b18-83db-68bd81176a44/crucible/regions/d54bdbc8-3d39-4d75-b027-38caadf2edd2                 108G   132G     89.1G  /data/regions/d54bdbc8-3d39-4d75-b027-38caadf2edd2
oxp_842bdd28-196e-4b18-83db-68bd81176a44/crucible/regions/e97909cc-c898-4b21-9c43-cc70d638e972                11.0G  19.0G     11.0G  /data/regions/e97909cc-c898-4b21-9c43-cc70d638e972
oxp_842bdd28-196e-4b18-83db-68bd81176a44/crypt                                                                23.3G  2.27T     23.0G  /pool/ext/842bdd28-196e-4b18-83db-68bd81176a44/crypt
oxp_842bdd28-196e-4b18-83db-68bd81176a44/crypt/debug                                                            99K   100G       99K  /pool/ext/842bdd28-196e-4b18-83db-68bd81176a44/crypt/debug
oxp_842bdd28-196e-4b18-83db-68bd81176a44/crypt/zone                                                            286M  2.27T      100K  /pool/ext/842bdd28-196e-4b18-83db-68bd81176a44/crypt/zone
oxp_842bdd28-196e-4b18-83db-68bd81176a44/crypt/zone/oxz_crucible_22992c56-bd5a-4d0f-86c5-d6f8e87b7bbb          286M  2.27T      286M  /pool/ext/842bdd28-196e-4b18-83db-68bd81176a44/crypt/zone/oxz_crucible_22992c56-bd5a-4d0f-86c5-d6f8e87b7bbb
oxp_92ba1667-a6f7-4913-9b00-14825384c7bf                                                                      79.4G  2.74T       24K  /oxp_92ba1667-a6f7-4913-9b00-14825384c7bf
oxp_92ba1667-a6f7-4913-9b00-14825384c7bf/crucible                                                             76.3G  2.74T       73K  /data
oxp_92ba1667-a6f7-4913-9b00-14825384c7bf/crucible/regions                                                     76.3G  2.74T       31K  /data/regions
oxp_92ba1667-a6f7-4913-9b00-14825384c7bf/crucible/regions/303257a2-7360-44c8-8867-30acf809dc67                9.11G  50.9G     9.11G  /data/regions/303257a2-7360-44c8-8867-30acf809dc67
oxp_92ba1667-a6f7-4913-9b00-14825384c7bf/crucible/regions/45a17a11-d9bc-4f67-be03-98f1cc1913e3                11.7G  18.3G     9.54G  /data/regions/45a17a11-d9bc-4f67-be03-98f1cc1913e3
oxp_92ba1667-a6f7-4913-9b00-14825384c7bf/crucible/regions/96644815-0612-476f-b701-050850d67c08                45.6M  30.0G     45.6M  /data/regions/96644815-0612-476f-b701-050850d67c08
oxp_92ba1667-a6f7-4913-9b00-14825384c7bf/crucible/regions/a34910d1-6601-4b5d-a05b-2b1ddf9951ae                2.06M  3.00G     2.06M  /data/regions/a34910d1-6601-4b5d-a05b-2b1ddf9951ae
oxp_92ba1667-a6f7-4913-9b00-14825384c7bf/crucible/regions/a6647869-ddfa-4a84-9fcb-bc45d29648f0                 684M  29.3G      684M  /data/regions/a6647869-ddfa-4a84-9fcb-bc45d29648f0
oxp_92ba1667-a6f7-4913-9b00-14825384c7bf/crucible/regions/e453f7cd-dbcb-4db0-952f-1bf04d267a07                4.28G  25.7G     4.28G  /data/regions/e453f7cd-dbcb-4db0-952f-1bf04d267a07
oxp_92ba1667-a6f7-4913-9b00-14825384c7bf/crypt                                                                3.04G  2.74T     2.77G  /pool/ext/92ba1667-a6f7-4913-9b00-14825384c7bf/crypt
oxp_92ba1667-a6f7-4913-9b00-14825384c7bf/crypt/debug                                                            99K   100G       99K  /pool/ext/92ba1667-a6f7-4913-9b00-14825384c7bf/crypt/debug
oxp_92ba1667-a6f7-4913-9b00-14825384c7bf/crypt/zone                                                            286M  2.74T      100K  /pool/ext/92ba1667-a6f7-4913-9b00-14825384c7bf/crypt/zone
oxp_92ba1667-a6f7-4913-9b00-14825384c7bf/crypt/zone/oxz_crucible_22452953-ee80-4659-a555-8e027bf205b0          286M  2.74T      286M  /pool/ext/92ba1667-a6f7-4913-9b00-14825384c7bf/crypt/zone/oxz_crucible_22452953-ee80-4659-a555-8e027bf205b0
oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3                                                                       295G  2.53T       24K  /oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3
oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crucible                                                              294G  2.53T       82K  /data
oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crucible/regions                                                      294G  2.53T       31K  /data/regions
oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crucible/regions/16ffbad2-9999-4747-8745-d9b185a1c22a                20.4M  30.0G     20.4M  /data/regions/16ffbad2-9999-4747-8745-d9b185a1c22a
oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crucible/regions/30e547bb-63f6-4860-b14e-e69582a53ad4                2.18G  3.82G     2.18G  /data/regions/30e547bb-63f6-4860-b14e-e69582a53ad4
oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crucible/regions/48e587bc-ef5c-4f58-8198-1e6307eee99a                41.9G   318G     41.9G  /data/regions/48e587bc-ef5c-4f58-8198-1e6307eee99a
oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crucible/regions/525b075f-98dc-4c7d-b7ec-d600924c6c5b                 323M  29.7G      323M  /data/regions/525b075f-98dc-4c7d-b7ec-d600924c6c5b
oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crucible/regions/8586fb32-a4c6-459b-ab64-5aefb0279e10                 136M   120G      136M  /data/regions/8586fb32-a4c6-459b-ab64-5aefb0279e10
oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crucible/regions/d9a0e052-83c4-4fe4-b3c0-07f29a30504a                2.74G  57.3G     2.74G  /data/regions/d9a0e052-83c4-4fe4-b3c0-07f29a30504a
oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crucible/regions/e6408cdb-c907-4adc-823c-b6fef0acb8ab                1.65G  7.35G     1.65G  /data/regions/e6408cdb-c907-4adc-823c-b6fef0acb8ab
oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crucible/regions/e6b11223-16b4-45a1-8f41-7e2e8f0b68c8                2.25G  57.7G     2.25G  /data/regions/e6b11223-16b4-45a1-8f41-7e2e8f0b68c8
oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crucible/regions/fb2af6a6-eda1-4c49-9530-e249be4f114e                5.38G  24.6G     4.84G  /data/regions/fb2af6a6-eda1-4c49-9530-e249be4f114e
oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crypt                                                                1.39G  2.53T      101K  /pool/ext/9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crypt
oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crypt/debug                                                          1.12G  98.9G     1.12G  /pool/ext/9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crypt/debug
oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crypt/zone                                                            286M  2.53T      100K  /pool/ext/9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crypt/zone
oxp_9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crypt/zone/oxz_crucible_e8f55a5d-65f9-436c-bc25-1d1a7070e876          286M  2.53T      286M  /pool/ext/9bfe385c-16dd-4209-bc0b-f28ae75d58e3/crypt/zone/oxz_crucible_e8f55a5d-65f9-436c-bc25-1d1a7070e876
oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b                                                                       156G  2.66T       24K  /oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b
oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crucible                                                              153G  2.66T     79.5K  /data
oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crucible/regions                                                      153G  2.66T       31K  /data/regions
oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crucible/regions/395f1069-9d5a-4eb2-9a7f-7b1d0247fff6                13.9G  76.1G     13.9G  /data/regions/395f1069-9d5a-4eb2-9a7f-7b1d0247fff6
oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crucible/regions/3fa289ea-54ef-43d3-b926-5092145becfe                1.56G  28.4G     1.56G  /data/regions/3fa289ea-54ef-43d3-b926-5092145becfe
oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crucible/regions/516d126e-8439-4c88-b0fe-4f4c12f23e62                2.76G  27.2G     2.76G  /data/regions/516d126e-8439-4c88-b0fe-4f4c12f23e62
oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crucible/regions/8a5d5358-20f1-4e01-b31d-21973d9fbe79                19.4G  40.6G     19.4G  /data/regions/8a5d5358-20f1-4e01-b31d-21973d9fbe79
oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crucible/regions/9257f7af-3332-4913-a013-548c02cc0f05                 194M  2.81G      194M  /data/regions/9257f7af-3332-4913-a013-548c02cc0f05
oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crucible/regions/ba5562b0-b4bc-4184-8dfd-c332136fb7f6                4.74G   115G     4.74G  /data/regions/ba5562b0-b4bc-4184-8dfd-c332136fb7f6
oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crucible/regions/d0274aef-fba0-4c60-a7bc-2929f4271b84                 652M  29.4G      652M  /data/regions/d0274aef-fba0-4c60-a7bc-2929f4271b84
oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crucible/regions/f9e6aa4d-b5e8-4e0d-bacd-ae0a0a6c48ce                1.10G  1.90G     1.10G  /data/regions/f9e6aa4d-b5e8-4e0d-bacd-ae0a0a6c48ce
oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crypt                                                                3.27G  2.66T     2.99G  /pool/ext/a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crypt
oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crypt/debug                                                            99K   100G       99K  /pool/ext/a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crypt/debug
oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crypt/zone                                                            286M  2.66T      100K  /pool/ext/a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crypt/zone
oxp_a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crypt/zone/oxz_crucible_ceeba69d-8c0a-47df-a37b-7f1b90f23016          286M  2.66T      286M  /pool/ext/a624a843-1c4e-41c3-a1d2-4be7a6c57e9b/crypt/zone/oxz_crucible_ceeba69d-8c0a-47df-a37b-7f1b90f23016
oxp_ab62b941-5f84-42c7-929d-295b20efffe7                                                                       201G  2.62T       24K  /oxp_ab62b941-5f84-42c7-929d-295b20efffe7
oxp_ab62b941-5f84-42c7-929d-295b20efffe7/crucible                                                              193G  2.62T     80.5K  /data
oxp_ab62b941-5f84-42c7-929d-295b20efffe7/crucible/regions                                                      193G  2.62T       28K  /data/regions
oxp_ab62b941-5f84-42c7-929d-295b20efffe7/crucible/regions/0f89a227-2612-42be-8fc8-ce01530379c1                2.46G  27.5G     2.46G  /data/regions/0f89a227-2612-42be-8fc8-ce01530379c1
oxp_ab62b941-5f84-42c7-929d-295b20efffe7/crucible/regions/20dd9cf4-a581-4ddd-9f2d-55788bbc9b5e                2.75G  27.2G     2.75G  /data/regions/20dd9cf4-a581-4ddd-9f2d-55788bbc9b5e
oxp_ab62b941-5f84-42c7-929d-295b20efffe7/crucible/regions/4ef467e7-7896-4a24-ba51-560a079b7d27                8.17M  12.0G     8.17M  /data/regions/4ef467e7-7896-4a24-ba51-560a079b7d27
oxp_ab62b941-5f84-42c7-929d-295b20efffe7/crucible/regions/5f751346-7d7f-40be-b76b-829ed364d8f2                1.51G  4.49G     1.51G  /data/regions/5f751346-7d7f-40be-b76b-829ed364d8f2
oxp_ab62b941-5f84-42c7-929d-295b20efffe7/crucible/regions/9c2fdd85-e8a7-4ead-8b48-b29f47fd9878                22.3G  37.7G     22.3G  /data/regions/9c2fdd85-e8a7-4ead-8b48-b29f47fd9878
oxp_ab62b941-5f84-42c7-929d-295b20efffe7/crucible/regions/aeeb1141-c4b1-4be7-9139-6f88811f44d0                10.5G   289G     10.5G  /data/regions/aeeb1141-c4b1-4be7-9139-6f88811f44d0
oxp_ab62b941-5f84-42c7-929d-295b20efffe7/crucible/regions/e30fa317-b4b7-4e9e-ae96-3933907f6cd1                1.06G  22.9G     1.06G  /data/regions/e30fa317-b4b7-4e9e-ae96-3933907f6cd1
oxp_ab62b941-5f84-42c7-929d-295b20efffe7/crypt                                                                8.90G  2.62T     8.32G  /pool/ext/ab62b941-5f84-42c7-929d-295b20efffe7/crypt
oxp_ab62b941-5f84-42c7-929d-295b20efffe7/crypt/debug                                                            99K   100G       99K  /pool/ext/ab62b941-5f84-42c7-929d-295b20efffe7/crypt/debug
oxp_ab62b941-5f84-42c7-929d-295b20efffe7/crypt/zone                                                            593M  2.62T      175K  /pool/ext/ab62b941-5f84-42c7-929d-295b20efffe7/crypt/zone
oxp_ab62b941-5f84-42c7-929d-295b20efffe7/crypt/zone/oxz_crucible_a014f12e-2636-4258-af76-e01d9b8d1c1f          286M  2.62T      286M  /pool/ext/ab62b941-5f84-42c7-929d-295b20efffe7/crypt/zone/oxz_crucible_a014f12e-2636-4258-af76-e01d9b8d1c1f
oxp_ab62b941-5f84-42c7-929d-295b20efffe7/crypt/zone/oxz_propolis-server_a6d03163-0f7a-4614-886d-d341434c28ab   307M  2.62T      307M  /pool/ext/ab62b941-5f84-42c7-929d-295b20efffe7/crypt/zone/oxz_propolis-server_a6d03163-0f7a-4614-886d-d341434c28ab
oxp_ab62b941-5f84-42c7-929d-295b20efffe7/crypt/zone/oxz_propolis-server_a8330d94-abfa-4570-ae33-424dc8d22461   136K  2.62T      136K  /pool/ext/ab62b941-5f84-42c7-929d-295b20efffe7/crypt/zone/oxz_propolis-server_a8330d94-abfa-4570-ae33-424dc8d22461
oxp_dbfdc981-1b81-4d7d-9449-9530890b199a                                                                      16.0G  2.80T       96K  /oxp_dbfdc981-1b81-4d7d-9449-9530890b199a
oxp_dbfdc981-1b81-4d7d-9449-9530890b199a/crucible                                                              192K  2.80T       96K  /data
oxp_dbfdc981-1b81-4d7d-9449-9530890b199a/crucible/regions                                                       96K  2.80T       96K  /data/regions
oxp_dbfdc981-1b81-4d7d-9449-9530890b199a/crypt                                                                16.0G  2.80T     15.7G  /pool/ext/dbfdc981-1b81-4d7d-9449-9530890b199a/crypt
oxp_dbfdc981-1b81-4d7d-9449-9530890b199a/crypt/debug                                                           200K   100G      200K  /pool/ext/dbfdc981-1b81-4d7d-9449-9530890b199a/crypt/debug
oxp_dbfdc981-1b81-4d7d-9449-9530890b199a/crypt/zone                                                            323M  2.80T      208K  /pool/ext/dbfdc981-1b81-4d7d-9449-9530890b199a/crypt/zone
oxp_dbfdc981-1b81-4d7d-9449-9530890b199a/crypt/zone/oxz_crucible_9a5a2fcf-44a0-4468-979a-a71686cef627          323M  2.80T      323M  /pool/ext/dbfdc981-1b81-4d7d-9449-9530890b199a/crypt/zone/oxz_crucible_9a5a2fcf-44a0-4468-979a-a71686cef627
rpool                                                                                                          701M  3.13G       24K  /rpool
rpool/ROOT                                                                                                     691M  3.13G       24K  legacy
rpool/ROOT/ramdisk                                                                                             691M  3.13G      691M  legacy
rpool/zone

From within the crucible zone

root@oxz_crucible_9a5a2fcf:~# df -h
Filesystem             Size   Used  Available Capacity  Mounted on
oxp_dbfdc981-1b81-4d7d-9449-9530890b199a/crypt/zone/oxz_crucible_9a5a2fcf-44a0-4468-979a-a71686cef627
                      2.80T 322.55M      2.80T     1%    /
/dev                      0      0          0     0%    /dev
/system/lib           3.80G 690.84M      3.13G    18%    /system/lib
/system/sbin          3.80G 690.84M      3.13G    18%    /system/sbin
/system/usr           3.80G 690.84M      3.13G    18%    /system/usr
proc                      0      0          0     0%    /proc
ctfs                      0      0          0     0%    /system/contract
mnttab                    0      0          0     0%    /etc/mnttab
objfs                     0      0          0     0%    /system/object
swap                 320.66G   220K    320.66G     1%    /etc/svc/volatile
sharefs                   0      0          0     0%    /etc/dfs/sharetab
/usr/lib/libc/libc_hwcap2.so.1
                      3.80G 690.84M      3.13G    18%    /system/lib/libc.so.1
fd                        0      0          0     0%    /dev/fd
swap                 320.66G      0    320.66G     0%    /tmp
swap                 320.66G     8K    320.66G     1%    /var/run
oxp_dbfdc981-1b81-4d7d-9449-9530890b199a/crucible
                      2.82T    96K      2.80T     1%    /data
oxp_dbfdc981-1b81-4d7d-9449-9530890b199a/crucible/regions
                      2.82T    96K      2.80T     1%    /data/regions

root@oxz_crucible_9a5a2fcf:~# ls -ltra /data
total 19
drwxr-xr-x   2 root     root           2 Jul 11 19:32 regions
drwxr-xr-x   3 root     root           3 Jul 11 19:32 .
drwxr-xr-x  17 root     root          18 Oct 18 17:49 ..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@leftwo @askfongjojo