BSOD when deleting a mointpoint folder in Explorer and zpool export (fixed in 2.3.0 rc6f)

[guenther-alka]

System information

Type | Version/Name

• -- | ---
  Distribution Name | Win 11 Pro 23/02
  Distribution Version |
  Kernel Version |
  Architecture |
  OpenZFS Version | 2.3.0 rc3

Describe the problem you're observing

zfs list
NAME          USED  AVAIL  REFER  MOUNTPOINT
usb          1.30M  13.6G   228K  /usb
usb/data      332K  13.6G   184K  /usb/data
usb/data/ds   148K  13.6G   148K  /usb/data/ds

In Explorer you can delete data/ds (nested ZFS filesystem).
To make folder data/ds visible again you can do a zpool export + import
but zpool export usb gives a BSOD

memory dump
https://drive.google.com/file/d/1gZdPfuTjP4iP_-BUgilBakdNab4pZHTv/view?usp=sharing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

[guenther-alka]

updated to rc4

delete a subolder in Explorer that is a zfs mountpint

zpool export pool: BSOD

[lundman]

Yeah this one is pretty straight forward;

stack

07 fffffa06`78509fc0 fffff803`4c801181     nt!KiBreakpointTrap+0x34e
08 fffffa06`7850a150 fffff803`4c8318ff     OpenZFS!spl_panic+0x71 [C:\src\openzfs\module\os\windows\spl\spl-err.c @ 85] 
09 fffffa06`7850a1b0 fffff803`4c83262d     OpenZFS!spl_assert+0x3f [C:\src\openzfs\include\os\windows\spl\sys\debug.h @ 172] 
0a fffffa06`7850a200 fffff803`4cb5e0e7     OpenZFS!vnode_rele+0x5d [C:\src\openzfs\module\os\windows\spl\spl-vnode.c @ 1405] 
0b fffffa06`7850a250 fffff803`4cb62f66     OpenZFS!zfs_fileobject_cleanup+0x357 [C:\src\openzfs\module\os\windows\zfs\zfs_vnops_windows.c @ 6443] 
0c fffffa06`7850a360 fffff803`4cb5f05d     OpenZFS!fsDispatcher+0x5d6 [C:\src\openzfs\module\os\windows\zfs\zfs_vnops_windows.c @ 7666] 
0d fffffa06`7850a4e0 fffff803`40c650c5     OpenZFS!dispatcher+0x37d [C:\src\openzfs\module\os\windows\zfs\zfs_vnops_windows.c @ 8076] 
0e fffffa06`7850a610 fffff803`3fb0a1fb     nt!IofCallDriver+0x55

The vnode

Local var @ 0xfffffa067850a240 Type vnode*
0xffff8006`b1e172e8 
   +0x000 FileHeader       : _FSRTL_ADVANCED_FCB_HEADER
   +0x058 AdvancedFcbHeaderMutex : _FAST_MUTEX
   +0x090 SectionObjectPointers : _SECTION_OBJECT_POINTERS
   +0x0a8 oplock           : (null) 
   +0x0b0 v_mutex          : kmutex
   +0x0e0 v_mount          : 0xffff8006`c5b34db0 mount
   +0x0e8 v_parent         : (null) 
   +0x0f0 v_data           : (null) 
   +0x0f8 v_reparse        : (null) 
   +0x100 security_descriptor : (null) 
   +0x108 v_flags          : 0x23
   +0x10c v_iocount        : 0
   +0x110 v_usecount       : 3
   +0x114 v_type           : 2
   +0x118 v_unlink         : 0
   +0x11c v_unused         : 0
   +0x120 v_reparse_size   : 0
   +0x128 v_id             : 0x96
   +0x130 v_easize         : 0
   +0x138 v_age            : 0n292893663700
   +0x140 resource         : _ERESOURCE
   +0x1a8 pageio_resource  : _ERESOURCE
   +0x210 lock             : _FILE_LOCK
   +0x270 share_access     : _SHARE_ACCESS
   +0x290 v_list           : list_node
   +0x2a0 v_fileobjects    : avl_tree

From the log

FFFF8006C09F5080: dprintf: zfs_vnops_windows_mount.c:1905:zfs_windows_unmount():
 zfs_windows_unmount: calling zfs_vfs_unmount
FFFF8006C09F5080: dprintf: zfs_vfsops.c:1591:zfs_vfs_unmount(): zfs_vfs_unmount

same time later

FFFF8006C09F5080: vflush start
FFFF8006C09F5080: already marked
FFFF8006C09F5080: vnode_recycle_int: marking FFFF8006B200E380 VNODE_MARKTERM
FFFF8006C09F5080: dprintf: zfs_vnops_windows.c:2028:zfs_vnop_reclaim():   zfs_vnop_recycle: releasing zp FFFF8006B18B1540 and vp FFFF8006B200E380
FFFF8006C09F5080: vflush start
FFFF8006C09F5080: already marked
FFFF8006C09F5080: FFFF8006B1E172E8 marked DEAD
FFFF8006C09F5080: Dropping 2 references
FFFF8006C09F5080: vp FFFF8006B1E172E8 left on DEAD list
FFFF8006C09F5080: already marked
FFFF8006C09F5080: FFFF8006B200E380 marked DEAD
FFFF8006C09F5080: vp FFFF8006B200E380 left on DEAD list
FFFF8006C09F5080: vflush: 2 reclaims processed.
```

Basically, unmount request comes in, and we know we are leaking references and really the unmount 
should stall here. The 2 leaks are inside the reparse point that was deleted. The vnode is marked DEAD.

Then the system goes and frees those open references to the reparse point, which includes decreasing 
usecount via vnode_rele(). This is asserted, since we shouldn't be decreasing a DEAD node. 

Quick fix might be to stop the deletion of reparsepoints for now, and discuss the ethics of that later.

[lundman]

f95034e

[guenther-alka]

fixed in rc5

[guenther-alka]

Problem not fixed

usb                                                   mountpoint  /usb               default
usb/data                                              mountpoint  legacy             local
usb/data/ds                                           mountpoint  /usb/data/ds       local
usb/data/ds/tank                                      mountpoint  /usb/data/ds/tank  inherited from usb/data/ds

After playing a little with unmount/mount I could suddenly again delete a subfolder ds in Explorer that is a ZFS mountpoint.

I then tried a pool export (resulting in a BSOD)
After reboot the folder/mointpoint is visible again but a zpool export again results in a BSOD

I then tried to switch mountpoint to legacy and back but zfs set mountpoint=legacy usb/data/ds also ended in a BSOD

The former workaround to set all mountpoints to legacy no longer helps as every zpool export ends in a BSOD

memory.dmp
https://drive.google.com/file/d/1FPoyYAVRwXVf1wchjIS5Ap4fSVuexK8g/view?usp=sharing

[guenther-alka]

weird
I am now in a situation that zpool destroy usb or zfs destroy dataset leads to a "busy, cannot destroy" or a BSOD

zfs list
NAME               USED  AVAIL  REFER  MOUNTPOINT
usb               1.02G  12.5G   188K  /usb
usb/data           576K  12.5G   208K  /usb/data
usb/data/ds        368K  12.5G   172K  /usb/data/ds
usb/data/ds/tank   196K  12.5G   196K  /usb/data/ds/tank
usb/volest        1.02G  13.6G    56K  -
``

[guenther-alka]

seems a more general problem
On my first try with create pool + create dataset + destroy pool I got a BSOD
Now I get busy

zpool create test physicaldrive1
Expanded path to '\\?\physicaldrive1'
working on dev '#1048576#15491661824#\\?\physicaldrive1'
setting path here '/dev/physicaldrive1'
setting physpath here '#1048576#15491661824#\\?\physicaldrive1'
PS C:\Users\me> zfs create test/data
PS C:\Users\me> zfs create test/data/1
PS C:\Users\me> zfs destroy -r test/data
zunmount(test/data/1,D:/data/1/) running
zunmount(test/data/1,D:/data/1/) returns 0
cannot destroy 'test/data/1': dataset is busy
zunmount(test/data,D:/data/) running
zunmount(test/data,D:/data/) returns 0
cannot destroy 'test/data': dataset is busy

[lundman]

Hmm maybe the dont-delete-reparse-point is too mean now

[lundman]

Ah heh

	if (vp->v_usecount == 0)
		dprintf("BUG vp %p already at 0\n", vp);
	else
		vnode_rele(vp); // Release longterm hold finally.

great, except dprintf is a macro starting with if (.... so that else goes to the wrong thing, and nothing was ever released.