PROPFIND and glob fix

examples/pytorch/pytorch_with_pelicanfs.ipynb

@@ -0,0 +1,100 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Pytorch And PelicanFS\n",
+    "\n",
+    "The following is an example of how to load in data from a `Pelican` origin for `PyTorch`\n",
+    "\n",
+    "## Installation\n",
+    "\n",
+    "Ensure both `FSSpec` and `PelicanFS` are installed in the environment. Under the hood, the IterableWrapper is using an FSSpec backend to load in data and the installation of PelicanFS registers the `osdf` protocol with FSSpec."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import torch\n",
+    "torch.utils.data.datapipes.utils.common.DILL_AVAILABLE = torch.utils._import_utils.dill_available()\n",
+    "from torchdata.datapipes.iter import IterableWrapper"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example of list_files_by_fsspec()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "['osdf:///chtc/PUBLIC/eturetsky/data/faces/0805personali01.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/1084239450_e76e00b7e7.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/10comm-decarlo.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/110276240_bec305da91.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/1198_0_861.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/137341995_e7c48e9a75.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/1383023626_8a49e4879a.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/144044282_87cf3ff76e.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/152601997_ec6429a43c.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/1549040388_b99e9fa295.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/1878519279_f905d4f34e.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/2046713398_91aaa6fe1c.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/2173711035_dbd53b4f9f.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/2210514040_6b03ff2629.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/2322901504_08122b01ba.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/2327253037_66a61ea6fe.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/2328398005_d328a70b4c.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/2370961440_6bc8ce346c.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/2382SJ8.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/252418361_440b75751b.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/262007783_943bbcf613.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/2633371780_45b740b670.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/2647088981_60e9fe40cd.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/2711409561_a0786a3d3d.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/2722779845_7fcb64a096.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/2795838930_0cc5aa5f41.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/2902323565_100017b63c.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/2902760364_89c50bde40.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/2956581526_cd803f2daa.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/297448785_b2dda4b2c0.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/299733036_fff5ea6f8e.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/303808204_1f744bc407.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3074791551_baee7fa0c1.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3152653555_68322314f3.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3264867945_fe18d442c1.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3273658251_b95f65c244.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3298715079_5af7c78fcb.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3325611505_ddc7beffa1.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3362762930_24f76cb89c.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/343583208_e986824d77.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3461016494_56cce9c984.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/348272697_832ce65324.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3534188114_2108895291.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3534189272_8ef88ba368.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3555944509_7b477069c6.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3574737496_6ee8207045.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/362167809_d5a5dcbfdb.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/363149951_8be04dc6c0.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3638950581_3387685d3a.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3646828311_bfeb429ef7.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3689162471_5f9ffb5aa0.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3718903026_c1bf5dfcf8.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3790616528_297c0ac935.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3855944735_e252959937.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3856149136_d4595ffdd4.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/3872768751_e60d7fdbd5.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/529447797_0f9d2fb756.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/57635685_d41c98f8ca.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/809285949_6889026b53.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/92053278_be61a225d2.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/96063776_bdb3617b64.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/97308305_4b737d0873.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/britney-bald.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/create_landmark_dataset.py', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/deeny.peggy.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/face_landmarks.csv', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/matt-mathes.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/person-7.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/person.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/person_TjahjonoDGondhowiardjo.jpg', 'osdf:///chtc/PUBLIC/eturetsky/data/faces/personalpic.jpg']\n"
+     ]
+    }
+   ],
+   "source": [
+    "dp = IterableWrapper([\"osdf:///chtc/PUBLIC/eturetsky/data/faces/\"]).list_files_by_fsspec()\n",
+    "print(list(dp))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example of open_files_by_fsspec()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "osdf:///chtc/PUBLIC/eturetsky/data/faces/0805personali01.jpg StreamWrapper<<_io.TextIOWrapper encoding='UTF-8'>>\n"
+     ]
+    }
+   ],
+   "source": [
+    "dp = IterableWrapper([\"osdf:///chtc/PUBLIC/eturetsky/data/faces/0805personali01.jpg\"]).open_files_by_fsspec()\n",
+    "for path, filestream in dp:\n",
+    "    print(path, filestream)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}