WebDAV interface for filesystem

[KernelDeimos]

WebDAV interface for filesystem

A WebDAV interface for accessing Puter's filesystem would allow Puter files to be accessed in any environment where a WebDAV driver is available. This would significantly increase the use-cases for Puter.

Puter's Filesystem

• The puter.js SDK (on the client) has a Filesystem module where you access the filesystem (ex: puter.fs.write)
• filesystem endpoints are then requested by puter.js. These endpoints typically access high-level filesystem operations
• high-level filesystem operations implement a lot of behavior. For example, recursive mkdir and delete are implemented here.
• low-level filesystem operations are meant to be the minimum required behavior for a filesystem. In preparation for the upcoming "mountpoints" feature, these operations delegate most of the behavior to a filesystem provider specified by the instance of...
• FSNodeContext; An instance of FSNodeContext represents a file or directory in the backend.
• PuterFSProvider is where Puter-filesystem-specific behavior is being moved to. LL operations access a filesystem provider on the node (FSNodeContext), then call a function on the provider. Right now PuterFSProvider is the only provider

The items above are roughly in order from (closest to client) to (closest to storage). WebDAV should be implemented just above "low-level filesystem operations".

Client <-> WebDAV interface <-> LL Operations

The WebDAV interface does not need puter.js support.

[dawit2123]

Hey @KernelDeimos I would like to take up this issue. Could you assign me to this issue, please?

[KernelDeimos]

Assigned! Let me know if you run into any hurdles

[HAIBALLA1]

Hey @KernelDeimos , It's Haiballa from Headstarter could you assign me to this issue, please?

[KernelDeimos]

Hello @HAIBALLA1 , are you working on this with @dawit2123 ? They are currently assigned to this issue

[dawit2123]

@HAIBALLA1 @KernelDeimos I wanted to let you know that I’ve been assigned to this issue and have already started working on it.

[KernelDeimos]

@dawit2123 You mentioned a week ago that you started working on this, how far along is it? Is there anything I can do to help?

[dawit2123]

@KernelDeimos I have an issue with making a WebDev driver interface. I used a WebDAV server to make the WebDAV, and then the authentication part should be done in the PUTer JS.

So my approach was validateUser will validate the user if it appears, and if not, it will return unauthenticated. And I used PuterFSProvider and FSNodeContext to build it on top of the low-level filesystem.

Then I exported startWebDAVServer and called it in the run-selfhosted.js.

The validateUser method works fine with authentication, but even if the user is found, it's returning 404. Unauthorized when I access it using PROPFIND in Postman using the URL http://localhost:1900/webdav and with the correct username and password.

I haven't used the authentication in webda-server because it should be dependable on the computer, but I am getting unauthorized, and I have tried so many approaches.

Do you see it and recommend to me what I should do and also if there's any other package that I should use to do it?

The code is attached down below:

webdav-server.js code is down below

const webdav = require('webdav-server').v2;
const bcrypt = require('bcrypt');
const express = require('express');
const { FSNodeContext } = require('../src/filesystem/FSNodeContext.js');
const { PuterFSProvider } = require('../src/modules/puterfs/lib/PuterFSProvider.js');
const { get_user } = require('../src/helpers');
const path = require('path');
  
class PuterFileSystem extends webdav.FileSystem {
    constructor(fsProvider, user, Context) {
        super("puter", { uid: 1, gid: 1 });
        this.fsProvider = fsProvider;
        this.user = user;
        this.Context = Context;
    }
  
    _getPath(filePath) {
        return path.normalize(filePath);
    }
  
    async _getFSNode(filePath) {
        const normalizedPath = this._getPath(filePath);
        return new FSNodeContext({
            services: this.Context.get('services'),
            selector: { path: normalizedPath },
            provider: this.fsProvider,
            fs: { node: this._getFSNode }
        });
    }
  
    // Implement required WebDAV methods
    async _openReadStream(ctx, filePath, callback) {
        try {
            const node = await this._getFSNode(filePath);
            const stream = await node.read();
            callback(null, stream);
        } catch (e) {
            callback(e);
        }
    }
  
    async _openWriteStream(ctx, filePath, callback) {
        try {
            const node = await this._getFSNode(filePath);
            const stream = await node.write();
            callback(null, stream);
        } catch (e) {
            callback(e);
        }
    }
  
    async _create(ctx, filePath, type, callback) {
        try {
            const node = await this._getFSNode(filePath);
            if (type === webdav.ResourceType.Directory) {
                await node.mkdir();
            } else {
                await node.create();
            }
            callback();
        } catch (e) {
            callback(e);
        }
    }
  
    async _delete(ctx, filePath, callback) {
        try {
            const node = await this._getFSNode(filePath);
            await node.delete();
            callback();
        } catch (e) {
            callback(e);
        }
    }
  
    // Implement other required methods (size, lastModifiedDate, etc.)
    async _size(ctx, filePath, callback) {
        try {
            const node = await this._getFSNode(filePath);
            const size = await node.size();
            callback(null, size);
        } catch (e) {
            callback(e);
        }
    }
}
  
async function validateUser(username, password, Context) {
    try {
        const services = Context.get('services');
  
        // Fetch user from Puter's authentication service
        const user = await get_user({ username, cached: false });
        if (!user) {
            console.log(`Authentication failed: User '${username}' not found.`);
            return null;
        }
  
        // Validate password with bcrypt
        const isMatch = await bcrypt.compare(password, user.password);
        if (!isMatch) {
            console.log(`Authentication failed: Incorrect password.`);
            return null;
        }
  
        console.log(`Authentication successful for user: ${username}`);
        return user;
    } catch (error) {
        console.error('Error during authentication:', error);
        return null;
    }
}
  
async function startWebDAVServer(port, Context) {
    const app = express();

    // Initialize Puter filesystem components
    const services = Context.get('services');
    const fsProvider = new PuterFSProvider(services);
    const puterFS = new PuterFileSystem(fsProvider, null, Context);
  
    const server = new webdav.WebDAVServer({
        port: port,
        autoSave: false,
        rootFileSystem: puterFS  // Use Puter filesystem as root
    });
  
    // Authentication middleware
    app.use(async (req, res, next) => {
        const authHeader = req.headers.authorization;
  
        const credentials = Buffer.from(authHeader.split(' ')[1], 'base64').toString();
        const [username, password] = credentials.split(':');
  
        try {
            const user = await validateUser(username, password, Context);
            if (!user) return res.status(401).send('Invalid credentials');
            req.user = user;
            next();
        } catch (error) {
            console.error('Authentication error:', error);
            res.status(500).send('Internal server error');
        }
    });

    // Mount WebDAV server
    app.use(webdav.extensions.express('/webdav', server));

    // Start server
    app.listen(port, () => {
        console.log(`Puter WebDAV server running on port ${port}`);
        console.log(`Access via: http://puter.localhost:${port}/webdav`);
    });
  
    return server;
}

module.exports = {
    startWebDAVServer
};

run-selfhosted.js code that I have changed is down below

import {startWebDAVServer} from '../src/backend/webdav/webdav-server.js';
const main = async () => {
    const {
        Kernel,
        EssentialModules,
        DatabaseModule,
        LocalDiskStorageModule,
        SelfHostedModule,
        BroadcastModule,
        TestDriversModule,
        PuterAIModule,
        PuterExecModule,
        InternetModule,
        MailModule,
        ConvertModule,
        DevelopmentModule,
    } = (await import('@heyputer/backend')).default;

    const k = new Kernel({
        entry_path: import.meta.filename
    });

    for ( const mod of EssentialModules ) {
        k.add_module(new mod());
    }

    k.add_module(new DatabaseModule());
    k.add_module(new LocalDiskStorageModule());
    k.add_module(new SelfHostedModule());
    k.add_module(new BroadcastModule());
    k.add_module(new TestDriversModule());
    k.add_module(new PuterAIModule());
    k.add_module(new PuterExecModule());
    k.add_module(new InternetModule());
    k.add_module(new MailModule());
    k.add_module(new ConvertModule());
    if ( process.env.UNSAFE_PUTER_DEV ) {
        k.add_module(new DevelopmentModule());
    }
    k.boot();
    const webdavContext = {
        get: (serviceName) => {
            // Special case to get the services container itself
            if (serviceName === 'services') return k.services;
            // Normal service resolution
            return k.services.get(serviceName, { optional: true });
        }
    };
    startWebDAVServer(1900, webdavContext);
};

Edited by @KernelDeimos for readability

[KernelDeimos]

Hi, I found this is a bit difficult to figure out but so far it seems like your authentication logic is perfectly fine - I do get the "Authentication successful for user" message in the console but the response is always 401 Unauthorized. After some debugging, I think what's happening is the instance of webdav.WebDAVServer thinks its supposed to handle authentication if it sees the http basic auth token.

2025-03-14_15-04-38.mp4

It turns out we can remove the header before passing along to the next middleware. I get another error after, but this looks like progress.

[dawit2123]

@KernelDeimos
I have solved the issue of the error that you've got. When I try the propfind command, there's an error that pops up from the file that's already done which is FSNodeContext that's: [INFO::fsnode-context] (571.757s) fetching entry: C:
WebDAV operation error: Error: FAILED TO GET THE CORRECT CONTEXT

The code that I have updated is down below>>>
`const webdav = require('webdav-server').v2;
const bcrypt = require('bcrypt');
const express = require('express');
const FSNodeContext = require('../src/filesystem/FSNodeContext.js');
const { PuterFSProvider } = require('../src/modules/puterfs/lib/PuterFSProvider.js');
const { get_user } = require('../src/helpers');
const path = require('path');
const APIError = require('../src/api/APIError.js');
const { NodePathSelector } = require('../src/filesystem/node/selectors'); // Import NodePathSelector

class PuterFileSystem extends webdav.FileSystem {
constructor(fsProvider, Context) {
/**

• Initializes a new instance of the PuterFileSystem class.

• @param {PuterFSProvider} fsProvider - The file system provider instance.

• @param {Context} Context - The context containing configuration and services.
  */

   super("puter", { uid: 1, gid: 1 });
   this.fsProvider = fsProvider;
   this.Context = Context;
   this.services = Context.get('services');
  

  }

  _getPath(filePath) {
  try {
  if (typeof filePath !== 'string') {
  filePath = filePath.toString();
  }
  return path.resolve('/', filePath).replace(/../g, '');
  } catch (e) {
  console.error("error in _getPath", e);
  throw e;
  }
  }

  async getFSNode(filePath) {
  const normalizedPath = this._getPath(filePath);
  return new FSNodeContext({
  services: this.services,
  selector: new NodePathSelector(normalizedPath), // Use NodePathSelector instance
  provider: this.fsProvider,
  fs: this.services.get('filesystem')
  });
  }

  async _type(ctx, filePath, callback) {
  try {
  const node = await this.getFSNode(filePath);
  const exists = await node.exists();
  if (!exists) {
  return callback(webdav.Errors.ResourceNotFound);
  }
  const isDir = await node.get('is_dir');
  callback(null, isDir ? webdav.ResourceType.Directory : webdav.ResourceType.File);
  } catch (e) {
  this._mapError(e, callback, '_type');
  }
  }

  async _exist(ctx, filePath, callback) {
  try {
  const node = await this.getFSNode(filePath);
  const exists = await node.exists();
  callback(null, exists);
  } catch (e) {
  this._mapError(e, callback, '_exist');
  }
  }

  async _openReadStream(ctx, filePath, callback) {
  try {
  const node = await this.getFSNode(filePath);
  if (await node.get('is_dir')) {
  return callback(webdav.Errors.IsADirectory);
  }
  const content = await this.services.get('filesystem').read(node);
  callback(null, content);
  } catch (e) {
  this._mapError(e, callback, '_openReadStream');
  }
  }

  async _openWriteStream(ctx, filePath, callback) {
  try {
  const node = await this.getFSNode(filePath);
  const parentPath = path.dirname(filePath);
  const parentNode = await this.getFSNode(parentPath);

       return callback(null, {
           write: async (content) => {
               await this.services.get('filesystem').write(node, content, {
                   parent: parentNode,
                   name: path.basename(filePath)
               });
           },
           end: callback
       });
   } catch (e) {
       this._mapError(e, callback, '_openWriteStream');
   }
  

  }

  async _create(ctx, filePath, type, callback) {
  try {
  console.log('Create operation is called for:', filePath);
  const parentPath = path.dirname(filePath);
  const name = path.basename(filePath);
  const parentNode = await this.getFSNode(parentPath);
  if (type === webdav.ResourceType.Directory) {
  console.log('making directory: ', name);
  await this.services.get('filesystem').mkdir(parentNode, name);
  } else {
  await this.services.get('filesystem').write(
  { path: filePath },
  Buffer.alloc(0),
  { parent: parentNode, name }
  );
  }
  callback();
  } catch (e) {
  this._mapError(e, callback, '_create');
  }
  }

  async _delete(ctx, filePath, callback) {
  try {
  const node = await this.getFSNode(filePath);
  if (await node.get('is_dir')) {
  await this.services.get('filesystem').rmdir(node);
  } else {
  await this.services.get('filesystem').unlink(node);
  }
  callback();
  } catch (e) {
  this._mapError(e, callback, '_delete');
  }
  }

  async _size(ctx, filePath, callback) {
  try {
  const node = await this.getFSNode(filePath);
  const size = await node.get('size');
  callback(null, size || 0);
  } catch (e) {
  this._mapError(e, callback, '_size');
  }
  }

  async _lastModifiedDate(ctx, filePath, callback) {
  try {
  const node = await this.getFSNode(filePath);
  const modified = await node.get('modified');
  callback(null, modified ? new Date(modified * 1000) : new Date());
  } catch (e) {
  this._mapError(e, callback, '_lastModifiedDate');
  }
  }

  async _move(ctx, srcPath, destPath, callback) {
  try {
  const srcNode = await this.getFSNode(srcPath);
  const destParent = await this.getFSNode(path.dirname(destPath));
  await this.services.get('filesystem').move(
  srcNode,
  destParent,
  path.basename(destPath)
  );
  callback();
  } catch (e) {
  this._mapError(e, callback, '_move');
  }
  }

  async _copy(ctx, srcPath, destPath, callback) {
  try {
  const srcNode = await this.getFSNode(srcPath);
  const destParent = await this.getFSNode(path.dirname(destPath));
  await this.services.get('filesystem').copy(
  srcNode,
  destParent,
  path.basename(destPath)
  );
  callback();
  } catch (e) {
  this._mapError(e, callback, '_copy');
  }
  }

  async _propertyManager(ctx, filePath, callback) {
  callback(null, {
  getProperties: async (name, callback) => {
  try {
  const node = await this.getFSNode(filePath);
  const entry = await node.fetchEntry();
  callback(null, {
  displayname: entry.name,
  getlastmodified: new Date(entry.modified * 1000).toUTCString(),
  getcontentlength: entry.size || '0',
  resourcetype: entry.is_dir ? ['collection'] : [],
  getcontenttype: entry.mime_type || 'application/octet-stream'
  });
  } catch (e) {
  this._mapError(e, callback, '_propertyManager');
  }
  }
  });
  }

  _mapError(e, callback, methodName) {
  console.error('WebDAV operation error:', e);
  if (e instanceof APIError) {
  switch (e.code) {
  case 'not_found': return callback(webdav.Errors.ResourceNotFound);
  case 'item_with_same_name_exists': return callback(webdav.Errors.InvalidOperation);
  case 'not_empty': return callback(webdav.Errors.Forbidden);
  default: return callback(webdav.Errors.InternalError);
  }
  }
  if (e instanceof TypeError && e.message.includes('Cannot read properties of undefined (reading 'isDirectory')')) {
  return callback(webdav.Errors.InternalServerError);
  }
  return callback(e);
  }
  }

async function validateUser(username, password, Context) {
try {
const services = Context.get('services');

    // Fetch user from Puter's authentication service
    const user = await get_user({ username, cached: false });
    if (!user) {
        console.log(`Authentication failed: User '${username}' not found.`);
        return null;
    }

    // Validate password with bcrypt
    const isMatch = await bcrypt.compare(password, user.password);

    if (!isMatch) {
        console.log(`Authentication failed: Incorrect password.`);
        return null;
    }

    console.log(`Authentication successful for user: ${username}`);
    return user;
} catch (error) {
    console.error('Error during authentication:', error);
    return null;
}

}

async function startWebDAVServer(port, Context) {
const app = express();
const fsProvider = new PuterFSProvider(Context.get('services'));
const puterFS = new PuterFileSystem(fsProvider, Context);

const server = new webdav.WebDAVServer({
    rootFileSystem: puterFS,
    autoSave: false,
    strictMode: false
});

// Add the missing functions to the PuterFileSystem prototype
PuterFileSystem.prototype.type = PuterFileSystem.prototype._type;
PuterFileSystem.prototype.exist = PuterFileSystem.prototype._exist;
PuterFileSystem.prototype.create = PuterFileSystem.prototype._create;
PuterFileSystem.prototype.delete = PuterFileSystem.prototype._delete;
PuterFileSystem.prototype.openReadStream = PuterFileSystem.prototype._openReadStream;
PuterFileSystem.prototype.openWriteStream = PuterFileSystem.prototype._openWriteStream;
PuterFileSystem.prototype.size = PuterFileSystem.prototype._size;
PuterFileSystem.prototype.lastModifiedDate = PuterFileSystem.prototype._lastModifiedDate;
PuterFileSystem.prototype.move = PuterFileSystem.prototype._move;
PuterFileSystem.prototype.copy = PuterFileSystem.prototype._copy;
PuterFileSystem.prototype.propertyManager = PuterFileSystem.prototype._propertyManager;

server.beforeRequest((ctx, next) => {
    ctx.response.setHeader('MS-Author-Via', 'DAV');
    next();
});

// Authentication middleware
app.use(async (req, res, next) => {
    const authHeader = req.headers.authorization;

    const credentials = Buffer.from(authHeader.split(' ')[1], 'base64').toString();
    const [username, password] = credentials.split(':');

    try {
        const user = await validateUser(username, password, Context);
        if (!user) return res.status(401).send('Invalid credentials');
        req.user = user;
        delete req.headers.authorization;
        next();
    } catch (error) {
        console.error('Authentication error:', error);
        res.status(500).send('Internal server error');
    }
});

app.use('/webdav', webdav.extensions.express('/', server));

app.listen(port, () => {
    console.log(`Puter WebDAV server running on port ${port}`);
});

return server;

}

module.exports = { startWebDAVServer };`

[KernelDeimos]

this error happens when there's no instance of Context from asyncLocalSorage. most likely it's because of how the new code is initialized. The snippet in your comment is difficult to review, can you open a draft PR instead?

[dawit2123]

@KernelDeimos
I have created a draft pull request: #1188
Please check it out.