Design Document: Dragonfly dfcache Streaming Upload/Download with Container Path MappingDfcache streaming design #10
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added design document to enhance dfcache client with streaming gRPC interfaces.
…dfcache This design document proposes enhancing the Dragonfly dfcache client with streaming-based persistent cache operations and container path mapping capabilities to address Kubernetes deployment challenges and improve large file transfer efficiency. Key features: - Streaming interfaces via WritePersistentCacheTask/ReadPersistentCacheTask - Container-to-host path mapping for Kubernetes environments - Backward compatibility with existing dfcache functionality - Enhanced progress tracking and error handling - Performance optimizations for large file transfers - Comprehensive testing strategy for production readiness
…dfcache This design document proposes enhancing the Dragonfly dfcache client with streaming-based persistent cache operations and container path mapping capabilities to address Kubernetes deployment challenges and improve large file transfer efficiency. Key features: - Streaming interfaces via WritePersistentCacheTask/ReadPersistentCacheTask - Container-to-host path mapping for Kubernetes environments - Backward compatibility with existing dfcache functionality - Enhanced progress tracking and error handling - Performance optimizations for large file transfers - Comprehensive testing strategy for production readiness
chlins
requested changes
Jul 21, 2025
| oneof response { | ||
| WritePersistentCacheTaskStartedRequest write_persistent_cache_task_started_request = 1; | ||
| WritePersistentCacheTaskFinishedRequest write_persistent_cache_task_finished_request = 2; | ||
| WriteChunkRequest write_chunk_request = 3; |
There was a problem hiding this comment.
What is the meaning of the chunk here, is it the chunk of the persistent cache task, if so, please align the name completely.
|
|
||
| message WritePersistentCacheTaskFinishedRequest {} | ||
| message WritePersistentCacheTaskResponse { string task_id = 1; } | ||
| message WriteChunkRequest { bytes content = 1; } |
| } | ||
| ``` | ||
|
|
||
| #### Connection Lifecycle Management |
There was a problem hiding this comment.
Could you draw a chart to describe the whole lifecycle?
| ``` | ||
|
|
||
| #### Optimized File Reading | ||
|
|
There was a problem hiding this comment.
Please summary the change you did for improving the read operation.
| ``` | ||
|
|
||
| #### Optimized File Writing with Zero-Copy | ||
|
|
| @@ -0,0 +1,416 @@ | |||
| # Design Document: Dragonfly dfcache Streaming Upload/Download with Container Path Mapping | |||
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Design Document: Dragonfly dfcache Streaming Upload/Download with Container Path Mapping
Overview
This design document proposes enhancing the Dragonfly dfcache client to support streaming-based persistent cache operations with container path mapping capabilities. The enhancement will leverage dfdaemon's new gRPC streaming interfaces (
WritePersistentCacheTaskandReadPersistentCacheTask) to enable efficient large file transfers while addressing the path mapping challenges commonly encountered in Kubernetes containerized environments.The project focuses on extending the existing Rust dfcache client within the
dragonflyoss/clientcodebase, maintaining full compatibility with Dragonfly's design principles and existing components while introducing advanced streaming capabilities and seamless container-to-host path translation.Motivation
Path Mapping Challenges: In Kubernetes environments, dfcache clients running inside containers face significant challenges due to path differences between container internal paths and host paths where dfdaemon operates. This creates access permission issues and prevents effective file import/export operations.
Large File Transfer Inefficiency: Current dfcache implementation relies on non-streaming
UploadPersistentCacheTaskRequestinterface, which lacks efficiency for large file uploads and provides limited progress control and fine-grained task management capabilities.Enhanced User Experience Requirements: Users require better progress indication, more comprehensive error handling, and improved task control functionality, especially when dealing with large files in production container environments.
Streaming Interface Adaptation: dfdaemon's introduction of new streaming gRPC interfaces (
WritePersistentCacheTaskandReadPersistentCacheTask) presents an opportunity to significantly improve file transfer efficiency and user experience while maintaining system stability.Goals
Streaming Interface Adaptation: Enable dfcache client to fully utilize dfdaemon's new streaming gRPC interfaces (
WritePersistentCacheTaskandReadPersistentCacheTask) for efficient large file import/export operations with enhanced progress control and task management capabilities.Container Path Mapping Implementation: Develop a robust path mapping mechanism that seamlessly translates container internal paths to host paths, resolving file access and permission challenges in Kubernetes environments.
User Experience Enhancement: Provide comprehensive progress indication, improved error handling, and enhanced task control functionality to deliver a superior user experience, particularly for large file operations in production environments.
Code Quality and Maintainability: Ensure all enhancements maintain the highest code quality standards while preserving the simplicity and maintainability of the dfcache client, with full backward compatibility for existing functionality.
Performance Optimization: Achieve optimal file transfer efficiency through streaming operations while maintaining minimal performance overhead, ensuring the solution scales effectively for large files and high-concurrency scenarios.