Resolved GitHub unit test failures due to attempts to fetch git describe details that are, in that
test environment, not available.
Picked up latest version of Package Fission. Commensurate with this, also upgraded various dependencies to be at least Ubuntu 22.04 and Golang 1.19 now required by Package Fission.
Resolved a lease deadlock scenario. As part of this fix, dismounts or shutdowns of iclients now triggers implicit releasing of all of their held leases.
There is now a /keepalive URL API that, if configured to automatically exit, will prevent that. This is a feature implemented to enable an overseer to have confidence that if the overseer cannot communiate with an imgr that the imgr will dismount and stop serving any volumes. This allows for the overseer to relocate the serving of the volume(s) previously served by the unreachable imgr to a different imgr without risking the unsupported case where more than one imgr thinks it should be serving the same volume.
This release marks the functional completeness point for ProxyFS. It is also notable in that the supported portions of popular file system test suites (most recently PJDFSTEST) now verify correctness. It is important to point out the limitations of this functionality, however. While ProxyFS provides a form of network attached storage, it has focused on traditional hierarchical file systems presented over a network. Thus, directories and files, along with symbolic links, are fully supported. Other inode types, however are not. Further, concepts like SETUID and SETGID are explicitly disabled as is the Sticky Bit (traits normally maintained in an inode's MODE attribute). As might be imagined, excluding test cases reliant on special inode attributes as well as inote types other than these three was a non-trivial exercise. But the result has been well worth it.
Enjoy this distributed file system atop your OpenStack Swift scale out storage!
This release marks the milestone of removing all previously skipped Unit Test cases and the passing of the full set. The focus of development on ProxyFS now moves into Functional Test completeness and Hardening areas. Performance tuning/optimization will follow those efforts.
This release marks the transition to the re-implementation of ProxyFS. The primary goal of the new release was to better support distributed FUSE clients of a ProxyFS file system. Initially, ProxyFS ran on one of an OpenStack Swift's Proxy nodes and was tighly integrated with the Proxy "ring" such that Object API access to a ProxyFS-managed portion of the namespace would map to a single ProxyFS service supporting that file system. For network file access, both NFS (via nfsd) and SMB (via Samba) were integrated. This approach had a number of drawbacks not the least of which was file system access data paths all traversed a single ProxyFS instance. Notably, Object API access only routed control path through the ProxyFS instance. Data transfers via the Object API to ProxyFS file systems still benefitted from the scale-out nature of OpenStack Swift.
To better address the scale-out need of the file system data path when accessed via POSIX, a FUSE client called PFSAgent was constructed as well. This approach nicely moved the POSIX file system access off of a Swift Proxy node. Alas, PFSAgent still imposed a single data path limitation for POSIX file system access to a ProxyFS file system.
In designing the changes required to PFSAgent to support a truly distributed data path, the complexities of integrating with the existing ProxyFS architecture proved overwhelming. Indeed, much of the complexity derived from the continued access to the file system provided directly by the central ProxyFS service that had become redundant. In addition, supporting Object API access as well presented even more challenge. Finally, even the tighly coupled integration of ProxyFS with the Proxy "ring" of Swift was less than ideal.
So a new approach was taken. This presented an opportunity to significantly clean up the "on-disk" layout of the file system. In addition, improvements to OpenStack Swift to support very large Containers obsoleted the previous requirement to map a ProxyFS file system on top of an entire Swift Account. Along the way, a couple of key ProxyFS features were deprecated:
-
Object API Access - what was often termed "Bi-Modal", the ability to read and write to file system elements depended upon the otherwise undesireable tight integration with each Swift Proxy... so was, for the time being, thought to be expendable
-
SnapShots - similarly, the ability to access a read-only view of the file system at some scheduled point in the past was a rarely cited feature that, for now, is not supported anymore
This is a big change to consume all at once. To aid in this, there are a couple of ways to start looking at the codebase:
-
The leading
iconvention - a quick perusal of the ProxyFS repo will reveal that a large body of code has been removed. Meanwhile, a new set of directories have arrived - each starting with the letteri. Also, many subdirectories follow a convention of containing a command line program in the parent directoryiXXX/with a package implementation subdirectory with thepkgsuffix (i.e.iXXX/iXXXpkg/). These are:iauth- a simplified "AUTH" plug-in framework for which a Swift Auth-compliant example is provided (iniauth/iauth-swift/)icert- a command-line tool to generate RootCA and signed certificatesickpt- an optional service to provide a highly available, strictly consistent solution for the CheckPoint record of each file system that is the only Swift entity subject to Eventual Consistency imprecisioniclient- a FUSE client for a ProxyFS file system of which there can be manyidestroy- a simple tool to clean out an OpenStack Swift Containerihtml- support files for embedded HTTP servers iniclientandimgrilayout- a complete specification of the "on-disk" layout of a ProxyFS file systemimgr- a centralized service that provides a CheckPoint'd management of a ProxyFS file system's Inode Table as well as a Lease Manager enabling the scale-out oficlientinstances presenting a ProxyFS file systemiswift- a minimal RAM emulation of OpenStack Swift API
-
A move to GoDoc (away from readme's and other forms of often out-of-date documentation) has been provided:
- To launch the GoDoc server:
godoc -goroot <path to your ProxyFS repo> - Browse to: http://localhost:6060/pkg/ for things like
github.com/NVIDIA/proxyfs/ilayout- the entire "on-disk" file system layout descriptiongithub.com/NVIDIA/proxyfs/imgr/imgrpkg- config and API info forimgrgithub.com/NVIDIA/proxyfs/iclient/iclientpkg- config and API info foriclient
- To launch the GoDoc server:
With such a dramatic departure from the 1.XX line, it should be expected that the current ProxyFS code base is in flux as problems are identified and addressed. In addition, a large body of performance tuning remains outstanding.
Welcome to 2.XX !!!
PFSAgent was getting a SIGSEGV on a DoSetAttr() operation when the inode's current stat() information had not been cached. The fix ensures that the current stat() information is in the cache (and, of course, coherent with the actual state of the inode).
As of 1.19.0, we should have indicated the impact to upgrading from prior releases... so doing that here now.
While many .conf changes have occurred since 1.18.0, all are backwards compatible for ProxyFS itself. Unfortunately, in PFSAgent, an important change in the RetryRPC mechanism has been implemented. In 1.18.0, the way a PFSAgent instance would perform an RpcMount is to send this request to the Swift Proxy. The pfs_middleware in the Swift Proxy would determine which ProxyFS instance serves the Volume being mounted... and forward the RpcMount request there. The response to RpcMount would include two items:
- IPAddr:Port# of RetryRPC TLS endpoint of the serving ProxyFS instance
- RootCA PublicKey PEMBlock used to "sign" the ProxyFS instance's TLS Cert
Release 1.19.0 moved to a new model where such certificates are
computed "outside" of ProxyFS (though the new icert tool can
be used to do so). As such, new configurations are available
to tell ProxyFS what to use. These are:
- [JSONRPCServer]RetryRPCCertFilePath
- [JSONRPCServer]RetryRPCKeyFilePath
Meanwhile, in PFSAgent, the RpcMount request needs to be sent directly to the serving ProxyFS instance. Swift Proxy, by way of pfs_middleware, will no longer provide this discovery. So to provide the following:
- [Agent]RetryRPCPublicIPAddr
- [Agent]RetryRPCPort
- [Agent]RetryRPCCACertFilePath
It is also now possible to operate without TLS protecting the RetryRPC
transport. To opt for TCP, instead of TLS, for the RetryRPC transport,
the ProxyFS configuration should either don't define RetryRPCCertFilePath
nor RetryRPCKeyFilePath or define them as empty strings and RetryRPC will
use TCP rather than TLS.
If opting for TCP instead of TLS for the RetryRPC transport, the PFSAgent configuration should either not define RetryRPCCACertFilePath or define it as an empty string.
TOX fixes for WSGI middleware in support of move to Python3.
Closed file state issues in PFSAgent when lookup(), readdirplus(), and stat() operations are performed alongside write() activity. Similar issues are resolved between lookup() and readdirplus() operations in the presence of parallel directory modifying operations. There was also a case where doing a simple file read() would lose stat() information when performed in the presence of write() operations.
New support for TLS connections to etcd added.
PFSAgent's connection to ProxyFS for metadata operations was previously transitioned to TLS. Now, TLS is optional (though, of course, highly recommended).
Replaced the auth plugin model with a new/simpler Golang plugin mechanism. Previously, the auth plugin was actually a standalone process launched by PFSAgent. As such, it could be written in a variety of languages. With the new Golang plugin model, the plugin must, of course, be written in Golang (or, at least, provide a Golang-callable function that, perhaps, wraps an implementation in another language).
A new implementation of Swift emulation, complete with Swift Auth
support, has been added (iswift). The tool is implemented with the
pattern of its implementation being in a package, iswiftpkg, below
iswift in the repository source tree enabling easy integration
with Golang code (e.g. tests) that need easy Swift emulation.
For easy generation of CA and Endpoint certificates (along the lines
of what openssl provides), a new icert tool has been added. Similar
to that of iswift, the implementation of icert is in subdirectory
package icertpkg.
A new RpcLease benchmark has been added. It is anticipated that Lease Management becomes the scale out bottleneck in the move to PFSAgent usage. Hence, it is important to quantify Lease performance in the target environment in order to anticipate file system performance limits at least in metadata heavy workloads. Presumably, bandwidth limitations will be proportional to the scale out of PFSAgent instances as well as the underlying Swift Object Store solution.
Many changes relate to the migration of this and other repositories from the old github.com/swiftstack/ tree to github.com/NVIDIA/ tree.
As the move to only support PFSAgent access to file systems (perhaps eventually restoring BiModal access as well), support for SMB and NFS is deprecated. This removes the tight linkage between ProxyFS and Samba necessitated by the former vfs submodule's support for now obsolete versions of Samba.
To restore SMB & NFS support, simply deploy PFSAgent (perhaps adjacent to ProxyFS) and use your distribution's Samba and NFSd support to export the mount point presented by PFSAgent.
This release introduces a lot of functionality development work targeted at the 2.0 version of ProxyFS. In short, with the move to supporting file systems only exposed by PFSAgent, the architecture of ProxyFS has proved inappriate in that it majorly supported data path operations that will no longer be used. Even metadata operations triggered by PFSAgent will now be moved to PFSAgent. All that remains in ProxyFS going forward is the centralized Lease Management functionality and the checkpoint operation. Further, the checkpoints only involve sync'ing the Inode Table that is, itself, greatly simplified (now only holding a "pointer" to where the state of an Inode can be found).
These changes, set to become operational in the 2.0 version of ProxyFS,
are being implemented in a parallel set of subdirectories. These mostly
follow the patter described for iswift and icert above. Specifically,
imgr is the upcoming implementation of the server side (replacing
proxyfsd) while iclient is the upcoming implementation of the client
side (replacing pfsagentd). The work is well underway and, indeed, the
bulk of the commits since 1.18.0, cover that effort. At this time, only
a small subset of the imgr work is operational. None of iclient has
been implemented.
One big switch in the 2.0 model is a move away from Swift Accounts as the top level of a ProxyFS file system. At the time of its adoption, the Swift Account was chosen so that ProxyFS could load balance its creation of Objects across many Swift Containers. At that time, Swift had modest limits on the number of Objects that could reside in a single Container, so this choice was then necessary. But Swift has dramatically improved support for very large numbers of Objects in a Container since that time. Making this move to a ProxyFS file system matching a Swift Container instead of a Swift Account adjusts to this new capability. It also avoids the unfortunate confusion over files and symlinks placed in the root of a ProxyFS file system were not visible via BiModal (Swift or S3 API) access. Finally, this move better aligns with supporting deployments where only S3 API access is used and, hence, the whole concept of a Swift Account is not involved (since S3 Buckets map to Swift Accounts).
To get a good handle on how the 2.0 file system will "look" in terms
of "on disk" representation, a new ilayout package is provided. It
is believed to be complete. It describes how the CheckPoint Object
(now Object Zero; previously a Header on the checkpoint container)
points to the file system SuperBlock. From there, the InodeTable
is described as well as the format of the metadata portion of each
Inode. The ilayout package also introduces a new pattern of heavy
use of Golang godoc mechanisms to make browsing of the data structures
and APIs more convenient. This pattern will be applied across all of the
new 2.0 components (see in particular imgrpkg underneath imgr for a
full description of how to interact with the new server side of ProxyFS).
In the meantime, the pre-2.0 functionality will be maintained in their
usual subdirectories (i.e. proxyfsd and pfsagentd).
Incorrect settings in some config values could crash ProxyFS upon startup or SIGHUP.
Newer Swift versions introduced incompatibilities in some of the development and test environments (e.g. saio/sait VMs, runway cookbooks, and the newly available Dockerfiles).
Liveness Checker is now optionally configured.
Debug HTTP Server is now optionally configured.
Significant number of HTTP RESTful APIs now provided to help debug and, as necessary, patch a corrupt file system.
Golang 1.15.5 is now minimally required. As part of this upgraded requirement, ProxyFS has converted to Go Modules.
The new Lease Management system has been added to ProxyFS to enable multiple PFSAgent instances to consistently access a common ProxyFS Volume in a way that enables scaling out the read and write data paths much in the same way BiModal (Swift & S3 API) access scales out. All metadata operations continue to be forwarded to ProxyFS for centralized processing, but reads and writes proceed with minimal interaction with the ProxyFS responsible for the Volume.
Use cases vary, but it had been hoped that scaling out read/write performance but leaving metadata operations to be managed centrally by a single ProxyFS instance would be sufficient. In the current crop of use cases, this turns out not to be quite the case. Further, supporting such a division of labor between PFSAgents and ProxyFS has proved quite challenging.
As such, post this release, a new model is being adopted where ProxyFS merely maintains the Inode Table that now maps to Swift Objects per file/directory/symlink (rather than today's model where each Inode Table Entry is the actual Inode). With the new Lease Management system, this will enable a fully scale out model for all data and (nearly) all metadata operations to be conducted exclusively in each PFSAgent instance.
This transition will not be backwards compatible and will therefore start with an increase in the "major" revision number (from 1 to 2).
Resolved lack of randomness in pfs_middleware when selecting ProxyFS instance to query to learn which ProxyFS instance is currently serving a Volume. If the first ProxyFS instance selected just happens to be unreachable, a 503 status could be returned by Swift Proxy to the HTTP Client. No amount of retries would previously select a different ProxyFS instance. Now, the list is uniquely randomized on each retry.
Resolved metadata defragmentation RESTful API (/meta-defrag) implementation's inability to defragment the last 1% of metadata.
Added support for building Docker Image for SAIO.
Added support for inserting PFSAgent instances into a Docker Image.
Enabled use of fractional percentage ranges in metadata defragmentation RESTful API (/meta-defrag?range=...-...). Since Volume metadata changes are frozen during a defragmentation operation, this is useful when the total amount of metadata is very large.
Enabled multiple file defragmentation RESTful API (/defrag/<filepath>) invocations to proceed in parallel. Previously only one such would execute at a time.
With liveness checker enabled on multiple nodes, a race condition could be hit in the "doFollower()" logic of the RaFT-lite cluster formed among the ProxyFS instances resulting in a crash.
PFSAgent had a variety of errors returning stale file sizes and, sometimes, content.
Picked up support for Swift 2.25.0.5 in test VMs/containers.
Prevent premature invalidation of dirty file inode caching in PFSAgent leading to subsequent incorrect (stale) read results.
Correct reporting of file size when a non-empty file is first accessed by PFSAgent.
Log rotation events are not logged.
Memory fragmentation has been improved particularly in calls to dump a configuration (package conf's ConfMap).
Avoid PFSAgent crashes when the lack of leases means it doesn't know that the underlying file inode has been removed when it is lazily flushing write data to it.
Provide a cap on remaining unbounded memory growth due to caching of extent maps for files in PFSAgent. The new tunable that limits this memory consumption is [Agent]ExtentMapEntryLimit.
Fixed memory leak in PFSAgent where file stats not recently accessed were cached indefinitely.
Fixed race condition in retryrpc path (PFSAgent->ProxyFS) that could be exposed when connections need to be reestablished and requests retransmitted.
Updated pfs-swift-load benchmarking tool to issue flushes as the file is closed so as not to overstate write performance for short tests.
Added full /debug/pprof support in both ProxyFS and PFSAgent on their standard embedded HTTP servers.
Opening a file with the O_TRUNC option in a PFSAgent-mounted file system would fail to truncate a pre-existing file.
Memory leak in ProxyFS when PFSAgent is in use has been resolved.
PFSAgent's Swift Auth PlugIn now may be located in the provisioned SAIO search path of both the root and vagrant users.
Detection of available space on each Swift device (Account, Container, and Object) is now enabled. If any device utilization crosses either of two thresholds, access to all Volumes will be restricted. The first threshold prevents writes to files in the file system. Since there are many more operations that actually consume device space (even deletes), exceeding the second threshold converts all Volumes to be read-only. The "/liveness" URL (in the embedded HTTPServer in ProxyFS) will now also report the disk space utilization of each Swift device as well.
In addition to the one-node SAIO VM already present, a new three-node SAIT ("Swift All In Three") set of VMs are now presented. This enables testing ProxyFS Cluster functionality.
This release also includes initial work in support of the Lease Management feature that will ultimately enable multiple PFSAgent instances to safely share read/write access to a common ProxyFS Volume. The new RPC is simply called "Lease" (i.e. Server.RpcLease) but remains non-functional at this point. Such functionality also includes an "upcall" mechanism such that ProxyFS may inform a PFSAgent instance of important events requiring their response (e.g. a conflicting Lease is needed by another PFSAgent instance, or the Volume is leaving and requires the PFSAgent to unmount it).
Stale versions of PFSAgent were able to generate a connection storm against ProxyFS resulting in exploding the number of open files for the ProxyFS process. This update prevents that condition and generates log entries to indicate which PFSAgent clients are out of date.
If one ProxyFS peer is stuck handling a SIGHUP, a subsequent ProxyFS instance could indefinitely block in its SIGHUP simply waiting for the first instance to resolve its issue (or be terminated). At most, this condition can now delay the second ProxyFS by only one second (see Cluster.MaxRequestDuration).
Added support for a pluggable Authentication mechanism in PFSAgent. See the README.md files in both pfsagentd/ and pfsagentd/pfsagentd-swift-auth-plugin/ for details.
Updated PFSAgent to tune mount initialization to obtimize parallel requests.
Object API will create directories and files readable and writeable by anybody from the file side.
Added performance stats and config reporting to PFSAgent HTTPServer.
User Agent used in Swift API calls now customized to identify ProxyFS and PFSAgent specifically.
Added missing write flow control to PFSAgent to limit RAM footprint and avoid overrunning Swift.
Fixed incorrect flushing during PFSAgent dismount.
See new CONFIGURING.md to learn about configuring ProxyFS.
Corrected a B+Tree balance violation (package sortedmap) that could result in worse than the theoretical worst-case metadata tree height.
Previously, if a file is moved over the top of another file, the
file being replaced would have its LinkCount reduced by one and,
yet, if the LinkCount reached zero would not be erased from the
volume. As such, the Inode and any of its referenced LogSegments
would remain in Swift unreferenced. This has now been corrected.
The fix to the B+Tree imbalance bug includes a new pfs-fsck failure being reported for currently imbalanced metadata trees. New volumes created after the fix has been applied will not suffer from this reported corruption that is, otherwise, only a performance impact.
The fix that cleans up an Inode when its LinkCount reaches zero,
once applied, will only correctly clean up Inodes deleted after
that point. Previously unlinked Inodes will, unfortunately,
remain in Swift unreferenced.
Added a new SwiftRetryDelayVariance setting to PFSAgent to perform HTTP Retries at varying backoff delays. This avoids certain lock-step pathologic cases when overrunning Swift.
Fixed hangs in PFSAgent hit under heavy write loads.
Fixed stale file size reported by stat() for PFSAgent files that
have been expanded via truncate --size.
Fixed crash in ProxyFS when pruning acknowledged RPC replies.
A number of hang conditions in PFSAgent when servicing large numbers of small file reads and writes have now been addressed.
A new mechanism used by PFSAgent to GET and PUT file data objects (LogSegments) avoids the 404 Not Found responses as well as the performance impact of unnecessarily invalidating the Swift memcache. Swift version 2.24.0.2 is now minimally required.
Buffering RPC responses from ProxyFS to PFSAgent previously could exhaust memory in extreme circumstances. This has now been addressed by implementing aggressive pruning of this buffer on the ProxyFS end.
PFSAgent needing to retry GETs would possibly crash requiring a restart. Retries should be expected from time to time.
PFSAgent GETS and PUTs to a Swift Proxy bypass the normal translation performed by ProxyFS to gain direct access to the LogSegments containing file data. This previously often caused 404 Not Found responses due to overwhelming the memcached and Swift Container servers for a variety of reasons. This update avoids this behavior entirely.
The aforementioned 404 Not Found fix required an update to Swift itself. Hence, in order for PFSAgent to operate at all, Swift must be at least at version 2.24 that includes a new mechanism by which PFSAgent requests the Swift Proxy to bypass consulting ProxyFS.
Overly short [Agent]SwiftTimeout settings resulted in PUT timeouts from PFSAgent to Swift that were fatal. The new suggested timeout moves from 20s to 10m to avoid this issue.
Very short [Agent]MaxFlushTime settings (e.g. 100ms) resulted in a lockup of writes arriving for a file that was in the middle of time-triggered flushes. For the previous suggestex timeout of 10s, this was very rarely an issue because either clients would flush their file writes long before this or they would easily exceed MaxFlushSize (suggested value of 10MiB) data triggering a flush. By lowering MaxFlushTime, time-triggered flushes would expose the condition. This has now ben resolved.
As mentioned above, [Agent]MaxFlushTime settings can be set very low (e.g. 100ms). At such short durations, performance on small file workloads dramatically improves. This is not well understood yet why the system doesn't respond more gracefully to such small file workloads, so PFSAgent users should make note to lower this value in their previous configurations that probably set MaxFlushTime to 10s. Indeed, with the aforemented fix for handling short MaxFlushTime settings, the new suggested value is now 200ms.
Continuing from 1.15.0, PFSAgent currently cannot run successfully on macOS due to some as yet unresolved support in package fission.
PFSAgent now uses a new path to ProxyFS for metadata operations that bypasses the hop through a Swift Proxy process. This new connection is long lived and secured by TLS. Should this connection drop, it will be reestablished in such a way that issued metadata requests are only executed once.
PFSAgent now utilizes a new "fission" package enabling multithreaded upcall servicing whereever possible. Linux will still, under some circumstances, serialize what it deems potentially conflicting operations.
Various fixes in PFSAgent write path resulted in an incoherent ExtentMap describing a file's contents in Swift such that a subsequent Read could read invalid data. This also affected files that are truncated or later extended.
PFSAgent currently cannot run successfully on macOS due to some as yet unresolved support in package fission.
To expose the TLS Port utilized by PFSAgent, [JSONRPCServer]RetryRPCPort specifies the port on the PublicIPAddr of the node hosting ProxyFS. This port (on each Swift Proxy node running ProxyFS) must be made accessible to any entity running PFSAgent.
Memory leak PFSAgent's read cache is fixed. This bug would present whenever a file is larger than the read cache line size and a read was issued to the portion of the file beyond the cache line size (assuming the file was written sequentially to one or more LogSegments lartger than cache line size).
Note that an Out-Of-Memory ("OOM") condition is still entirely possible with PFSAgent deployed as the only limits to its read cache memory consumption are the ReadCacheLineSize and ReadCacheLineCount configuration parameters.
Allow chmod to work on a directory via PFSAgent.
Added an online FSCK tool (pfs-fsck). Note that the tool will not "stop the Add bucketstat counters to measure all checkpoint operations and how long individual parts of checkpoint processing take. Add some bucketstat counters to measure extent map lookups and updates for Read() and Write() operations and their analogous object operations. Add bucketstat counters to measure B+Tree flush operations.
Added pfs_middleware configuration data to /info resource.
Significantly improve the performance of concurrent sync operations by batching checkpoint operations. If multiple threads request a checkpoint, only perform one checkpoint instead of one for each request.
Significantly improve the performance of the FUSE mountpoint by treating a Flush() operation as a no-op, which it is. It does not imply any sort of persistence guarantees.
Pick up sortedmap.TouchItem() fix in 1.6.1 (glide update).
Fix a few bugs in confgen.
Changes to runway environment for pfsagent mount points and allow users to easily enable/disable core dumps.
Fixes to confgen search for template files.
Add confgen tool to generate SMB, VIP, NFS and FUSE configuration files from a proxyfs configuration
Fix a bug in ProxyFS in retry of chunked put operations that caused a panic.
Sundry PFSAgent bug fixes.
Removed an unnecessary checkpoint performed before each PFSAgent RpcWrote operation that is generated as each LogSegment is PUT to Swift. The prior behavior put a strain on the checkpointing system when a large set of small files are uploaded via PFSAgent exposed FUSE mount points. Note that explicit flushes (fsync() or fdatasync() calls) will still trigger a checkpoint so that ProxyFS/PFSAgent can honor the request faithfully.
The "mount retry" fix was actually misnamed. What it actually does is
attempt to issue multiple Checkpoint HEAD requests and achieve agreement
on what is returned by a majority quorum. In each such HEAD request,
a retry logic pre-existed where anything other than a "200 OK" would
be retried... hence the overloading of the term "retry". Anyway, in the
case where mkproxyfs is being run to format a previously empty Swift
Account, the retry logic could take a very long time (e.g. minutes) to
give up. Thus, by performing each of those Checkpoint HEAD requests in
series, the formatting process could take an excessive amount of time
just coming to the decision that the Swift Account needs to be formatted.
This fix simply performs those quorum Checkpoint HEAD requests (that each
will be retried a number of times) in parallel thus returning the total
time for the format back to what it was prior to the 1.12.0 "mount retry"
fix.
Added an online FSCK tool (pfs-fsck). Note that the tool will not "stop the world". As such, it is possible for it to report false positives for missing objects (both metadata and file data). Re-running the tool will typically note those false positives were false and move on... but perhaps find more. As such, the tool is really only reliably able to avoid announcing false positives on an otherwise idle volume.
Implemented support for a "Recycle Bin" mechanism where objects are simply marked as being in the Recycle Bin rather than aggressively deleted. If an attempt is made to later access the object, a log message will report this and statistics bumped to indicate the condition that would have been a file system corruption had the object actually been deleted. Note that this feature is normally disabled.
Layout Report has been streamlined to, by default, report the "running count" of the metrics rather than actually counting them (which would involve paging in all the metadata...a potentially very time consuming activity that must proceed while the "world is stopped"). It is still possible to do the brute force "count"... and note any discrepencies with the "running count" previously reported.
The pfs-swift-load tool has been enhanced to support arbitrarily deep paths. This is provided to demonstrate specifically the impact of file path length on SMB performance.
PFSAgent, like ProxyFS itself, now supports (at least) an HTTP query to report the running version. This may also be used to confirm that PFSAgent is up and operating on a particular Volume.
Addressed stale/cached stat results for (FUSE and) NFS mount points by shortening the timeout...with the tradeoff that this could increase overhead of certain metadata querying operations somewhat.
Added support for HTTPS in ProxyFS Agent. Previously, Swift Proxies behind a TLS-terminating Load Balancer would result in an attempt to use non-HTTPS connections following authentication via the Swift API.
Added volume mount retry logic to handle the theoretical case where a stale Checkpoint Header is returned during the Volume Mount process.
Added logging of each Checkpoint Header operation. This amounts to a log message generated once every ten seconds per Volume typically...though clients issuing FLUSH/SYNC operations may accellerate this.
In addition, periodic stats are logged as well (in addition to the limited set regularly reported earlier). Default interval is 10 minutes per Volume.
This is a small delta from 1.11.1 to temporarily disable deletion of thought-to-be unreferenced objects in the checkpoint container. A working theory of one such issue is that a object holding metadata for a volume was inadvertantly thought to no longer be referenced. As such, the object was scheduled for deletion. Upon a subsequent re-mount, the object could not be found and the re-mount failed.
A new (temporary) boolean setting in the FSGlobals section titled MetadataRecycleBin will, for now, default to TRUE and, instead of issuing DELETEs on objects in the checkpoint container thought to now be unreferenced, a new header will be applied to them titled RecycleBin (with a value of true).
Support for storing Volume Checkpoints in an ETCD instance are now supported. This feature is enabled by setting an optional key to true along with other keys specifying the endpoints of the ETCD instance and the name of the key to use for each Volume's Checkpoint.
Object PUTs over existing file system directories behavior has been corrected. For instance, now a PUT over an empty directory will replace it.
Adapted to latest (2.21.0.4) version of Swift.
Alpha version of PFSAgent is now available. This tool presents a FUSE mount point of a Volume served by a ProxyFS instance by using the new PROXYFS HTTP Method of (pfs_middleware in) Swift.
The COALESCE HTTP method used by S3 Multi-Part Upload had a number of deficiencies. Most visibly, when COALESCE was issued to overwrite an existing object, metadata was not applied correctly.
Objects formed via COALESCE now have a correctly set non-MD5 ETAG.
GET issued to a Container specifying a marker now returns the objects in the Container following the marker. Previously, if the marker indicated an Object in the Container's directory (i.e. rather than a subdirectory of the Container), the list of following Objects would be empty.
SnapShotPolicy changes now picked up during SIGHUP.
ProxyFS typically should not expect errors coming from Swift. Indeed, the only expected errors are during Volume/Account formatting as the "mkproxyfs" tool attempts to verify that the underlying Account is, indeed, pristine. But other unexpected errors were logged along with being retried. This release ensures that any payload returned by Swift with bad HTTP Status is also logged.
A number of Swift and S3 operations are necessarily path-based. Hence, although the RESTful Object-based APIs purport to be atomic, this is actually impossible to honor in the presence of File-based access. In attempts to retain the atomicity of the RESTful Object-based APIs, several deadlock conditions were unaddressed. One such deadlock addressed in both 1.8.0.6 and 1.9.2/1.9.5 involved the contention between the COALESCE method that references an unbounded number of file paths with other path and Inode-based APIs. This release addresses in a much more encompassing way all the contentions that could arise within and amongst all of the path and Inode-based APIs.
The SnapShotPolicy feature introduced in 1.8.0 was inadvertantly disabled in 1.9.5 due to the conversion to the new transitions package mechanism introduction. As such, only explicitly created SnapShots would be created. This release restores the ability to schedule SnapShots via a per-Volume SnapShotPolicy.
Lock tracking capabilities have been significantly enhanced that will provide several mechanisms with which both deadlocks and unusually high latency conditions can be examined with a performant mechanism that, when enabled, clearly reports what contentions are at the root of the observed condition. This instrumentation includes instrumenting both the various Mutexes that serialize data structure access as well as per-Inode locks that serialize client operations.
A modern file system such as ProxyFS is obliged to support nearly unbounded parameters governing nearly every enumerated aspect. To clients, this includes support for things like extremely large files as well as a huge number of such files. Internally, management of extremely fragmented files is also a demanding requirement. Despite this, constrained compute environments should also be supported. This release introduces support for particularly the 32-bit architecture of Arm7L based systems.
While not a new issue, the release focused on exposing the inherent incompatibility between path-based RESTful Object APIs and File Access. Indeed, such issues are impossible to fully accomodate. In addition, a key feature of the S3 API is support for so-called multi-part uploads. This is accomplished by clients uploading each part - perhaps simultaneously - to unique Objects. Once all parts have been uploaded, a Multi-Part Put Complete operation is performed that requests that, logically, all of the parts are combined to form the resultant single Object. Support for this fianl step is implemented by means of a new COALESCE HTTP Method effectively added to the OpenStack Swift API. Unfortunately, this is where the "impedance mismatch" between the hierarchical nature of the File System clashes with the "flat" nature of an Object API such as S3 (and, for that matter, OpenStack Swift).
The key issue is how to represent a Directory in the File System hierarchy. At this point, a Container (or Bucket) listing (via GET) will report both Objects (Files) and Directories. This is in conflict with an Object-only system that lacks any sort of Directory Inode concept. Indeed, several typical client operations are confused by the presence of Directories in the Container/Bucket listing (not the least of which is the widely used BOTO Python library used in S3 access). This conflict remains in the current release.
Made StatVfs() responses configurable. Note that values for space (total, free, available) continue to be set to artificial values... but at least they are now settable.
Re-worked logic for various path-based operations (i.e. operations invoked via Swift or S3 APIs) could result in a deadlock when combined with file-based operations. These have now largely been resolved (see Issues section for details on what is not).
Modified pfsconfjson{|packed} to auto-upgrade supplied .conf files to report the VolumeGroup-translated form (if up-conversion would be done by ProxyFS itself).
COALESCE method, when targeted at an existing Object, would previously fail to recover the overwritten Object's space.
Various pre-VolumeGroup->VolumeGroup auto-upgrade patterns could result in false reporting of a Volume being moved to a VolumeGroup where it already exists in response to SIGHUP. Just restarting ProxyFS would not see this issue.
While much work was completed towards avoiding deadlock situations resulting from path-based (i.e. Swift/S3 API) operations, the work is not yet complete. A GET on a Container/Bucket that recurses could still result in a deadlock but the bug fix for this case largely closes that window. A PUT also has the potential for another deadlock situation that is equally very unlikely. No test case has been able to expose these remaining deadlocks so they remain theoretical.
Resolved race condition when simultaneous first references to
an Inode are executed resulting in a lock blocking any further
access to the Inode. This condition was frequently seen when
attempting a multi-part upload via the S3 or Swift HTTP APIs
as it would be typical/expected that the uploading of all the
parts of a new file would begin roughly at the same time.
It is now possible to perform builds and unit tests on the same node where an active ProxyFS session is in operation. Previously, identical TCP and UDP Ports were being used by default leading to bind() failures.
Updated to leverage Golang 1.11.4 features.
Added support for X-Object-Sysmeta-Container-Update-Override-Etag.
Added support for end-marker query params.
Added support for fetching a ReadPlan for a given file
via the HTTP interface. The ReadPlan may then be used to
HEAD or GET the individual LogSegments that, when stitched
together, represent the contents of the file.
Liveness monitoring now active among all ProxyFS instances
visable via JSON response to an HTTP Query on the embedded
HTTP Server's Port for /liveness.
Added support for VolumeGroups where the set of Volumes in a VolumeGroup are what is assigned to a Peer/Node rather than each individual Volume.
The response to fsstat(1), statfs(2), and statvfs(3) returns capacities that were hardcoded (e.g. Total Space of 1TiB). While it is currently not possible for such reporting to represent actual capacities, new config values are available to adjust the reported values. In addition, the defaults have been increased (e.g. Total Space is now reported as 100 TiB).
A race condition triggered by e.g. multi-part uploads could render the targeted portion of a file system indefinitely locked requiring a restart of the proxyfsd daemon. This release prevents this race condition.
Fix a bug that caused proxyfsd to leak "connection slots" from the ChunkedConnectionPool or the NonChunkedConnectionPool if the noauth proxy was not running when proxyfs tried to open a new connection (in swiftclient.acquireChunkedConnection() and swiftclient.acquireNonChunkedConnection()). This could happen during a reload of the noauth proxy when a new configuraiton is pushed.
Fix a bug that caused proxyfsd to exit if the SwiftStack controller deleted a file system using its two step process of first inactivating it (one reconfig event) and then deleting it (second reconfig event). In that case, the UnregisterForEvents() would be called twice and it would call logger.Fatalf() to complain, causing Samba (smbd) to exit and making the SMB client unhappy.
Fix a bug that caused proxyfsd to exit if the noauth proxy server restarted while file system i/o to Swift was in progress. In particular this could happen during a reconfig triggered by a controller config push.
Fix a bug in the snapshot code that generated snapshot names that contained a colon which Windows SMB clients find hard to cope with.
Fix a bug introduced in 1.8.0 that triggered a NULL pointer dereference if an HTTP GET request specified a byte range without an ending offset.
Add filesystem snapshots which are created and destroyed based on policies that are applied to each file system. Snapshots are accessible via the newly created "/.snapshot/<snapshot_name>" directory in the root of each filesystem.
Add bucketized statistics for package swiftclient, fs, and headhunter. The statistics can be queried via the built-in web server, using the URL "//localhost:15346:/stats" in runway environments and "//<private_IPaddr>:1534:/stats" in stand alone environments, where <private_IPaddr> is the private IP address used for the backend network.
Note: the bucketized statistics API, output format, and URL is unstable and changing quickly. It will be different in the next release.
Change config file format (for the *.conf files) so that user defined names for: Volume PhysicalContainerLayout FlowControl Peer are now preceeded by one of the strings "Volume:", "PhysicalContainerLayout:", "FlowControl:", or "Peer:", as appropriate, to insure that names are unique.
Initial work on a "liveness detector" to determine whether ProxyFS/Samba/NFS are currently up and serving file requests. This is part of an HA solution that we are developing.
Fix a bug that could cause data corruption if a file write sent via the FUSE or NFS interface failed on the first PUT attempt and had to be retried, in which case the PUT could be retried with incorrect data for the file. This bug was exacerbated by the next bug, which could cause PUT requests to exceed the 60 sec server timeout deadline.
Fix a bug where "log segments" containing data written to files were not closed and flushed after the 10 sec deadline (a bug in inFlightFileInodeDataFlusher() that could extend the PUT request beyond the Swift server's timeout deadline of 60 sec).
Fix a bug where ProxyFS would either move to a new container for file log segments too quickly ("MaxObjectsPerContainer" was not being checked against correctly).
Insure that ProxyFS will not accept requests via the FUSE interface while its re-reading its configuration (could lead to corruption).
Add additional units tests for sequential writes.
Improvements to the mock Swift testing environment, ramswift, to free memory when ramswift is restarted within a test.
Update generatedfiles Makefile target to make files that are now necessary.
Reworked proxyfsd daemon startup logic to avoid possible race conditions during startup.
Reworked ramswift daemon startup logic to avoid race conditions sometimes hit when running tests.
-
With the advent of Golang v1.11, new support for WebAssembly has arrived. To build for a WebAssembly target, setting GOOS=js and GOARCH=wasm is required. Unfortunately, the arrival of these two new values ("js" and "wasm"), Go source files ending with _js.go, in particular, will only be compiled if GOOS=js has been set. Previously (Golang v1.10 and prior), a file ending in js.go would always be included. This release captures a name change to various static files generated in package httpserver by adding an underscore ("") just before ".go" to avoid this new Golang behavior.
-
Enhance the swiftclient chunked put unit tests to try and cover concurrency and many more failure/retry scenarios. Add a new config file variable in the
SwiftClientsection,ChecksumChunkedPutChunks`` which defaults to 'false'. If set totrue` then data cached in a chunked put connection has a checksum computed when it is Sent and checked frequently on subsequent operations.
- Fix panic in inode cache discard thread.
- Rework flush logic to fix deadlocks when connections are exhausted.
- Added support for configuring the NoAuth Swift Proxy to an IP Address other than a default of (IPv4) localhost (127.0.0.1). Note that in the future, the defaulting to localhost may be removed, so users should take care to specify the NoAuth Swift Proxy IP Address in their configurations in the future.
- Added support for the "delimiter=" option for Swift API GET requests. This, along with the "prefix=" option, enables viewing of the directory hierarchy inside a Container in the same way one would view a file system.
- Added support for SnapShots. These are invoked via the RESTful API exposed by the embedded HTTP Server inside each proxyfsd instance. Note that this API is only reachable via the PrivateIPAddr and performs not authentication/authorization. You can find the new RESTful methods underneath the /Volume/ resource (along with FSCK, Scrub, and LayoutMap). Both JSON (textual) and formatted HTML is available.
- Added support for viewing a FileInode's ExtentMap via this same RESTful API underneath the /Volume/ resource (adjacent to the above-mentioned SnapShot resource).
- RPC Timeouts from pfs_middleware to proxyfsd have new defaults and are controllable via two distinct parameters optionally specified in the [filter:pfs] section of the proxy-server.conf:
rpc_finder_timeout specified in (floating point) seconds defaults to 3.0 applies when searching for a proxyfsd instance to ask where a particular Swift Account is being served
rpc_timeout specified in (floating point) seconds defaults to 30.0 applies to requests to the specific proxyfsd instance serving a particular BiModal Swift Account
- SMB clients making multiple mounts to the same Samba/ProxyFS instance could encounter all sessions/mounts being closed when requesting any one of them to terminate. This has now been corrected.
- Previously, a cache of Inode structures did not support eviction. As a result, a very large number of Inodes accessed since a ProxyFS instance was started could exhaust memory. To address this, a new background thread discards non-dirty Inodes from the Inode Cache. The behavior of the Inode Cache eviction thread is tuned by:
MaxBytesInodeCache - defaults to 10485760 (10MB) InodeCacheEvictInterval - defaults to 1s (disabled if 0s)
- As of this version, the metadata format has been updated from V2 to V3 in support of the SnapShot functionality. Unfortunately there is no going back. Once a V2 volume is mounted, it is immediately upgraded to V3 despite not (yet) having any SnapShots declared.
Ignore SIGPIPE, SIGCHLD, and some other signals that were causing proxyfsd to exit when it shouldn't.
there is no release 1.5.2.
Partially botched 1.5.0 release packaging.
Move to go version 1.10.
Significant improvements to fsck performance and what it validates. Fsck now detects and cleans up unreferenced objects in the checkpoint container. Fix a bug in fsck that caused sparse files to be flagged corrupt and deleted. Complementary to fsck, add "scrub" jobs that validate the object maps (extent maps) for files.
Fix a bug that caused B+Tree nodes to become quite large, which had a significant performance impact on large file systems.
Prettify the http pages generated by proxyfs.
Fix a bug in the B+Tree code that caused old objects in the .checkpoint container to become unreferenced instead of deleted.
Support Travis continuous integration testing on github.
- Support for hidden SMB Shares now available
- Development environment now enhanced with pin'd versions of dependencies
- Preliminary work for supporting an RPO of Zero in place but inactive
- Submodule vfs now no longer depends upon pre-installed submodule jrpcclient
- ProxyFS now built with a standard Makefile (obsoleting regression_test.py)
- Fix cross-container DLO authorization
- Metadata Recovery Point Objective ("RPO") is non-zero (except for file flush operations)
- Segment fault while handling SIGHUP during log rotation and volume migration
- Metadata Recovery Point Objective ("RPO") is non-zero (except for file flush operations)
- Added support for "async" flush in SMB (allows multiple simultaneous flushes)
- Above "async" flush resolves SMB 2 and above write issues with "strict sync = yes" setting in smb.conf
- Metadata Recovery Point Objective ("RPO") is non-zero (except for file flush operations)
- Source is now available on GitHub
- Added support for S3 Multi-part Uploads
- Formatting of a Volume File System now made explicit with
mkproxyfstool - Volumes may be added and removed (via SIGHUP) without restarting ProxyFS
- Configuration files now allow identical section names for different types of sections
- Support added for a distinct Storage Policy for metadata
- New RESTful API added for FSCK management via HTTP
- Recover trapped resources in Samba when ProxyFS halts
- Fixed memory leaks and slow performance during RoboCopy
- Resolved unbounded memory consumption as file systems grow
- Fix for ctime not being updated during various operations
- Fix for missing first file in a directory if it would sort before "."
- Specification of a Storage Policy for file data now honored
- Corruption following metadata checkpoint failures now halted
- Metadata Recovery Point Objective ("RPO") is non-zero (except for file flush operations)
- Caching of metadata in RAM now configurable
- Samba parameters now specified via the standard /etc/samba/smb.conf mechanism
- Fixed memory leaks in readdir() APIs issued via SMB
- Fixed metadata on objects set via Swift/S3 API
- Named Streams are disabled in SMB (enabling this is TBD)
- Upgrading metadata checkpointing from V1 to V2 experiences process hangs in some cases
- Updates to HTTP COALESCE Method
- Improved flushing of affected Swift connections during SIGHUP (reload)
- Improved dataflow during high number of unflushed open file traffic
- Resolved memory leaks in Samba processes during heavy Robocopy activity
- Resolved potential deadlock for unflushed files that are removed
- Hardened error handling between Samba & ProxyFS processes
- Named Streams are disabled in SMB (enabling this is TBD)
- Upgrading metadata checkpointing from V1 to V2 experiences process hangs in some cases
- Improved Object ETag MD5 handling
- Object SLO uploads converted to COALESCE'd Objects/Files
- Non BiModal Accounts remain accessible even when no ProxyFS nodes are available
- Named Streams are disabled in SMB (enabling this is TBD)
- Upgrading metadata checkpointing from V1 to V2 experiences process hangs in some cases
- Added statistics logging
- Fixed BiModal IP Address reporting following SIGHUP reload
- Fixed issue with large transfers causing Swift API errors
- Named Streams are disabled in SMB (enabling this is TBD)
- Upgrading metadata checkpointing from V1 to V2 experiences process hangs in some cases
Note: This was just a re-tagging of 0.53.0.1
- Named Streams are disabled in SMB (enabling this is TBD)
- Upgrading metadata checkpointing from V1 to V2 experiences process hangs in some cases
- Added support for Samba version 4.6
- Fixed memory leak in smbd resulting from a closed TCP connection to proxyfsd
- Named Streams are disabled in SMB (enabling this is TBD)
- Upgrading metadata checkpointing from V1 to V2 experiences process hangs in some cases
- Added avaibility improvements for ProxyFS Swift clusters to continue when a ProxyFS node is down
- Significantly improved logging during startup and shutdown
- New
mkproxyfstool now available to format Volumes (Swift Accounts)
- Embedded HTTP Server now reports current configuration once startup/restart (SIGHUP) completes
- HTTP Head on ProxyFS-hosted Objects now returns proper HTTPStatus
- Resolved incomplete file locking semantics for SMB
- Resolved issue where a file being written is deleted before its data has been flushed
- Corrected behavior of readdir() enabling callers to bound the size of the returned list
- Corrected permissions checking & metadata updating
- Resolved NFS (FUSE) issue where the underlying file system state failed to reset during restart
- Resolved SMB (smbd) memory leak resulting from unmount/remount sequence
- SMB (smbd) memory leaks resulting from restarting the ProxyFS process (proxyfsd) underneath it
- Named Streams are disabled in SMB (enabling this is TBD)
- Upgrading metadata checkpointing from V1 to V2 experiences process hangs in some cases
- Support for disabling volumes added
- Fixed missing flushing of modified files leading to zero-lengthed files
- Named Streams are disabled in SMB (enabling this is TBD)
- Upgrading metadata checkpointing from V1 to V2 experiences process hangs in some cases
- Improved metadata checkpointing mechanism (V2) performs optimized garbage collection
- Fixed clean-up of FUSE (and NFS) mount point upon restart after failure
- Fixed memory leaks in SMBd for readdir(), getxattr(), chdir() and list xattr
- Fixed race condition between time-based flushes and on-going write traffic
- Fixed multi-threaded socket management code in resolving DNS names
- Fixed missing support for file names containing special characters
- Named Streams are disabled in SMB (enabling this is TBD)
- Upgrading metadata checkpointing from V1 to V2 experiences process hangs in some cases
- Enhanced tolerance for intermittent Swift errors
- Read Cache now consumes a configurable percentage of available memory
- Flow Controls now get a weighted fraction of total Read Cache memory
- Configuration reload now supported via SIGHUP signal
- Fixed embedded HTTP Server handling of "empty" URLs
- Removed memory leaks in SMB handling
- Resolved potential corruption when actively written files are flushed
- Memory leak in SMB directory reading and extended attribute reading
- Process restart may leave NFS mount point in a hung state
- Named Streams are disabled in SMB (enabling this is TBD)