-
Notifications
You must be signed in to change notification settings - Fork 2
/
lessfs.1
executable file
·223 lines (219 loc) · 11 KB
/
lessfs.1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
.\"
.\" lessfs.1 - the *roff document processor source for the lessfs manual
.\" You may contact the author by:
.\" e-mail: [email protected]
.\"
.TH lessfs 1 .\" "Lessfs reference" v1.1.0-alpha1 "2010"
.SH NAME
lessfs - An inline data deduplicating filesystem
.SH SYNTAX
.nf
\fBlessfs\fR /etc/lessfs.cfg /mountpoint\\
[\fB-o\fR big_writes,max_read=nnnn,max_write=nnnn]
.fi
.SH VERSION
This man page documents lessfs version 1.1.0-alpha1
.SH DESCRIPTION
\fBlessfs\fR is a filesystem that performs inline data deduplication. lessfs uses the 192-bit (24-byte) tiger hash algorithm by default to compare the data. The filesystem compresses the unique data blocks before writing them to disk.
.br
Lessfs supports a large number of compression protocols. See COMPRESSION for details.
.br
Lessfs 1.1 also supports a wide range of hashes to choose from with hash lenghts ranging from 192 to 512 bit. See HASHNAME for details.
\FB\
.PP
.SS OPTIONS
.IP "big_writes"
Greatly improves throughput when the filesystem is used for backup purposes. It requires a recent kernel >=2.6.26 and a recent version of libfuse.
Must be used with max_read and max_write.
.IP "max_read"
Specify the blocksize, lessfs requires max_read and max_write to be equal.
Valid values are : 4096,8192,16384,32768,65536,131072
.IP "max_write"
See max_read
.IP "Please read the fuse documentation for other fuse related options."
.IP "Examples of usage:"
.nf
Mounting lessfs with BLKSIZE specified in the configuration file is very simple.
lessfs /etc/lessfs.cfg /fuse
This will result in lessfs being mounted with the arguments:
-ohard_remove,negative_timeout,entry_timeout=0,\\
attr_timeout=0,use_ino,readdir_ino,default_permissions,allow_other,\\
big_writes,max_read=BLKSIZE,max_write=BLKSIZE
You can override the default settings removing the BLKSIZE line from the
configuration file and specifying the options on the commandline.
To mount lessfs with a \fB4k\fR blocksize.
lessfs /etc/lessfs.cfg /fuse -o negative_timeout=0,\\
entry_timeout=0,attr_timeout=0,use_ino,\\
readdir_ino,default_permissions,allow_other,\\
max_read=4096,max_write=4096
To mount lessfs with a \fB128k\fR blocksize.
lessfs /etc/lessfs.cfg /fuse -o negative_timeout=0,\\
entry_timeout=0,attr_timeout=0,use_ino,\\
readdir_ino, default_permissions,allow_other,\fBbig_writes\fR,\\
max_read=131072,max_write=131072
\fB*\fR A recent kernel >=2.6.26 and a recent version of libfuse are required.
\fBWARNING :\fR DO NOT USE max_read/max_write > 4096 with kernels older then 2.6.26.
.fi
.SS LESSFS CONFIGURATION FILE.
lessfs requires a configuration file that contains information about the
tokyocabinet databases that actually hold the filesystem data. You may choose
not to include some of the variables in the configuration file. In this case you
should export the variables into the environment.
Example: export BLOCKDATA_PATH=/data/dta
This feature can be used for scripting.
The configuration file should otherwise contain the following variables:
.IP BLOCKDATA_IO_TYPE
.nf
The way that lessfs stores the actual data.
Valid options are \fBfile_io\fR or \fBtokyocabinet\fR
Example: BLOCKDATA_IO_TYPE=file_io
tokyocabinet is selected by default.
.fi
.IP "BLOCKDATA_PATH"
.nf
The path to a directory that holds the blockdata database or file.
Example with tokyocabinet : BLOCKDATA_PATH=/data/dta
Example with file_io : BLOCKDATA_PATH=/data/dta/mydtafile
.fi
.IP "BLOCKDATA_BS"
.nf
The bucketsize of the blockdata database.
See the tokyocabinet documentation for details.
http://tokyocabinet.sourceforge.net/spex-en.html
Example: BLOCKDATA_BS=10485760
.fi
.IP "BLOCKUSAGE_PATH"
The path to the directory that holds the database that contains blockusage information.
.nf
Example: BLOCKUSAGE_PATH=/data/mta
.fi
.IP "BLOCKUSAGE_BS"
The bucketsize of the blockusage database. See BLOCKDATA_BS.
.IP "DIRENT_PATH"
The path to the directory that holds the directory structures.
.nf
.IP "DIRENT_BS"
The bucketsize of the dirent database. See BLOCKDATA_BS.
Example: DIRENT_BS=1048576
.fi
.IP "FILEBLOCK_PATH"
The path to the directory that holds the database that contains the tigerhash of the inode-blocknr combination. The data that belongs to the hash can be found in the BLOCKDATA database.
.IP "FILEBLOCK_BS"
The bucketsize of the fileblock database. See BLOCKDATA_BS.
Example: FILEBLOCK_BS=10485760
.IP "META_PATH"
The path to the database that contains the metadata (struct stat) of the files.
.IP "META_BS"
See BLOCKDATA_BS.
.IP "HARDLINK_PATH"
The path to the database that contains data about hardlinks.
.IP "HARDLINK_BS"
See BLOCKDATA_BS.
.IP "SYMLINK_PATH"
The path to the database that contains data about symlinks.
.IP "SYMLINK_BS"
See BLOCKDATA_BS.
.IP "FREELIST_PATH"
Only needed with the file_io backend, ignored otherwise. Is used to keep track of free (deleted) chunks of data.
.IP "FREELIST_BS"
Only needed with the file_io backend.
See BLOCKDATA_BS.
.IP LISTEN_IP=127.0.0.1
Specify the ip address where the lessfs management port binds to. Lessfs supports freeze and database defragmentation commands through this interface.
.IP LISTEN_PORT=100
Specifies the port where the management interface binds on. Default is port 100.
.IP MIN_SPACE_FREE=10
Specifies the minimum percentage or free space that has to be available. Lessfs suspends all I/O when less then the minumum required space is reached. This value defaults to 10%.
.IP MIN_SPACE_CLEAN=25
When MIN_SPACE_CLEAN is configured the program specified by CLEAN_PROGRAM will be executed. This program/script can be used to warn the administator or automatically delete old files. Please note that when CLEAN_PROGRAM is used with the file_io backend deleting files from lessfs will not free disk space on the underlying filesystem since file_io only marks deleted blocks as free in the freelist database.
.IP CLEAN_PROGRAM=/usr/local/bin/warn_me.sh
.IP HASHLEN = 24
The lenght of the hash that identifies the blocks of data measured in bytes. Minimum 20, maximum 32.
.br
The default value is 24 (192 bits).
.IP CACHESIZE=512
The cachesize megabytes that lessfs is allowed to use as write cache.
.IP COMMIT_INTERVAL=30
.IP MAX_THREADS=2
Do not set MAX_THREADS higher then the number of CPU cores available in the system. Lessfs will use MAX_THREADS CPU cores for LZO or QUICKLZ compression to compress MAX_THREADS data blocks in parallel. The performance will suffer when this number is set higher then the number of CPU cores that are available in the system.
.IP COMMIT_INTERVAL=30
Maximum age of entries that are kept in the cache. After this the cache will be written to disk.
.IP DYNAMIC_DEFRAGMENTATION=on
Enable tokyocabinets automatic defragmentation feature. Default = off when not specified.
.IP COREDUMPSIZE=25600000
Enable generation of coredumps for debugging. Default = off. Only usefull when lessfs is compiled with CFLAGS=-ggdb2
.IP SYNC_RELAX=0
Valid options are : 0 (default) , 1 or 2
.br
\fB0\fR Flush all caches in lessfs for an inode and sync the tokyocabinet databases to disk when fsync is called for an inode.
.br
\fB1\fR Do not sync the tokyocabinet databases to the disk when fsync is called on an inode. The inode data will be written directly to the databases. In case of a crash the databases themselves might not be committed to disk. This feature improves some types of I/O and is especially useful with NFS. There is a trade-off between more speed and the chance of possible loss of data. \fBUse with caution.\fR
.br
\fB2\fR Living on the edge. Do not flush the caches in lessfs and do not sync the tokyocabinet databases to disk. \fBUse with extreme caution.\fR
.IP ENCRYPT_DATA=off
.br
Enable data encryption. Requires lessfs to be configured with --with-crypto
.br
Valid options are : off (default) or on
.IP ENCRYPT_META=off
.br
Enable meta data encryption. Requires lessfs to be configured with --with-crypto and is only valid with ENCRYPT_DATA=on
.br
It is sometime usefull to disable meta data encryption. For example when one stores email messages in qmail/maildir format the name of the messages is not sensitive. To gain performance you might choose to encrypt only the data of the messages. \fBUse with caution.\fR
.br
Valid options are : on (default) or off
.IP ENABLE_TRANSACTIONS=on
Enable or disable transactions. Without transactions enabled lessfs will need to be repaired with fsck after a crash. The downside of enabling transactions is a mild performance impact.
.br
Valid options are : off (default) or on
.IP COMPRESSION=qlz
Valid options are disabled or none, qlz,lzo,gzip,bzip and deflate
.IP HASHNAME=TIGER192
Lessfs supports the following hashes: MHASH_SHA256, MHASH_SHA512, MHASH_WHIRLPOOL, MHASH_HAVAL256, MHASH_SNEFRU256, MHASH_RIPEMD256 and TIGER192. The tiger hash is used by default. The hash lenght can be anything between 24 and 64 bytes. Unlike the compression protocol, the hash and hash lenght can not be changed after formatting the filesystem.
.IP BLKSIZE=131072
When BLKSIZE is specified in the configuration file lessfs can be mounted without specifying options. In this case lessfs will use a number of default options that make sense in most cases. If you would like to specify the options yourself then remove this line from the configuration.
.br
Valid options are: 4096,8192,16384,32768,65536,131072
.IP PASSWORD=somepassword
When lessfs is used with encryption support enabled it is allowed though not recommended to specify the password in the configuration file. It is also possible to export the password in the environment with export PASSWORD=somepassword. Normally lessfs will prompt for a password when the filesystem is mounted. This is the default.
.IP DEBUG=2
Valid options are 0..5.
.br
\fB0\fR Disable logging with the exception of critical errors.
.br
\fB1\fR Enable logging of critical errors and warnings.
.br
\fB2\fR Enable logging of critical errors, warnings and informational messages.
.br
\fB3\fR Enable logging of critical errors, warnings, informational messages and debug messages.
.br
\fB>3\fR Enable logging of critical errors, warnings, informational messages and debug and other messages.
.SH COMPRESSION STATISTICS
Lessfs compression statistics can be obtained from the .lessfs/lessfs_stats file that is located in the root of the filesystem.
.SH DIAGNOSTICS
To debug the filesystem configure lessfs with : ./configure --enable-debug
.br
Change the value of DEBUG to 5 in lessfs.cfg before you run lessfs.
.PP
Email bug reports to
.BR [email protected] .
Be sure to include the word ``lessfsbug'' somewhere in the ``Subject:'' field.
.SH COPYRIGHT
Copyright (C) 2008-2009 Mark Ruijter
.PP
You can redistribute lessfs and/or modify it
under the terms of either
(1) the GNU General Public License as published by
the Free Software Foundation; or (2) obtain a commercial license
by contacting the Author.
You should have received a copy of the GNU General Public License
along with this program. If not, see \fB<http://www.gnu.org/licenses/>\fR.
.PP
lessfs is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
or FITNESS FOR A PARTICULAR PURPOSE.
See the GNU General Public License for more details.
.SH AUTHOR
.PP
Mark Ruijter <[email protected]>
.br