1 bdf5b5cd 2005-07-13 devnull .TH VENTI-FMT 8
3 bdf5b5cd 2005-07-13 devnull buildindex,
4 bdf5b5cd 2005-07-13 devnull checkarenas,
5 bdf5b5cd 2005-07-13 devnull checkindex,
7 bdf5b5cd 2005-07-13 devnull fmtarenas,
10 bdf5b5cd 2005-07-13 devnull fmtisect,
11 bdf5b5cd 2005-07-13 devnull syncindex \- prepare and maintain a venti server
12 bdf5b5cd 2005-07-13 devnull .SH SYNOPSIS
14 bdf5b5cd 2005-07-13 devnull .B venti/fmtarenas
20 bdf5b5cd 2005-07-13 devnull .I arenasize
24 bdf5b5cd 2005-07-13 devnull .I blocksize
29 bdf5b5cd 2005-07-13 devnull .B venti/fmtisect
35 bdf5b5cd 2005-07-13 devnull .I blocksize
40 72efb666 2005-07-24 devnull .B venti/fmtbloom
43 72efb666 2005-07-24 devnull .I nblocks
54 bdf5b5cd 2005-07-13 devnull .B venti/fmtindex
58 bdf5b5cd 2005-07-13 devnull .I venti.conf
60 bdf5b5cd 2005-07-13 devnull .B venti/conf
64 bdf5b5cd 2005-07-13 devnull .I partition
66 bdf5b5cd 2005-07-13 devnull .I configfile
68 bdf5b5cd 2005-07-13 devnull .if t .sp 0.5
70 bdf5b5cd 2005-07-13 devnull .B venti/buildindex
80 bdf5b5cd 2005-07-13 devnull .I venti.conf
82 bdf5b5cd 2005-07-13 devnull .B venti/checkindex
88 bdf5b5cd 2005-07-13 devnull .I blockcachesize
90 bdf5b5cd 2005-07-13 devnull .I venti.conf
93 bdf5b5cd 2005-07-13 devnull .B venti/checkarenas
98 bdf5b5cd 2005-07-13 devnull .SH DESCRIPTION
99 bdf5b5cd 2005-07-13 devnull These commands aid in the setup, maintenance, and debugging of
100 bdf5b5cd 2005-07-13 devnull venti servers.
102 d32deab1 2020-08-16 rsc .MR venti (7)
103 bdf5b5cd 2005-07-13 devnull for an overview of the venti system and
104 d32deab1 2020-08-16 rsc .MR venti (8)
105 bdf5b5cd 2005-07-13 devnull for an overview of the data structures used by the venti server.
107 bdf5b5cd 2005-07-13 devnull Note that the units for the various sizes in the following
108 bdf5b5cd 2005-07-13 devnull commands can be specified by appending
113 bdf5b5cd 2005-07-13 devnull to indicate kilobytes, megabytes, or gigabytes respectively.
114 bdf5b5cd 2005-07-13 devnull .SS Formatting
115 bdf5b5cd 2005-07-13 devnull To prepare a server for its initial use, the arena partitions and
116 bdf5b5cd 2005-07-13 devnull the index sections must be formatted individually, with
117 bdf5b5cd 2005-07-13 devnull .I fmtarenas
119 bdf5b5cd 2005-07-13 devnull .IR fmtisect .
120 bdf5b5cd 2005-07-13 devnull Then the
121 bdf5b5cd 2005-07-13 devnull collection of index sections must be combined into a venti
122 bdf5b5cd 2005-07-13 devnull index with
123 bdf5b5cd 2005-07-13 devnull .IR fmtindex .
125 bdf5b5cd 2005-07-13 devnull .I Fmtarenas
126 bdf5b5cd 2005-07-13 devnull formats the given
127 bdf5b5cd 2005-07-13 devnull .IR file ,
128 bdf5b5cd 2005-07-13 devnull typically a disk partition, into an arena partition.
129 bdf5b5cd 2005-07-13 devnull The arenas in the partition are given names of the form
130 bdf5b5cd 2005-07-13 devnull .IR name%d ,
133 bdf5b5cd 2005-07-13 devnull is replaced with a sequential number starting at 0.
135 bdf5b5cd 2005-07-13 devnull Options to
136 bdf5b5cd 2005-07-13 devnull .I fmtarenas
139 bdf5b5cd 2005-07-13 devnull .BI -a " arenasize
140 bdf5b5cd 2005-07-13 devnull The arenas are of
141 bdf5b5cd 2005-07-13 devnull .I arenasize
142 bdf5b5cd 2005-07-13 devnull bytes. The default is
143 bdf5b5cd 2005-07-13 devnull .BR 512M ,
144 bdf5b5cd 2005-07-13 devnull which was selected to provide a balance
145 bdf5b5cd 2005-07-13 devnull between the number of arenas and the ability to copy an arena to external
146 bdf5b5cd 2005-07-13 devnull media such as recordable CDs and tapes.
148 bdf5b5cd 2005-07-13 devnull .BI -b " blocksize
149 bdf5b5cd 2005-07-13 devnull The size, in bytes, for read and write operations to the file.
150 bdf5b5cd 2005-07-13 devnull The size is recorded in the file, and is used by applications that access the arenas.
151 bdf5b5cd 2005-07-13 devnull The default is
152 bdf5b5cd 2005-07-13 devnull .BR 8k .
155 bdf5b5cd 2005-07-13 devnull Create a `version 4' arena partition for backwards compatibility with old servers.
156 bdf5b5cd 2005-07-13 devnull The default is version 5, used by the current venti server.
159 bdf5b5cd 2005-07-13 devnull Do not zero the data sections of the arenas.
160 bdf5b5cd 2005-07-13 devnull Using this option reduces the formatting time
161 bdf5b5cd 2005-07-13 devnull but should only be used when it is known that the file was already zeroed.
162 bdf5b5cd 2005-07-13 devnull (Version 4 only; version 5 sections are not and do not need to be zeroed.)
165 bdf5b5cd 2005-07-13 devnull .I Fmtisect
166 bdf5b5cd 2005-07-13 devnull formats the given
167 bdf5b5cd 2005-07-13 devnull .IR file ,
168 bdf5b5cd 2005-07-13 devnull typically a disk partition, as a venti index section with the specified
169 bdf5b5cd 2005-07-13 devnull .IR name .
170 bdf5b5cd 2005-07-13 devnull Each of the index sections in a venti configuration must have a unique name.
172 bdf5b5cd 2005-07-13 devnull Options to
173 bdf5b5cd 2005-07-13 devnull .I fmtisect
176 bdf5b5cd 2005-07-13 devnull .BI -b " bucketsize
177 bdf5b5cd 2005-07-13 devnull The size of an index bucket, in bytes.
178 bdf5b5cd 2005-07-13 devnull All the index sections within a index must have the same bucket size.
179 bdf5b5cd 2005-07-13 devnull The default is
180 bdf5b5cd 2005-07-13 devnull .BR 8k .
183 bdf5b5cd 2005-07-13 devnull Create a `version 1' index section for backwards compatibility with old servers.
184 bdf5b5cd 2005-07-13 devnull The default is version 2, used by the current venti server.
187 bdf5b5cd 2005-07-13 devnull Do not zero the index.
188 bdf5b5cd 2005-07-13 devnull Using this option reduces the formatting time
189 bdf5b5cd 2005-07-13 devnull but should only be used when it is known that the file was already zeroed.
190 bdf5b5cd 2005-07-13 devnull (Version 1 only; version 2 sections are not and do not need to be zeroed.)
193 72efb666 2005-07-24 devnull .I Fmtbloom
194 72efb666 2005-07-24 devnull formats the given
196 64ef09ff 2011-05-10 rsc as a Bloom filter
198 d32deab1 2020-08-16 rsc .MR venti (7) ).
199 72efb666 2005-07-24 devnull The options are:
200 64ef09ff 2011-05-10 rsc .TF "\fL-s\fI size"
203 72efb666 2005-07-24 devnull .BI -n " nblock \fR| " -N " nhash
204 72efb666 2005-07-24 devnull The number of blocks expected to be indexed by the filter
205 72efb666 2005-07-24 devnull or the number of hash functions to use.
209 72efb666 2005-07-24 devnull is given, it is used, along with the total size of the filter,
210 72efb666 2005-07-24 devnull to compute an appropriate
211 72efb666 2005-07-24 devnull .IR nhash .
213 72efb666 2005-07-24 devnull .BI -s " size
214 64ef09ff 2011-05-10 rsc The size of the Bloom filter. The default is the total size of the file.
215 72efb666 2005-07-24 devnull In either case,
217 72efb666 2005-07-24 devnull is rounded down to a power of two.
222 72efb666 2005-07-24 devnull argument in the commands above can be of the form
223 72efb666 2005-07-24 devnull .IB file : lo - hi
224 72efb666 2005-07-24 devnull to specify a range of the file.
228 72efb666 2005-07-24 devnull are specified in bytes but can have the usual
233 72efb666 2005-07-24 devnull suffixes.
238 72efb666 2005-07-24 devnull may be omitted.
239 72efb666 2005-07-24 devnull This notation eliminates the need to
240 72efb666 2005-07-24 devnull partition raw disks on non-Plan 9 systems.
242 bdf5b5cd 2005-07-13 devnull .I Fmtindex
243 bdf5b5cd 2005-07-13 devnull reads the configuration file
244 bdf5b5cd 2005-07-13 devnull .I venti.conf
245 bdf5b5cd 2005-07-13 devnull and initializes the index sections to form a usable index structure.
246 bdf5b5cd 2005-07-13 devnull The arena files and index sections must have previously been formatted
248 bdf5b5cd 2005-07-13 devnull .I fmtarenas
250 bdf5b5cd 2005-07-13 devnull .I fmtisect
251 bdf5b5cd 2005-07-13 devnull respectively.
253 bdf5b5cd 2005-07-13 devnull The function of a venti index is to map a SHA1 fingerprint to a location
254 bdf5b5cd 2005-07-13 devnull in the data section of one of the arenas. The index is composed of
255 bdf5b5cd 2005-07-13 devnull blocks, each of which contains the mapping for a fixed range of possible
256 bdf5b5cd 2005-07-13 devnull fingerprint values.
257 bdf5b5cd 2005-07-13 devnull .I Fmtindex
258 bdf5b5cd 2005-07-13 devnull determines the mapping between SHA1 values and the blocks
259 bdf5b5cd 2005-07-13 devnull of the collection of index sections. Once this mapping has been determined,
260 bdf5b5cd 2005-07-13 devnull it cannot be changed without rebuilding the index.
261 bdf5b5cd 2005-07-13 devnull The basic assumption in the current implementation is that the index
262 bdf5b5cd 2005-07-13 devnull structure is sufficiently empty that individual blocks of the index will rarely
263 bdf5b5cd 2005-07-13 devnull overflow. The total size of the index should be about 2% to 10% of
264 bdf5b5cd 2005-07-13 devnull the total size of the arenas, but the exact percentage depends both on the
265 bdf5b5cd 2005-07-13 devnull index block size and the compressed size of blocks stored.
266 bdf5b5cd 2005-07-13 devnull See the discussion in
267 d32deab1 2020-08-16 rsc .MR venti (8)
268 bdf5b5cd 2005-07-13 devnull for more.
270 bdf5b5cd 2005-07-13 devnull .I Fmtindex
271 bdf5b5cd 2005-07-13 devnull also computes a mapping between a linear address space and
272 bdf5b5cd 2005-07-13 devnull the data section of the collection of arenas. The
274 bdf5b5cd 2005-07-13 devnull option can be used to add additional arenas to an index.
275 bdf5b5cd 2005-07-13 devnull To use this feature,
276 bdf5b5cd 2005-07-13 devnull add the new arenas to
277 bdf5b5cd 2005-07-13 devnull .I venti.conf
278 bdf5b5cd 2005-07-13 devnull after the existing arenas and then run
279 bdf5b5cd 2005-07-13 devnull .I fmtindex
280 bdf5b5cd 2005-07-13 devnull .BR -a .
282 bdf5b5cd 2005-07-13 devnull A copy of the above mappings is stored in the header for each of the index sections.
283 bdf5b5cd 2005-07-13 devnull These copies enable
284 bdf5b5cd 2005-07-13 devnull .I buildindex
285 bdf5b5cd 2005-07-13 devnull to restore a single index section without rebuilding the entire index.
287 bdf5b5cd 2005-07-13 devnull To make it easier to bootstrap servers, the configuration
288 bdf5b5cd 2005-07-13 devnull file can be stored in otherwise empty space
289 bdf5b5cd 2005-07-13 devnull at the beginning of any venti partitions using
290 bdf5b5cd 2005-07-13 devnull .IR conf .
291 bdf5b5cd 2005-07-13 devnull A partition so branded with a configuration file can
292 bdf5b5cd 2005-07-13 devnull be used in place of a configuration file when invoking any
293 bdf5b5cd 2005-07-13 devnull of the venti commands.
294 bdf5b5cd 2005-07-13 devnull By default,
296 bdf5b5cd 2005-07-13 devnull prints the configuration stored in
297 bdf5b5cd 2005-07-13 devnull .IR partition .
298 bdf5b5cd 2005-07-13 devnull When invoked with the
302 bdf5b5cd 2005-07-13 devnull reads a configuration file from
303 bdf5b5cd 2005-07-13 devnull .I configfile
304 bdf5b5cd 2005-07-13 devnull (or else standard input)
305 bdf5b5cd 2005-07-13 devnull and stores it in
306 bdf5b5cd 2005-07-13 devnull .IR partition .
307 bdf5b5cd 2005-07-13 devnull .SS Checking and Rebuilding
309 bdf5b5cd 2005-07-13 devnull .I Buildindex
310 bdf5b5cd 2005-07-13 devnull populates the index for the Venti system described in
311 bdf5b5cd 2005-07-13 devnull .IR venti.conf .
312 bdf5b5cd 2005-07-13 devnull The index must have previously been formatted using
313 bdf5b5cd 2005-07-13 devnull .IR fmtindex .
314 bdf5b5cd 2005-07-13 devnull This command is typically used to build a new index for a Venti
315 bdf5b5cd 2005-07-13 devnull system when the old index becomes too small, or to rebuild
316 bdf5b5cd 2005-07-13 devnull an index after media failure.
317 bdf5b5cd 2005-07-13 devnull Small errors in an index can usually be fixed with
318 cbc99e3e 2011-06-03 rsc .IR checkindex ,
320 cbc99e3e 2011-06-03 rsc .I checkindex
321 cbc99e3e 2011-06-03 rsc requires a large temporary workspace and
322 cbc99e3e 2011-06-03 rsc .I buildindex
325 bdf5b5cd 2005-07-13 devnull Options to
326 bdf5b5cd 2005-07-13 devnull .I buildindex
328 64ef09ff 2011-05-10 rsc .TF "\fL-M\fI imemsize"
332 cbc99e3e 2011-06-03 rsc Reinitialise the Bloom filter, if any.
335 cbc99e3e 2011-06-03 rsc `Dumb' mode; run all three passes.
337 cbc99e3e 2011-06-03 rsc .BI -i " isect
338 cbc99e3e 2011-06-03 rsc Only rebuild index section
340 cbc99e3e 2011-06-03 rsc may be repeated to rebuild multiple sections.
343 cbc99e3e 2011-06-03 rsc is special and just reads the arenas.
345 cbc99e3e 2011-06-03 rsc .BI -M " imemsize
346 bdf5b5cd 2005-07-13 devnull The amount of memory, in bytes, to use for caching raw disk accesses while running
347 bdf5b5cd 2005-07-13 devnull .IR buildindex .
348 bdf5b5cd 2005-07-13 devnull (This is not a property of the created index.)
349 cbc99e3e 2011-06-03 rsc The usual suffices apply.
350 cbc99e3e 2011-06-03 rsc The default is 256M.
353 bdf5b5cd 2005-07-13 devnull .I Checkindex
354 bdf5b5cd 2005-07-13 devnull examines the Venti index described in
355 bdf5b5cd 2005-07-13 devnull .IR venti.conf .
356 bdf5b5cd 2005-07-13 devnull The program detects various error conditions including:
357 bdf5b5cd 2005-07-13 devnull blocks that are not indexed, index entries for blocks that do not exist,
358 bdf5b5cd 2005-07-13 devnull and duplicate index entries.
359 bdf5b5cd 2005-07-13 devnull If requested, an attempt can be made to fix errors that are found.
363 bdf5b5cd 2005-07-13 devnull file, usually a disk partition, must be large enough to store a copy of the index.
364 bdf5b5cd 2005-07-13 devnull This temporary space is used to perform a merge sort of index entries
365 bdf5b5cd 2005-07-13 devnull generated by reading the arenas.
367 bdf5b5cd 2005-07-13 devnull Options to
368 bdf5b5cd 2005-07-13 devnull .I checkindex
371 bdf5b5cd 2005-07-13 devnull .BI -B " blockcachesize
372 bdf5b5cd 2005-07-13 devnull The amount of memory, in bytes, to use for caching raw disk accesses while running
373 bdf5b5cd 2005-07-13 devnull .IR checkindex .
374 bdf5b5cd 2005-07-13 devnull The default is 8k.
377 bdf5b5cd 2005-07-13 devnull Attempt to fix any errors that are found.
380 bdf5b5cd 2005-07-13 devnull .I Checkarenas
381 bdf5b5cd 2005-07-13 devnull examines the Venti arenas contained in the given
382 bdf5b5cd 2005-07-13 devnull .IR file .
383 bdf5b5cd 2005-07-13 devnull The program detects various error conditions, and optionally attempts
384 bdf5b5cd 2005-07-13 devnull to fix any errors that are found.
386 bdf5b5cd 2005-07-13 devnull Options to
387 bdf5b5cd 2005-07-13 devnull .I checkarenas
391 bdf5b5cd 2005-07-13 devnull For each arena, scan the entire data section.
392 bdf5b5cd 2005-07-13 devnull If this option is omitted, only the end section of
393 bdf5b5cd 2005-07-13 devnull the arena is examined.
396 bdf5b5cd 2005-07-13 devnull Attempt to fix any errors that are found.
399 bdf5b5cd 2005-07-13 devnull Increase the verbosity of output.
401 bdf5b5cd 2005-07-13 devnull .SH SOURCE
402 bdf5b5cd 2005-07-13 devnull .B \*9/src/cmd/venti/srv
403 bdf5b5cd 2005-07-13 devnull .SH SEE ALSO
404 d32deab1 2020-08-16 rsc .MR venti (7) ,
405 d32deab1 2020-08-16 rsc .MR venti (8)
406 bdf5b5cd 2005-07-13 devnull .SH BUGS
407 bdf5b5cd 2005-07-13 devnull .I Buildindex
408 bdf5b5cd 2005-07-13 devnull should allow an individual index section to be rebuilt.
