1 1a8f27c3 2004-05-14 devnull .TH VENTIAUX 8
3 1a8f27c3 2004-05-14 devnull buildindex,
4 1a8f27c3 2004-05-14 devnull checkarenas,
5 1a8f27c3 2004-05-14 devnull checkindex,
8 1a8f27c3 2004-05-14 devnull fmtarenas,
10 1a8f27c3 2004-05-14 devnull fmtisect,
12 1a8f27c3 2004-05-14 devnull rdarenablocks,
14 1a8f27c3 2004-05-14 devnull wrarenablocks,
15 1a8f27c3 2004-05-14 devnull write \- Venti maintenance and debugging commands
16 1a8f27c3 2004-05-14 devnull .SH SYNOPSIS
17 1a8f27c3 2004-05-14 devnull .B venti/buildindex
20 1a8f27c3 2004-05-14 devnull .I blockcachesize
25 1a8f27c3 2004-05-14 devnull .I venti.config
28 1a8f27c3 2004-05-14 devnull .B venti/checkarenas
34 1a8f27c3 2004-05-14 devnull .B venti/checkindex
40 1a8f27c3 2004-05-14 devnull .I blockcachesize
42 1a8f27c3 2004-05-14 devnull .I venti.config
45 1a8f27c3 2004-05-14 devnull .B venti/conf
49 1a8f27c3 2004-05-14 devnull .I partition
51 1a8f27c3 2004-05-14 devnull .I configfile
54 1a8f27c3 2004-05-14 devnull .B venti/copy
65 1a8f27c3 2004-05-14 devnull .B venti/fmtarenas
71 1a8f27c3 2004-05-14 devnull .I arenasize
75 1a8f27c3 2004-05-14 devnull .I blocksize
80 1a8f27c3 2004-05-14 devnull .B venti/fmtindex
84 1a8f27c3 2004-05-14 devnull .I venti.config
86 1a8f27c3 2004-05-14 devnull .B venti/fmtisect
92 1a8f27c3 2004-05-14 devnull .I blocksize
97 1a8f27c3 2004-05-14 devnull .B venti/rdarena
101 1a8f27c3 2004-05-14 devnull .I arenapart
102 1a8f27c3 2004-05-14 devnull .I arenaname
104 1a8f27c3 2004-05-14 devnull .B venti/read
109 1a8f27c3 2004-05-14 devnull .I score
114 1a8f27c3 2004-05-14 devnull .B venti/wrarena
117 1a8f27c3 2004-05-14 devnull .I fileoffset
123 1a8f27c3 2004-05-14 devnull .I arenafile
125 1a8f27c3 2004-05-14 devnull .I clumpoffset
128 1a8f27c3 2004-05-14 devnull .B venti/write
140 1a8f27c3 2004-05-14 devnull .SH DESCRIPTION
141 1a8f27c3 2004-05-14 devnull These commands aid in the setup, maintenance, and debugging of
142 1a8f27c3 2004-05-14 devnull Venti servers.
144 1a8f27c3 2004-05-14 devnull .IR venti (8)
146 1a8f27c3 2004-05-14 devnull .IR venti.conf (6)
147 1a8f27c3 2004-05-14 devnull for an overview of the data structures stored by Venti.
149 1a8f27c3 2004-05-14 devnull Note that the units for the various sizes in the following
150 1a8f27c3 2004-05-14 devnull commands can be specified by appending
155 1a8f27c3 2004-05-14 devnull to indicate kilobytes, megabytes, or gigabytes respectively.
157 1a8f27c3 2004-05-14 devnull .I Buildindex
158 1a8f27c3 2004-05-14 devnull populates the index for the Venti system described in
159 1a8f27c3 2004-05-14 devnull .IR venti.config .
160 1a8f27c3 2004-05-14 devnull The index must have previously been formatted using
161 1a8f27c3 2004-05-14 devnull .IR fmtindex .
162 1a8f27c3 2004-05-14 devnull This command is typically used to build a new index for a Venti
163 1a8f27c3 2004-05-14 devnull system when the old index becomes too small, or to rebuild
164 1a8f27c3 2004-05-14 devnull an index after media failure.
165 1a8f27c3 2004-05-14 devnull Small errors in an index can usually be fixed with
166 1a8f27c3 2004-05-14 devnull .IR checkindex .
170 1a8f27c3 2004-05-14 devnull file, usually a disk partition, must be large enough to store a copy of the index.
171 1a8f27c3 2004-05-14 devnull This temporary space is used to perform a merge sort of index entries
172 1a8f27c3 2004-05-14 devnull generated by reading the arenas.
174 1a8f27c3 2004-05-14 devnull Options to
175 1a8f27c3 2004-05-14 devnull .I buildindex
178 1a8f27c3 2004-05-14 devnull .BI -B " blockcachesize
179 1a8f27c3 2004-05-14 devnull The amount of memory, in bytes, to use for caching raw disk accesses while running
180 1a8f27c3 2004-05-14 devnull .IR buildindex .
181 1a8f27c3 2004-05-14 devnull (This is not a property of the created index.)
182 1a8f27c3 2004-05-14 devnull The default is 8k.
185 1a8f27c3 2004-05-14 devnull Do not zero the index.
186 1a8f27c3 2004-05-14 devnull This option should only be used when it is known that the index was already zeroed.
189 1a8f27c3 2004-05-14 devnull .I Checkarenas
190 1a8f27c3 2004-05-14 devnull examines the Venti arenas contained in the given
191 1a8f27c3 2004-05-14 devnull .IR file .
192 1a8f27c3 2004-05-14 devnull The program detects various error conditions, and optionally attempts
193 1a8f27c3 2004-05-14 devnull to fix any errors that are found.
195 1a8f27c3 2004-05-14 devnull Options to
196 1a8f27c3 2004-05-14 devnull .I checkarenas
200 1a8f27c3 2004-05-14 devnull For each arena, scan the entire data section.
201 1a8f27c3 2004-05-14 devnull If this option is omitted, only the end section of
202 1a8f27c3 2004-05-14 devnull the arena is examined.
205 1a8f27c3 2004-05-14 devnull Attempt to fix any errors that are found.
208 1a8f27c3 2004-05-14 devnull Increase the verbosity of output.
211 1a8f27c3 2004-05-14 devnull .I Checkindex
212 1a8f27c3 2004-05-14 devnull examines the Venti index described in
213 1a8f27c3 2004-05-14 devnull .IR venti.config .
214 1a8f27c3 2004-05-14 devnull The program detects various error conditions including:
215 1a8f27c3 2004-05-14 devnull blocks that are not indexed, index entries for blocks that do not exist,
216 1a8f27c3 2004-05-14 devnull and duplicate index entries.
217 1a8f27c3 2004-05-14 devnull If requested, an attempt can be made to fix errors that are found.
221 1a8f27c3 2004-05-14 devnull file, usually a disk partition, must be large enough to store a copy of the index.
222 1a8f27c3 2004-05-14 devnull This temporary space is used to perform a merge sort of index entries
223 1a8f27c3 2004-05-14 devnull generated by reading the arenas.
225 1a8f27c3 2004-05-14 devnull Options to
226 1a8f27c3 2004-05-14 devnull .I checkindex
229 1a8f27c3 2004-05-14 devnull .BI -B " blockcachesize
230 1a8f27c3 2004-05-14 devnull The amount of memory, in bytes, to use for caching raw disk accesses while running
231 1a8f27c3 2004-05-14 devnull .IR checkindex .
232 1a8f27c3 2004-05-14 devnull The default is 8k.
235 1a8f27c3 2004-05-14 devnull Attempt to fix any errors that are found.
238 1a8f27c3 2004-05-14 devnull .I Fmtarenas
239 1a8f27c3 2004-05-14 devnull formats the given
240 1a8f27c3 2004-05-14 devnull .IR file ,
241 1a8f27c3 2004-05-14 devnull typically a disk partition, into a number of
244 1a8f27c3 2004-05-14 devnull The arenas are given names of the form
245 1a8f27c3 2004-05-14 devnull .IR name%d ,
248 1a8f27c3 2004-05-14 devnull is replaced with a sequential number starting at 0.
250 1a8f27c3 2004-05-14 devnull Options to
251 1a8f27c3 2004-05-14 devnull .I fmtarenas
254 1a8f27c3 2004-05-14 devnull .BI -a " arenasize
255 1a8f27c3 2004-05-14 devnull The arenas are of
256 1a8f27c3 2004-05-14 devnull .I arenasize
257 1a8f27c3 2004-05-14 devnull bytes. The default is 512 megabytes, which was selected to provide a balance
258 1a8f27c3 2004-05-14 devnull between the number of arenas and the ability to copy an arena to external
259 1a8f27c3 2004-05-14 devnull media such as recordable CDs and tapes.
261 1a8f27c3 2004-05-14 devnull .BI -b " blocksize
262 1a8f27c3 2004-05-14 devnull The size, in bytes, for read and write operations to the file.
263 1a8f27c3 2004-05-14 devnull The size is recorded in the file, and is used by applications that access the arenas.
264 1a8f27c3 2004-05-14 devnull The default is 8k.
267 1a8f27c3 2004-05-14 devnull Do not zero the data sections of the arenas.
268 1a8f27c3 2004-05-14 devnull Using this option reduces the formatting time
269 1a8f27c3 2004-05-14 devnull but should only be used when it is known that the file was already zeroed.
271 1a8f27c3 2004-05-14 devnull .I Fmtindex
272 1a8f27c3 2004-05-14 devnull takes the
273 1a8f27c3 2004-05-14 devnull .IR venti.conf (6)
275 1a8f27c3 2004-05-14 devnull .I venti.config
276 1a8f27c3 2004-05-14 devnull and initializes the index sections to form a usable index structure.
277 1a8f27c3 2004-05-14 devnull The arena files and index sections must have previously been formatted
279 1a8f27c3 2004-05-14 devnull .I fmtarenas
281 1a8f27c3 2004-05-14 devnull .I fmtisect
282 1a8f27c3 2004-05-14 devnull respectively.
284 1a8f27c3 2004-05-14 devnull The function of a Venti index is to map a SHA1 fingerprint to a location
285 1a8f27c3 2004-05-14 devnull in the data section of one of the arenas. The index is composed of
286 1a8f27c3 2004-05-14 devnull blocks, each of which contains the mapping for a fixed range of possible
287 1a8f27c3 2004-05-14 devnull fingerprint values.
288 1a8f27c3 2004-05-14 devnull .I Fmtindex
289 1a8f27c3 2004-05-14 devnull determines the mapping between SHA1 values and the blocks
290 1a8f27c3 2004-05-14 devnull of the collection of index sections. Once this mapping has been determined,
291 1a8f27c3 2004-05-14 devnull it cannot be changed without rebuilding the index.
292 1a8f27c3 2004-05-14 devnull The basic assumption in the current implementation is that the index
293 1a8f27c3 2004-05-14 devnull structure is sufficiently empty that individual blocks of the index will rarely
294 1a8f27c3 2004-05-14 devnull overflow. The total size of the index should be about 2% to 10% of
295 1a8f27c3 2004-05-14 devnull the total size of the arenas, but the exact depends both the index block size
296 1a8f27c3 2004-05-14 devnull and the compressed size of block stored to Venti.
298 1a8f27c3 2004-05-14 devnull .I Fmtindex
299 1a8f27c3 2004-05-14 devnull also computes a mapping between a linear address space and
300 1a8f27c3 2004-05-14 devnull the data section of the collection of arenas. The
302 1a8f27c3 2004-05-14 devnull option can be used to add additional arenas to an index.
303 1a8f27c3 2004-05-14 devnull To use this feature,
304 1a8f27c3 2004-05-14 devnull add the new arenas to
305 1a8f27c3 2004-05-14 devnull .I venti.config
306 1a8f27c3 2004-05-14 devnull after the existing arenas and then run
307 1a8f27c3 2004-05-14 devnull .I fmtindex
308 1a8f27c3 2004-05-14 devnull .BR -a .
310 1a8f27c3 2004-05-14 devnull A copy of the above mappings is stored in the header for each of the index sections.
311 1a8f27c3 2004-05-14 devnull These copies enable
312 1a8f27c3 2004-05-14 devnull .I buildindex
313 1a8f27c3 2004-05-14 devnull to restore a single index section without rebuilding the entire index.
315 1a8f27c3 2004-05-14 devnull .I Fmtisect
316 1a8f27c3 2004-05-14 devnull formats the given
317 1a8f27c3 2004-05-14 devnull .IR file ,
318 1a8f27c3 2004-05-14 devnull typically a disk partition, as a Venti index section with the specified
319 1a8f27c3 2004-05-14 devnull .IR name .
320 1a8f27c3 2004-05-14 devnull One or more formatted index sections are combined into a Venti
321 1a8f27c3 2004-05-14 devnull index using
322 1a8f27c3 2004-05-14 devnull .IR fmtindex .
323 1a8f27c3 2004-05-14 devnull Each of the index sections within an index must have a unique name.
325 1a8f27c3 2004-05-14 devnull Options to
326 1a8f27c3 2004-05-14 devnull .I fmtisect
329 1a8f27c3 2004-05-14 devnull .BI -b " blocksize
330 1a8f27c3 2004-05-14 devnull The size, in bytes, for read and write operations to the file.
331 1a8f27c3 2004-05-14 devnull All the index sections within a index must have the same block size.
332 1a8f27c3 2004-05-14 devnull The default is 8k.
335 1a8f27c3 2004-05-14 devnull Do not zero the index.
336 1a8f27c3 2004-05-14 devnull Using this option reduces the formatting time
337 1a8f27c3 2004-05-14 devnull but should only be used when it is known that the file was already zeroed.
340 1a8f27c3 2004-05-14 devnull .I Rdarena
341 1a8f27c3 2004-05-14 devnull extracts the named
342 1a8f27c3 2004-05-14 devnull .I arena
343 1a8f27c3 2004-05-14 devnull from the arena partition
344 1a8f27c3 2004-05-14 devnull .I arenapart
345 1a8f27c3 2004-05-14 devnull and writes this arena to standard output.
346 1a8f27c3 2004-05-14 devnull This command is typically used to back up an arena to external media.
349 1a8f27c3 2004-05-14 devnull option generates more verbose output on standard error.
351 1a8f27c3 2004-05-14 devnull .I Wrarena
352 1a8f27c3 2004-05-14 devnull writes the blocks contained in the arena
353 1a8f27c3 2004-05-14 devnull .I arenafile
354 1a8f27c3 2004-05-14 devnull (typically, the output of
355 1a8f27c3 2004-05-14 devnull .IR rdarena )
356 1a8f27c3 2004-05-14 devnull to a Venti server.
357 1a8f27c3 2004-05-14 devnull It is typically used to reinitialize a Venti server from backups of the arenas.
358 1a8f27c3 2004-05-14 devnull For example,
361 1a8f27c3 2004-05-14 devnull venti/rdarena /dev/sdC0/arenas arena.0 >external.media
362 1a8f27c3 2004-05-14 devnull venti/wrarena -h venti2 external.media
365 1a8f27c3 2004-05-14 devnull writes the blocks contained in
366 1a8f27c3 2004-05-14 devnull .B arena.0
367 1a8f27c3 2004-05-14 devnull to the Venti server
368 1a8f27c3 2004-05-14 devnull .B venti2
369 1a8f27c3 2004-05-14 devnull (typically not the one using
370 1a8f27c3 2004-05-14 devnull .BR /dev/sdC0/arenas ).
374 1a8f27c3 2004-05-14 devnull option specifies that the arena starts at byte
375 1a8f27c3 2004-05-14 devnull .I fileoffset
376 1a8f27c3 2004-05-14 devnull (default
379 1a8f27c3 2004-05-14 devnull .I arenafile .
380 1a8f27c3 2004-05-14 devnull This is useful for reading directly from
381 1a8f27c3 2004-05-14 devnull the Venti arena partition:
384 1a8f27c3 2004-05-14 devnull venti/wrarena -h venti2 -o 335872 /dev/sdC0/arenas
387 1a8f27c3 2004-05-14 devnull (In this example, 335872 is the offset shown in the Venti
388 1a8f27c3 2004-05-14 devnull server's index list (344064) minus one block (8192).
389 1a8f27c3 2004-05-14 devnull You will need to substitute your own arena offsets
390 1a8f27c3 2004-05-14 devnull and block size.)
392 1a8f27c3 2004-05-14 devnull Finally, the optional
393 1a8f27c3 2004-05-14 devnull .I offset
394 1a8f27c3 2004-05-14 devnull argument specifies that the writing should begin with the
395 1a8f27c3 2004-05-14 devnull clump starting at
396 1a8f27c3 2004-05-14 devnull .I offset
397 1a8f27c3 2004-05-14 devnull within the arena.
398 1a8f27c3 2004-05-14 devnull .I Wrarena
399 1a8f27c3 2004-05-14 devnull prints the offset it stopped at (because there were no more data blocks).
400 1a8f27c3 2004-05-14 devnull This could be used to incrementally back up a Venti server
401 1a8f27c3 2004-05-14 devnull to another Venti server:
404 1a8f27c3 2004-05-14 devnull last=`{cat last}
405 1a8f27c3 2004-05-14 devnull venti/wrarena -h venti2 -o 335872 /dev/sdC0/arenas $last >output
406 1a8f27c3 2004-05-14 devnull awk '/^end offset/ { print $3 }' offset >last
409 1a8f27c3 2004-05-14 devnull Of course, one would need to add wrapper code to keep track
410 1a8f27c3 2004-05-14 devnull of which arenas have been processed.
412 1a8f27c3 2004-05-14 devnull .B /sys/src/cmd/venti/backup.example
413 1a8f27c3 2004-05-14 devnull for a version that does this.
417 1a8f27c3 2004-05-14 devnull .I write
418 1a8f27c3 2004-05-14 devnull read and write blocks from a running Venti server.
419 1a8f27c3 2004-05-14 devnull They are intended to ease debugging of the server.
420 1a8f27c3 2004-05-14 devnull The default
422 1a8f27c3 2004-05-14 devnull is the environment variable
423 1a8f27c3 2004-05-14 devnull .BR $venti ,
424 1a8f27c3 2004-05-14 devnull followed by the network metaname
425 1a8f27c3 2004-05-14 devnull .BR $venti .
428 1a8f27c3 2004-05-14 devnull is the decimal type of block to be read or written.
431 1a8f27c3 2004-05-14 devnull is specified for
432 1a8f27c3 2004-05-14 devnull .I read ,
433 1a8f27c3 2004-05-14 devnull all types are tried, and a command-line is printed to
434 1a8f27c3 2004-05-14 devnull show the type that eventually worked.
437 1a8f27c3 2004-05-14 devnull is specified for
438 1a8f27c3 2004-05-14 devnull .I write ,
439 1a8f27c3 2004-05-14 devnull .B VtDataType
441 1a8f27c3 2004-05-14 devnull is used.
443 1a8f27c3 2004-05-14 devnull reads the block named by
444 1a8f27c3 2004-05-14 devnull .I score
445 1a8f27c3 2004-05-14 devnull (a SHA1 hash)
446 1a8f27c3 2004-05-14 devnull from the Venti server and writes it to standard output.
447 1a8f27c3 2004-05-14 devnull .I Write
448 1a8f27c3 2004-05-14 devnull reads a block from standard input and attempts to write
449 1a8f27c3 2004-05-14 devnull it to the Venti server.
450 1a8f27c3 2004-05-14 devnull If successful, it prints the score of the block on the server.
453 1a8f27c3 2004-05-14 devnull walks the entire tree of blocks rooted at
454 1a8f27c3 2004-05-14 devnull .I score ,
455 1a8f27c3 2004-05-14 devnull copying all the blocks visited during the walk from
456 1a8f27c3 2004-05-14 devnull the Venti server at network address
458 1a8f27c3 2004-05-14 devnull to the Venti server at network address
459 1a8f27c3 2004-05-14 devnull .I dst .
462 1a8f27c3 2004-05-14 devnull (a decimal block type for
463 1a8f27c3 2004-05-14 devnull .IR score )
464 1a8f27c3 2004-05-14 devnull is omitted, all types will be tried in sequence
465 1a8f27c3 2004-05-14 devnull until one is found that works.
468 1a8f27c3 2004-05-14 devnull flag runs the copy in ``fast'' mode: if a block is already on
469 1a8f27c3 2004-05-14 devnull .IR dst ,
470 1a8f27c3 2004-05-14 devnull the walk does not descend below it, on the assumption that all its
471 1a8f27c3 2004-05-14 devnull children are also already on
472 1a8f27c3 2004-05-14 devnull .IR dst .
473 1a8f27c3 2004-05-14 devnull Without this flag, the copy often transfers many times more
474 1a8f27c3 2004-05-14 devnull data than necessary.
476 1a8f27c3 2004-05-14 devnull To make it easier to bootstrap servers, the configuration
477 1a8f27c3 2004-05-14 devnull file can be stored at the beginning of any Venti partitions using
478 1a8f27c3 2004-05-14 devnull .IR conf .
479 1a8f27c3 2004-05-14 devnull A partition so branded with a configuration file can
480 1a8f27c3 2004-05-14 devnull be used in place of a configuration file when invoking any
481 1a8f27c3 2004-05-14 devnull of the venti commands.
482 1a8f27c3 2004-05-14 devnull By default,
484 1a8f27c3 2004-05-14 devnull prints the configuration stored in
485 1a8f27c3 2004-05-14 devnull .IR partition .
486 1a8f27c3 2004-05-14 devnull When invoked with the
490 1a8f27c3 2004-05-14 devnull reads a configuration file from
491 1a8f27c3 2004-05-14 devnull .I configfile
492 1a8f27c3 2004-05-14 devnull (or else standard input)
493 1a8f27c3 2004-05-14 devnull and stores it in
494 1a8f27c3 2004-05-14 devnull .IR partition .
495 1a8f27c3 2004-05-14 devnull .SH SOURCE
496 1a8f27c3 2004-05-14 devnull .B /sys/src/cmd/venti
497 1a8f27c3 2004-05-14 devnull .SH "SEE ALSO"
498 1a8f27c3 2004-05-14 devnull .IR venti (8),
499 1a8f27c3 2004-05-14 devnull .IR venti.conf (6)
500 1a8f27c3 2004-05-14 devnull .SH BUGS
501 1a8f27c3 2004-05-14 devnull .I Buildindex
502 1a8f27c3 2004-05-14 devnull should allow an individual index section to be rebuilt.
503 1a8f27c3 2004-05-14 devnull The merge sort could be performed in the space used to store the
504 1a8f27c3 2004-05-14 devnull index rather than requiring a temporary file.
