1 bdf5b5cd 2005-07-13 devnull .TH VENTI-FMT 8
3 bdf5b5cd 2005-07-13 devnull buildindex,
4 bdf5b5cd 2005-07-13 devnull checkarenas,
5 bdf5b5cd 2005-07-13 devnull checkindex,
7 bdf5b5cd 2005-07-13 devnull fmtarenas,
10 bdf5b5cd 2005-07-13 devnull fmtisect,
11 bdf5b5cd 2005-07-13 devnull syncindex \- prepare and maintain a venti server
12 bdf5b5cd 2005-07-13 devnull .SH SYNOPSIS
14 bdf5b5cd 2005-07-13 devnull .B venti/fmtarenas
20 bdf5b5cd 2005-07-13 devnull .I arenasize
24 bdf5b5cd 2005-07-13 devnull .I blocksize
29 bdf5b5cd 2005-07-13 devnull .B venti/fmtisect
35 bdf5b5cd 2005-07-13 devnull .I blocksize
40 72efb666 2005-07-24 devnull .B venti/fmtbloom
43 72efb666 2005-07-24 devnull .I nblocks
54 bdf5b5cd 2005-07-13 devnull .B venti/fmtindex
58 bdf5b5cd 2005-07-13 devnull .I venti.conf
60 bdf5b5cd 2005-07-13 devnull .B venti/conf
64 bdf5b5cd 2005-07-13 devnull .I partition
66 bdf5b5cd 2005-07-13 devnull .I configfile
68 bdf5b5cd 2005-07-13 devnull .if t .sp 0.5
70 bdf5b5cd 2005-07-13 devnull .B venti/buildindex
73 bdf5b5cd 2005-07-13 devnull .I blockcachesize
78 bdf5b5cd 2005-07-13 devnull .I venti.conf
81 bdf5b5cd 2005-07-13 devnull .B venti/checkindex
87 bdf5b5cd 2005-07-13 devnull .I blockcachesize
89 bdf5b5cd 2005-07-13 devnull .I venti.conf
92 bdf5b5cd 2005-07-13 devnull .B venti/checkarenas
98 bdf5b5cd 2005-07-13 devnull .B venti/copy
104 bdf5b5cd 2005-07-13 devnull .I score
108 bdf5b5cd 2005-07-13 devnull .SH DESCRIPTION
109 bdf5b5cd 2005-07-13 devnull These commands aid in the setup, maintenance, and debugging of
110 bdf5b5cd 2005-07-13 devnull venti servers.
112 bdf5b5cd 2005-07-13 devnull .IR venti (7)
113 bdf5b5cd 2005-07-13 devnull for an overview of the venti system and
114 bdf5b5cd 2005-07-13 devnull .IR venti (8)
115 bdf5b5cd 2005-07-13 devnull for an overview of the data structures used by the venti server.
117 bdf5b5cd 2005-07-13 devnull Note that the units for the various sizes in the following
118 bdf5b5cd 2005-07-13 devnull commands can be specified by appending
123 bdf5b5cd 2005-07-13 devnull to indicate kilobytes, megabytes, or gigabytes respectively.
124 bdf5b5cd 2005-07-13 devnull .SS Formatting
125 bdf5b5cd 2005-07-13 devnull To prepare a server for its initial use, the arena partitions and
126 bdf5b5cd 2005-07-13 devnull the index sections must be formatted individually, with
127 bdf5b5cd 2005-07-13 devnull .I fmtarenas
129 bdf5b5cd 2005-07-13 devnull .IR fmtisect .
130 bdf5b5cd 2005-07-13 devnull Then the
131 bdf5b5cd 2005-07-13 devnull collection of index sections must be combined into a venti
132 bdf5b5cd 2005-07-13 devnull index with
133 bdf5b5cd 2005-07-13 devnull .IR fmtindex .
135 bdf5b5cd 2005-07-13 devnull .I Fmtarenas
136 bdf5b5cd 2005-07-13 devnull formats the given
137 bdf5b5cd 2005-07-13 devnull .IR file ,
138 bdf5b5cd 2005-07-13 devnull typically a disk partition, into an arena partition.
139 bdf5b5cd 2005-07-13 devnull The arenas in the partition are given names of the form
140 bdf5b5cd 2005-07-13 devnull .IR name%d ,
143 bdf5b5cd 2005-07-13 devnull is replaced with a sequential number starting at 0.
145 bdf5b5cd 2005-07-13 devnull Options to
146 bdf5b5cd 2005-07-13 devnull .I fmtarenas
149 bdf5b5cd 2005-07-13 devnull .BI -a " arenasize
150 bdf5b5cd 2005-07-13 devnull The arenas are of
151 bdf5b5cd 2005-07-13 devnull .I arenasize
152 bdf5b5cd 2005-07-13 devnull bytes. The default is
153 bdf5b5cd 2005-07-13 devnull .BR 512M ,
154 bdf5b5cd 2005-07-13 devnull which was selected to provide a balance
155 bdf5b5cd 2005-07-13 devnull between the number of arenas and the ability to copy an arena to external
156 bdf5b5cd 2005-07-13 devnull media such as recordable CDs and tapes.
158 bdf5b5cd 2005-07-13 devnull .BI -b " blocksize
159 bdf5b5cd 2005-07-13 devnull The size, in bytes, for read and write operations to the file.
160 bdf5b5cd 2005-07-13 devnull The size is recorded in the file, and is used by applications that access the arenas.
161 bdf5b5cd 2005-07-13 devnull The default is
162 bdf5b5cd 2005-07-13 devnull .BR 8k .
165 bdf5b5cd 2005-07-13 devnull Create a `version 4' arena partition for backwards compatibility with old servers.
166 bdf5b5cd 2005-07-13 devnull The default is version 5, used by the current venti server.
169 bdf5b5cd 2005-07-13 devnull Do not zero the data sections of the arenas.
170 bdf5b5cd 2005-07-13 devnull Using this option reduces the formatting time
171 bdf5b5cd 2005-07-13 devnull but should only be used when it is known that the file was already zeroed.
172 bdf5b5cd 2005-07-13 devnull (Version 4 only; version 5 sections are not and do not need to be zeroed.)
175 bdf5b5cd 2005-07-13 devnull .I Fmtisect
176 bdf5b5cd 2005-07-13 devnull formats the given
177 bdf5b5cd 2005-07-13 devnull .IR file ,
178 bdf5b5cd 2005-07-13 devnull typically a disk partition, as a venti index section with the specified
179 bdf5b5cd 2005-07-13 devnull .IR name .
180 bdf5b5cd 2005-07-13 devnull Each of the index sections in a venti configuration must have a unique name.
182 bdf5b5cd 2005-07-13 devnull Options to
183 bdf5b5cd 2005-07-13 devnull .I fmtisect
186 bdf5b5cd 2005-07-13 devnull .BI -b " bucketsize
187 bdf5b5cd 2005-07-13 devnull The size of an index bucket, in bytes.
188 bdf5b5cd 2005-07-13 devnull All the index sections within a index must have the same bucket size.
189 bdf5b5cd 2005-07-13 devnull The default is
190 bdf5b5cd 2005-07-13 devnull .BR 8k .
193 bdf5b5cd 2005-07-13 devnull Create a `version 1' index section for backwards compatibility with old servers.
194 bdf5b5cd 2005-07-13 devnull The default is version 2, used by the current venti server.
197 bdf5b5cd 2005-07-13 devnull Do not zero the index.
198 bdf5b5cd 2005-07-13 devnull Using this option reduces the formatting time
199 bdf5b5cd 2005-07-13 devnull but should only be used when it is known that the file was already zeroed.
200 bdf5b5cd 2005-07-13 devnull (Version 1 only; version 2 sections are not and do not need to be zeroed.)
203 72efb666 2005-07-24 devnull .I Fmtbloom
204 72efb666 2005-07-24 devnull formats the given
206 72efb666 2005-07-24 devnull as a bloom filter
208 72efb666 2005-07-24 devnull .IR venti (7)).
209 72efb666 2005-07-24 devnull The options are:
211 72efb666 2005-07-24 devnull .BI -n " nblock \fR| " -N " nhash
212 72efb666 2005-07-24 devnull The number of blocks expected to be indexed by the filter
213 72efb666 2005-07-24 devnull or the number of hash functions to use.
217 72efb666 2005-07-24 devnull is given, it is used, along with the total size of the filter,
218 72efb666 2005-07-24 devnull to compute an appropriate
219 72efb666 2005-07-24 devnull .IR nhash .
221 72efb666 2005-07-24 devnull .BI -s " size
222 72efb666 2005-07-24 devnull The size of the bloom filter. The default is the total size of the file.
223 72efb666 2005-07-24 devnull In either case,
225 72efb666 2005-07-24 devnull is rounded down to a power of two.
230 72efb666 2005-07-24 devnull argument in the commands above can be of the form
231 72efb666 2005-07-24 devnull .IB file : lo - hi
232 72efb666 2005-07-24 devnull to specify a range of the file.
236 72efb666 2005-07-24 devnull are specified in bytes but can have the usual
241 72efb666 2005-07-24 devnull suffixes.
246 72efb666 2005-07-24 devnull may be omitted.
247 72efb666 2005-07-24 devnull This notation eliminates the need to
248 72efb666 2005-07-24 devnull partition raw disks on non-Plan 9 systems.
250 bdf5b5cd 2005-07-13 devnull .I Fmtindex
251 bdf5b5cd 2005-07-13 devnull reads the configuration file
252 bdf5b5cd 2005-07-13 devnull .I venti.conf
253 bdf5b5cd 2005-07-13 devnull and initializes the index sections to form a usable index structure.
254 bdf5b5cd 2005-07-13 devnull The arena files and index sections must have previously been formatted
256 bdf5b5cd 2005-07-13 devnull .I fmtarenas
258 bdf5b5cd 2005-07-13 devnull .I fmtisect
259 bdf5b5cd 2005-07-13 devnull respectively.
261 bdf5b5cd 2005-07-13 devnull The function of a venti index is to map a SHA1 fingerprint to a location
262 bdf5b5cd 2005-07-13 devnull in the data section of one of the arenas. The index is composed of
263 bdf5b5cd 2005-07-13 devnull blocks, each of which contains the mapping for a fixed range of possible
264 bdf5b5cd 2005-07-13 devnull fingerprint values.
265 bdf5b5cd 2005-07-13 devnull .I Fmtindex
266 bdf5b5cd 2005-07-13 devnull determines the mapping between SHA1 values and the blocks
267 bdf5b5cd 2005-07-13 devnull of the collection of index sections. Once this mapping has been determined,
268 bdf5b5cd 2005-07-13 devnull it cannot be changed without rebuilding the index.
269 bdf5b5cd 2005-07-13 devnull The basic assumption in the current implementation is that the index
270 bdf5b5cd 2005-07-13 devnull structure is sufficiently empty that individual blocks of the index will rarely
271 bdf5b5cd 2005-07-13 devnull overflow. The total size of the index should be about 2% to 10% of
272 bdf5b5cd 2005-07-13 devnull the total size of the arenas, but the exact percentage depends both on the
273 bdf5b5cd 2005-07-13 devnull index block size and the compressed size of blocks stored.
274 bdf5b5cd 2005-07-13 devnull See the discussion in
275 bdf5b5cd 2005-07-13 devnull .IR venti (8)
276 bdf5b5cd 2005-07-13 devnull for more.
278 bdf5b5cd 2005-07-13 devnull .I Fmtindex
279 bdf5b5cd 2005-07-13 devnull also computes a mapping between a linear address space and
280 bdf5b5cd 2005-07-13 devnull the data section of the collection of arenas. The
282 bdf5b5cd 2005-07-13 devnull option can be used to add additional arenas to an index.
283 bdf5b5cd 2005-07-13 devnull To use this feature,
284 bdf5b5cd 2005-07-13 devnull add the new arenas to
285 bdf5b5cd 2005-07-13 devnull .I venti.conf
286 bdf5b5cd 2005-07-13 devnull after the existing arenas and then run
287 bdf5b5cd 2005-07-13 devnull .I fmtindex
288 bdf5b5cd 2005-07-13 devnull .BR -a .
290 bdf5b5cd 2005-07-13 devnull A copy of the above mappings is stored in the header for each of the index sections.
291 bdf5b5cd 2005-07-13 devnull These copies enable
292 bdf5b5cd 2005-07-13 devnull .I buildindex
293 bdf5b5cd 2005-07-13 devnull to restore a single index section without rebuilding the entire index.
295 bdf5b5cd 2005-07-13 devnull To make it easier to bootstrap servers, the configuration
296 bdf5b5cd 2005-07-13 devnull file can be stored in otherwise empty space
297 bdf5b5cd 2005-07-13 devnull at the beginning of any venti partitions using
298 bdf5b5cd 2005-07-13 devnull .IR conf .
299 bdf5b5cd 2005-07-13 devnull A partition so branded with a configuration file can
300 bdf5b5cd 2005-07-13 devnull be used in place of a configuration file when invoking any
301 bdf5b5cd 2005-07-13 devnull of the venti commands.
302 bdf5b5cd 2005-07-13 devnull By default,
304 bdf5b5cd 2005-07-13 devnull prints the configuration stored in
305 bdf5b5cd 2005-07-13 devnull .IR partition .
306 bdf5b5cd 2005-07-13 devnull When invoked with the
310 bdf5b5cd 2005-07-13 devnull reads a configuration file from
311 bdf5b5cd 2005-07-13 devnull .I configfile
312 bdf5b5cd 2005-07-13 devnull (or else standard input)
313 bdf5b5cd 2005-07-13 devnull and stores it in
314 bdf5b5cd 2005-07-13 devnull .IR partition .
315 bdf5b5cd 2005-07-13 devnull .SS Checking and Rebuilding
317 bdf5b5cd 2005-07-13 devnull .I Buildindex
318 bdf5b5cd 2005-07-13 devnull populates the index for the Venti system described in
319 bdf5b5cd 2005-07-13 devnull .IR venti.conf .
320 bdf5b5cd 2005-07-13 devnull The index must have previously been formatted using
321 bdf5b5cd 2005-07-13 devnull .IR fmtindex .
322 bdf5b5cd 2005-07-13 devnull This command is typically used to build a new index for a Venti
323 bdf5b5cd 2005-07-13 devnull system when the old index becomes too small, or to rebuild
324 bdf5b5cd 2005-07-13 devnull an index after media failure.
325 bdf5b5cd 2005-07-13 devnull Small errors in an index can usually be fixed with
326 bdf5b5cd 2005-07-13 devnull .IR checkindex .
330 bdf5b5cd 2005-07-13 devnull file, usually a disk partition, must be large enough to store a copy of the index.
331 bdf5b5cd 2005-07-13 devnull This temporary space is used to perform a merge sort of index entries
332 bdf5b5cd 2005-07-13 devnull generated by reading the arenas.
334 bdf5b5cd 2005-07-13 devnull Options to
335 bdf5b5cd 2005-07-13 devnull .I buildindex
338 bdf5b5cd 2005-07-13 devnull .BI -B " blockcachesize
339 bdf5b5cd 2005-07-13 devnull The amount of memory, in bytes, to use for caching raw disk accesses while running
340 bdf5b5cd 2005-07-13 devnull .IR buildindex .
341 bdf5b5cd 2005-07-13 devnull (This is not a property of the created index.)
342 bdf5b5cd 2005-07-13 devnull The default is 8k.
345 bdf5b5cd 2005-07-13 devnull Do not zero the index.
346 bdf5b5cd 2005-07-13 devnull This option should only be used when it is known that the index was already zeroed.
347 bdf5b5cd 2005-07-13 devnull (Version 1 indexes only; see the discussion in
348 bdf5b5cd 2005-07-13 devnull .I fmtindex
352 bdf5b5cd 2005-07-13 devnull .I Checkindex
353 bdf5b5cd 2005-07-13 devnull examines the Venti index described in
354 bdf5b5cd 2005-07-13 devnull .IR venti.conf .
355 bdf5b5cd 2005-07-13 devnull The program detects various error conditions including:
356 bdf5b5cd 2005-07-13 devnull blocks that are not indexed, index entries for blocks that do not exist,
357 bdf5b5cd 2005-07-13 devnull and duplicate index entries.
358 bdf5b5cd 2005-07-13 devnull If requested, an attempt can be made to fix errors that are found.
362 bdf5b5cd 2005-07-13 devnull file, usually a disk partition, must be large enough to store a copy of the index.
363 bdf5b5cd 2005-07-13 devnull This temporary space is used to perform a merge sort of index entries
364 bdf5b5cd 2005-07-13 devnull generated by reading the arenas.
366 bdf5b5cd 2005-07-13 devnull Options to
367 bdf5b5cd 2005-07-13 devnull .I checkindex
370 bdf5b5cd 2005-07-13 devnull .BI -B " blockcachesize
371 bdf5b5cd 2005-07-13 devnull The amount of memory, in bytes, to use for caching raw disk accesses while running
372 bdf5b5cd 2005-07-13 devnull .IR checkindex .
373 bdf5b5cd 2005-07-13 devnull The default is 8k.
376 bdf5b5cd 2005-07-13 devnull Attempt to fix any errors that are found.
379 bdf5b5cd 2005-07-13 devnull .I Checkarenas
380 bdf5b5cd 2005-07-13 devnull examines the Venti arenas contained in the given
381 bdf5b5cd 2005-07-13 devnull .IR file .
382 bdf5b5cd 2005-07-13 devnull The program detects various error conditions, and optionally attempts
383 bdf5b5cd 2005-07-13 devnull to fix any errors that are found.
385 bdf5b5cd 2005-07-13 devnull Options to
386 bdf5b5cd 2005-07-13 devnull .I checkarenas
390 bdf5b5cd 2005-07-13 devnull For each arena, scan the entire data section.
391 bdf5b5cd 2005-07-13 devnull If this option is omitted, only the end section of
392 bdf5b5cd 2005-07-13 devnull the arena is examined.
395 bdf5b5cd 2005-07-13 devnull Attempt to fix any errors that are found.
398 bdf5b5cd 2005-07-13 devnull Increase the verbosity of output.
400 bdf5b5cd 2005-07-13 devnull .SH SOURCE
401 bdf5b5cd 2005-07-13 devnull .B \*9/src/cmd/venti/srv
402 bdf5b5cd 2005-07-13 devnull .SH SEE ALSO
403 bdf5b5cd 2005-07-13 devnull .IR venti (7),
404 bdf5b5cd 2005-07-13 devnull .IR venti (8)
405 bdf5b5cd 2005-07-13 devnull .SH BUGS
406 bdf5b5cd 2005-07-13 devnull .I Buildindex
407 bdf5b5cd 2005-07-13 devnull should allow an individual index section to be rebuilt.
408 bdf5b5cd 2005-07-13 devnull The merge sort could be performed in the space used to store the
409 bdf5b5cd 2005-07-13 devnull index rather than requiring a temporary file.
