1 bdf5b5cd 2005-07-13 devnull .TH VENTI-FMT 8
3 bdf5b5cd 2005-07-13 devnull buildindex,
4 bdf5b5cd 2005-07-13 devnull checkarenas,
5 bdf5b5cd 2005-07-13 devnull checkindex,
7 bdf5b5cd 2005-07-13 devnull fmtarenas,
10 bdf5b5cd 2005-07-13 devnull syncindex \- prepare and maintain a venti server
11 bdf5b5cd 2005-07-13 devnull .SH SYNOPSIS
13 bdf5b5cd 2005-07-13 devnull .B venti/fmtarenas
19 bdf5b5cd 2005-07-13 devnull .I arenasize
23 bdf5b5cd 2005-07-13 devnull .I blocksize
28 bdf5b5cd 2005-07-13 devnull .B venti/fmtisect
34 bdf5b5cd 2005-07-13 devnull .I blocksize
39 bdf5b5cd 2005-07-13 devnull .B venti/fmtindex
43 bdf5b5cd 2005-07-13 devnull .I venti.conf
45 bdf5b5cd 2005-07-13 devnull .B venti/conf
49 bdf5b5cd 2005-07-13 devnull .I partition
51 bdf5b5cd 2005-07-13 devnull .I configfile
53 bdf5b5cd 2005-07-13 devnull .if t .sp 0.5
55 bdf5b5cd 2005-07-13 devnull .B venti/buildindex
58 bdf5b5cd 2005-07-13 devnull .I blockcachesize
63 bdf5b5cd 2005-07-13 devnull .I venti.conf
66 bdf5b5cd 2005-07-13 devnull .B venti/checkindex
72 bdf5b5cd 2005-07-13 devnull .I blockcachesize
74 bdf5b5cd 2005-07-13 devnull .I venti.conf
77 bdf5b5cd 2005-07-13 devnull .B venti/checkarenas
83 bdf5b5cd 2005-07-13 devnull .B venti/copy
93 bdf5b5cd 2005-07-13 devnull .SH DESCRIPTION
94 bdf5b5cd 2005-07-13 devnull These commands aid in the setup, maintenance, and debugging of
95 bdf5b5cd 2005-07-13 devnull venti servers.
97 bdf5b5cd 2005-07-13 devnull .IR venti (7)
98 bdf5b5cd 2005-07-13 devnull for an overview of the venti system and
99 bdf5b5cd 2005-07-13 devnull .IR venti (8)
100 bdf5b5cd 2005-07-13 devnull for an overview of the data structures used by the venti server.
102 bdf5b5cd 2005-07-13 devnull Note that the units for the various sizes in the following
103 bdf5b5cd 2005-07-13 devnull commands can be specified by appending
108 bdf5b5cd 2005-07-13 devnull to indicate kilobytes, megabytes, or gigabytes respectively.
109 bdf5b5cd 2005-07-13 devnull .SS Formatting
110 bdf5b5cd 2005-07-13 devnull To prepare a server for its initial use, the arena partitions and
111 bdf5b5cd 2005-07-13 devnull the index sections must be formatted individually, with
112 bdf5b5cd 2005-07-13 devnull .I fmtarenas
114 bdf5b5cd 2005-07-13 devnull .IR fmtisect .
115 bdf5b5cd 2005-07-13 devnull Then the
116 bdf5b5cd 2005-07-13 devnull collection of index sections must be combined into a venti
117 bdf5b5cd 2005-07-13 devnull index with
118 bdf5b5cd 2005-07-13 devnull .IR fmtindex .
120 bdf5b5cd 2005-07-13 devnull .I Fmtarenas
121 bdf5b5cd 2005-07-13 devnull formats the given
122 bdf5b5cd 2005-07-13 devnull .IR file ,
123 bdf5b5cd 2005-07-13 devnull typically a disk partition, into an arena partition.
124 bdf5b5cd 2005-07-13 devnull The arenas in the partition are given names of the form
125 bdf5b5cd 2005-07-13 devnull .IR name%d ,
128 bdf5b5cd 2005-07-13 devnull is replaced with a sequential number starting at 0.
130 bdf5b5cd 2005-07-13 devnull Options to
131 bdf5b5cd 2005-07-13 devnull .I fmtarenas
134 bdf5b5cd 2005-07-13 devnull .BI -a " arenasize
135 bdf5b5cd 2005-07-13 devnull The arenas are of
136 bdf5b5cd 2005-07-13 devnull .I arenasize
137 bdf5b5cd 2005-07-13 devnull bytes. The default is
138 bdf5b5cd 2005-07-13 devnull .BR 512M ,
139 bdf5b5cd 2005-07-13 devnull which was selected to provide a balance
140 bdf5b5cd 2005-07-13 devnull between the number of arenas and the ability to copy an arena to external
141 bdf5b5cd 2005-07-13 devnull media such as recordable CDs and tapes.
143 bdf5b5cd 2005-07-13 devnull .BI -b " blocksize
144 bdf5b5cd 2005-07-13 devnull The size, in bytes, for read and write operations to the file.
145 bdf5b5cd 2005-07-13 devnull The size is recorded in the file, and is used by applications that access the arenas.
146 bdf5b5cd 2005-07-13 devnull The default is
147 bdf5b5cd 2005-07-13 devnull .BR 8k .
150 bdf5b5cd 2005-07-13 devnull Create a `version 4' arena partition for backwards compatibility with old servers.
151 bdf5b5cd 2005-07-13 devnull The default is version 5, used by the current venti server.
154 bdf5b5cd 2005-07-13 devnull Do not zero the data sections of the arenas.
155 bdf5b5cd 2005-07-13 devnull Using this option reduces the formatting time
156 bdf5b5cd 2005-07-13 devnull but should only be used when it is known that the file was already zeroed.
157 bdf5b5cd 2005-07-13 devnull (Version 4 only; version 5 sections are not and do not need to be zeroed.)
160 bdf5b5cd 2005-07-13 devnull .I Fmtisect
161 bdf5b5cd 2005-07-13 devnull formats the given
162 bdf5b5cd 2005-07-13 devnull .IR file ,
163 bdf5b5cd 2005-07-13 devnull typically a disk partition, as a venti index section with the specified
164 bdf5b5cd 2005-07-13 devnull .IR name .
165 bdf5b5cd 2005-07-13 devnull Each of the index sections in a venti configuration must have a unique name.
167 bdf5b5cd 2005-07-13 devnull Options to
168 bdf5b5cd 2005-07-13 devnull .I fmtisect
171 bdf5b5cd 2005-07-13 devnull .BI -b " bucketsize
172 bdf5b5cd 2005-07-13 devnull The size of an index bucket, in bytes.
173 bdf5b5cd 2005-07-13 devnull All the index sections within a index must have the same bucket size.
174 bdf5b5cd 2005-07-13 devnull The default is
175 bdf5b5cd 2005-07-13 devnull .BR 8k .
178 bdf5b5cd 2005-07-13 devnull Create a `version 1' index section for backwards compatibility with old servers.
179 bdf5b5cd 2005-07-13 devnull The default is version 2, used by the current venti server.
182 bdf5b5cd 2005-07-13 devnull Do not zero the index.
183 bdf5b5cd 2005-07-13 devnull Using this option reduces the formatting time
184 bdf5b5cd 2005-07-13 devnull but should only be used when it is known that the file was already zeroed.
185 bdf5b5cd 2005-07-13 devnull (Version 1 only; version 2 sections are not and do not need to be zeroed.)
187 bdf5b5cd 2005-07-13 devnull .I Fmtindex
188 bdf5b5cd 2005-07-13 devnull reads the configuration file
189 bdf5b5cd 2005-07-13 devnull .I venti.conf
190 bdf5b5cd 2005-07-13 devnull and initializes the index sections to form a usable index structure.
191 bdf5b5cd 2005-07-13 devnull The arena files and index sections must have previously been formatted
193 bdf5b5cd 2005-07-13 devnull .I fmtarenas
195 bdf5b5cd 2005-07-13 devnull .I fmtisect
196 bdf5b5cd 2005-07-13 devnull respectively.
198 bdf5b5cd 2005-07-13 devnull The function of a venti index is to map a SHA1 fingerprint to a location
199 bdf5b5cd 2005-07-13 devnull in the data section of one of the arenas. The index is composed of
200 bdf5b5cd 2005-07-13 devnull blocks, each of which contains the mapping for a fixed range of possible
201 bdf5b5cd 2005-07-13 devnull fingerprint values.
202 bdf5b5cd 2005-07-13 devnull .I Fmtindex
203 bdf5b5cd 2005-07-13 devnull determines the mapping between SHA1 values and the blocks
204 bdf5b5cd 2005-07-13 devnull of the collection of index sections. Once this mapping has been determined,
205 bdf5b5cd 2005-07-13 devnull it cannot be changed without rebuilding the index.
206 bdf5b5cd 2005-07-13 devnull The basic assumption in the current implementation is that the index
207 bdf5b5cd 2005-07-13 devnull structure is sufficiently empty that individual blocks of the index will rarely
208 bdf5b5cd 2005-07-13 devnull overflow. The total size of the index should be about 2% to 10% of
209 bdf5b5cd 2005-07-13 devnull the total size of the arenas, but the exact percentage depends both on the
210 bdf5b5cd 2005-07-13 devnull index block size and the compressed size of blocks stored.
211 bdf5b5cd 2005-07-13 devnull See the discussion in
212 bdf5b5cd 2005-07-13 devnull .IR venti (8)
213 bdf5b5cd 2005-07-13 devnull for more.
215 bdf5b5cd 2005-07-13 devnull .I Fmtindex
216 bdf5b5cd 2005-07-13 devnull also computes a mapping between a linear address space and
217 bdf5b5cd 2005-07-13 devnull the data section of the collection of arenas. The
219 bdf5b5cd 2005-07-13 devnull option can be used to add additional arenas to an index.
220 bdf5b5cd 2005-07-13 devnull To use this feature,
221 bdf5b5cd 2005-07-13 devnull add the new arenas to
222 bdf5b5cd 2005-07-13 devnull .I venti.conf
223 bdf5b5cd 2005-07-13 devnull after the existing arenas and then run
224 bdf5b5cd 2005-07-13 devnull .I fmtindex
225 bdf5b5cd 2005-07-13 devnull .BR -a .
227 bdf5b5cd 2005-07-13 devnull A copy of the above mappings is stored in the header for each of the index sections.
228 bdf5b5cd 2005-07-13 devnull These copies enable
229 bdf5b5cd 2005-07-13 devnull .I buildindex
230 bdf5b5cd 2005-07-13 devnull to restore a single index section without rebuilding the entire index.
232 bdf5b5cd 2005-07-13 devnull To make it easier to bootstrap servers, the configuration
233 bdf5b5cd 2005-07-13 devnull file can be stored in otherwise empty space
234 bdf5b5cd 2005-07-13 devnull at the beginning of any venti partitions using
235 bdf5b5cd 2005-07-13 devnull .IR conf .
236 bdf5b5cd 2005-07-13 devnull A partition so branded with a configuration file can
237 bdf5b5cd 2005-07-13 devnull be used in place of a configuration file when invoking any
238 bdf5b5cd 2005-07-13 devnull of the venti commands.
239 bdf5b5cd 2005-07-13 devnull By default,
241 bdf5b5cd 2005-07-13 devnull prints the configuration stored in
242 bdf5b5cd 2005-07-13 devnull .IR partition .
243 bdf5b5cd 2005-07-13 devnull When invoked with the
247 bdf5b5cd 2005-07-13 devnull reads a configuration file from
248 bdf5b5cd 2005-07-13 devnull .I configfile
249 bdf5b5cd 2005-07-13 devnull (or else standard input)
250 bdf5b5cd 2005-07-13 devnull and stores it in
251 bdf5b5cd 2005-07-13 devnull .IR partition .
252 bdf5b5cd 2005-07-13 devnull .SS Checking and Rebuilding
254 bdf5b5cd 2005-07-13 devnull .I Buildindex
255 bdf5b5cd 2005-07-13 devnull populates the index for the Venti system described in
256 bdf5b5cd 2005-07-13 devnull .IR venti.conf .
257 bdf5b5cd 2005-07-13 devnull The index must have previously been formatted using
258 bdf5b5cd 2005-07-13 devnull .IR fmtindex .
259 bdf5b5cd 2005-07-13 devnull This command is typically used to build a new index for a Venti
260 bdf5b5cd 2005-07-13 devnull system when the old index becomes too small, or to rebuild
261 bdf5b5cd 2005-07-13 devnull an index after media failure.
262 bdf5b5cd 2005-07-13 devnull Small errors in an index can usually be fixed with
263 bdf5b5cd 2005-07-13 devnull .IR checkindex .
267 bdf5b5cd 2005-07-13 devnull file, usually a disk partition, must be large enough to store a copy of the index.
268 bdf5b5cd 2005-07-13 devnull This temporary space is used to perform a merge sort of index entries
269 bdf5b5cd 2005-07-13 devnull generated by reading the arenas.
271 bdf5b5cd 2005-07-13 devnull Options to
272 bdf5b5cd 2005-07-13 devnull .I buildindex
275 bdf5b5cd 2005-07-13 devnull .BI -B " blockcachesize
276 bdf5b5cd 2005-07-13 devnull The amount of memory, in bytes, to use for caching raw disk accesses while running
277 bdf5b5cd 2005-07-13 devnull .IR buildindex .
278 bdf5b5cd 2005-07-13 devnull (This is not a property of the created index.)
279 bdf5b5cd 2005-07-13 devnull The default is 8k.
282 bdf5b5cd 2005-07-13 devnull Do not zero the index.
283 bdf5b5cd 2005-07-13 devnull This option should only be used when it is known that the index was already zeroed.
284 bdf5b5cd 2005-07-13 devnull (Version 1 indexes only; see the discussion in
285 bdf5b5cd 2005-07-13 devnull .I fmtindex
289 bdf5b5cd 2005-07-13 devnull .I Checkindex
290 bdf5b5cd 2005-07-13 devnull examines the Venti index described in
291 bdf5b5cd 2005-07-13 devnull .IR venti.conf .
292 bdf5b5cd 2005-07-13 devnull The program detects various error conditions including:
293 bdf5b5cd 2005-07-13 devnull blocks that are not indexed, index entries for blocks that do not exist,
294 bdf5b5cd 2005-07-13 devnull and duplicate index entries.
295 bdf5b5cd 2005-07-13 devnull If requested, an attempt can be made to fix errors that are found.
299 bdf5b5cd 2005-07-13 devnull file, usually a disk partition, must be large enough to store a copy of the index.
300 bdf5b5cd 2005-07-13 devnull This temporary space is used to perform a merge sort of index entries
301 bdf5b5cd 2005-07-13 devnull generated by reading the arenas.
303 bdf5b5cd 2005-07-13 devnull Options to
304 bdf5b5cd 2005-07-13 devnull .I checkindex
307 bdf5b5cd 2005-07-13 devnull .BI -B " blockcachesize
308 bdf5b5cd 2005-07-13 devnull The amount of memory, in bytes, to use for caching raw disk accesses while running
309 bdf5b5cd 2005-07-13 devnull .IR checkindex .
310 bdf5b5cd 2005-07-13 devnull The default is 8k.
313 bdf5b5cd 2005-07-13 devnull Attempt to fix any errors that are found.
316 bdf5b5cd 2005-07-13 devnull .I Checkarenas
317 bdf5b5cd 2005-07-13 devnull examines the Venti arenas contained in the given
318 bdf5b5cd 2005-07-13 devnull .IR file .
319 bdf5b5cd 2005-07-13 devnull The program detects various error conditions, and optionally attempts
320 bdf5b5cd 2005-07-13 devnull to fix any errors that are found.
322 bdf5b5cd 2005-07-13 devnull Options to
323 bdf5b5cd 2005-07-13 devnull .I checkarenas
327 bdf5b5cd 2005-07-13 devnull For each arena, scan the entire data section.
328 bdf5b5cd 2005-07-13 devnull If this option is omitted, only the end section of
329 bdf5b5cd 2005-07-13 devnull the arena is examined.
332 bdf5b5cd 2005-07-13 devnull Attempt to fix any errors that are found.
335 bdf5b5cd 2005-07-13 devnull Increase the verbosity of output.
337 bdf5b5cd 2005-07-13 devnull .SH SOURCE
338 bdf5b5cd 2005-07-13 devnull .B \*9/src/cmd/venti/srv
339 bdf5b5cd 2005-07-13 devnull .SH SEE ALSO
340 bdf5b5cd 2005-07-13 devnull .IR venti (7),
341 bdf5b5cd 2005-07-13 devnull .IR venti (8)
342 bdf5b5cd 2005-07-13 devnull .SH BUGS
343 bdf5b5cd 2005-07-13 devnull .I Buildindex
344 bdf5b5cd 2005-07-13 devnull should allow an individual index section to be rebuilt.
345 bdf5b5cd 2005-07-13 devnull The merge sort could be performed in the space used to store the
346 bdf5b5cd 2005-07-13 devnull index rather than requiring a temporary file.
