Blob


1 .TH VENTI 8
2 .SH NAME
3 venti \- archival storage server
4 .SH SYNOPSIS
5 .B venti/venti
6 [
7 .B -Ldsw
8 ]
9 [
10 .B -a
11 .I address
12 ]
13 [
14 .B -B
15 .I blockcachesize
16 ]
17 [
18 .B -c
19 .I config
20 ]
21 .PP
22 .B " "
23 [
24 .B -C
25 .I lumpcachesize
26 ]
27 [
28 .B -h
29 .I httpaddress
30 ]
31 [
32 .B -I
33 .I indexcachesize
34 ]
35 [
36 .B -W
37 .I webroot
38 ]
39 .SH DESCRIPTION
40 Venti is a SHA1-addressed archival storage server.
41 See
42 .IR venti (7)
43 for a full introduction to the system.
44 This page documents the structure and operation of the server.
45 .PP
46 A venti server requires multiple disks or disk partitions,
47 each of which must be properly formatted before the server
48 can be run.
49 .SS Disk
50 The venti server maintains three disk structures, typically
51 stored on raw disk partitions:
52 the append-only
53 .IR "data log" ,
54 which holds, in sequential order,
55 the contents of every block written to the server;
56 the
57 .IR index ,
58 which helps locate a block in the data log given its score;
59 and optionally the
60 .IR "bloom filter" ,
61 a concise summary of which scores are present in the index.
62 The data log is the primary storage.
63 To improve the robustness, it should be stored on
64 a device that provides RAID functionality.
65 The index and the bloom filter are optimizations
66 employed to access the data log efficiently and can be rebuilt
67 if lost or damaged.
68 .PP
69 The data log is logically split into sections called
70 .IR arenas ,
71 typically sized for easy offline backup
72 (e.g., 500MB).
73 A data log may comprise many disks, each storing
74 one or more arenas.
75 Such disks are called
76 .IR "arena partitions" .
77 Arena partitions are filled in the order given in the configuration.
78 .PP
79 The index is logically split into block-sized pieces called
80 .IR buckets ,
81 each of which is responsible for a particular range of scores.
82 An index may be split across many disks, each storing many buckets.
83 Such disks are called
84 .IR "index sections" .
85 .PP
86 The index must be sized so that no bucket is full.
87 When a bucket fills, the server must be shut down and
88 the index made larger.
89 Since scores appear random, each bucket will contain
90 approximately the same number of entries.
91 Index entries are 40 bytes long. Assuming that a typical block
92 being written to the server is 8192 bytes and compresses to 4096
93 bytes, the active index is expected to be about 1% of
94 the active data log.
95 Storing smaller blocks increases the relative index footprint;
96 storing larger blocks decreases it.
97 To allow variation in both block size and the random distribution
98 of scores to buckets, the suggested index size is 5% of
99 the active data log.
100 .PP
101 The (optional) bloom filter is a large bitmap that is stored on disk but
102 also kept completely in memory while the venti server runs.
103 It helps the venti server efficiently detect scores that are
104 .I not
105 already stored in the index.
106 The bloom filter starts out zeroed.
107 Each score recorded in the bloom filter is hashed to choose
108 .I nhash
109 bits to set in the bloom filter.
110 A score is definitely not stored in the index of any of its
111 .I nhash
112 bits are not set.
113 The bloom filter thus has two parameters:
114 .I nhash
115 (maximum 32)
116 and the total bitmap size
117 (maximum 512MB, 2\s-2\u32\d\s+2 bits).
118 .PP
119 The bloom filter should be sized so that
120 .I nhash
121 \(mu
122 .I nblock
123 \(<=
124 0.7 \(mu
125 .IR b ,
126 where
127 .I nblock
128 is the expected number of blocks stored on the server
129 and
130 .I b
131 is the bitmap size in bits.
132 The false positive rate of the bloom filter when sized
133 this way is approximately 2\s-2\u\-\fInblock\fR\d\s+2.
134 .I Nhash
135 less than 10 are not very useful;
136 .I nhash
137 greater than 24 are probably a waste of memory.
138 .I Fmtbloom
139 (see
140 .IR venti-fmt (8))
141 can be given either
142 .I nhash
143 or
144 .IR nblock ;
145 if given
146 .IR nblock ,
147 it will derive an appropriate
148 .IR nhash .
149 .SS Memory
150 Venti can make effective use of large amounts of memory
151 for various caches.
152 .PP
153 The
154 .I "lump cache
155 holds recently-accessed venti data blocks, which the server refers to as
156 .IR lumps .
157 The lump cache should be at least 1MB but can profitably be much larger.
158 The lump cache can be thought of as the level-1 cache:
159 read requests handled by the lump cache can
160 be served instantly.
161 .PP
162 The
163 .I "block cache
164 holds recently-accessed
165 .I disk
166 blocks from the arena partitions.
167 The block cache needs to be able to simultaneously hold two blocks
168 from each arena plus four blocks for the currently-filling arena.
169 The block cache can be thought of as the level-2 cache:
170 read requests handled by the block cache are slower than those
171 handled by the lump cache, since the lump data must be extracted
172 from the raw disk blocks and possibly decompressed, but no
173 disk accesses are necessary.
174 .PP
175 The
176 .I "index cache
177 holds recently-accessed or prefetched
178 index entries.
179 The index cache needs to be able to hold index entries
180 for three or four arenas, at least, in order for prefetching
181 to work properly. Each index entry is 50 bytes.
182 Assuming 500MB arenas of
183 128,000 blocks that are 4096 bytes each after compression,
184 the minimum index cache size is about 6MB.
185 The index cache can be thought of as the level-3 cache:
186 read requests handled by the index cache must still go
187 to disk to fetch the arena blocks, but the costly random
188 access to the index is avoided.
189 .PP
190 The size of the index cache determines how long venti
191 can sustain its `burst' write throughput, during which time
192 the only disk accesses on the critical path
193 are sequential writes to the arena partitions.
194 For example, if you want to be able to sustain 10MB/s
195 for an hour, you need enough index cache to hold entries
196 for 36GB of blocks. Assuming 8192-byte blocks,
197 you need room for almost five million index entries.
198 Since index entries are 50 bytes each, you need 250MB
199 of index cache.
200 If the background index update process can make a single
201 pass through the index in an hour, which is possible,
202 then you can sustain the 10MB/s indefinitely (at least until
203 the arenas are all filled).
204 .PP
205 The
206 .I "bloom filter
207 requires memory equal to its size on disk,
208 as discussed above.
209 .PP
210 A reasonable starting allocation is to
211 divide memory equally (in thirds) between
212 the bloom filter, the index cache, and the lump and block caches;
213 the third of memory allocated to the lump and block caches
214 should be split unevenly, with more (say, two thirds)
215 going to the block cache.
216 .SS Network
217 The venti server announces two network services, one
218 (conventionally TCP port
219 .BR venti ,
220 17034) serving
221 the venti protocol as described in
222 .IR venti (7),
223 and one serving HTTP
224 (conventionally TCP port
225 .BR venti ,
226 80).
227 .PP
228 The venti web server provides the following
229 URLs for accessing status information:
230 .TP
231 .B /index
232 A summary of the usage of the arenas and index sections.
233 .TP
234 .B /xindex
235 An XML version of
236 .BR /index .
237 .TP
238 .B /storage
239 Brief storage totals.
240 .TP
241 .BI /set/ variable
242 The current integer value of
243 .IR variable .
244 Variables are:
245 .BR compress ,
246 whether or not to compress blocks
247 (for debugging);
248 .BR logging ,
249 whether to write entries to the debugging logs;
250 .BR stats ,
251 whether to collect run-time statistics;
252 .BR icachesleeptime ,
253 the time in milliseconds between successive updates
254 of megabytes of the index cache;
255 .BR arenasumsleeptime ,
256 the time in milliseconds between reads while
257 checksumming an arena in the background.
258 The two sleep times should be (but are not) managed by venti;
259 they exist to provide more experience with their effects.
260 The other variables exist only for debugging and
261 performance measurement.
262 .TP
263 .BI /set/ variable / value
264 Set
265 .I variable
266 to
267 .IR value .
268 .TP
269 .BI /graph/ name / param / param / \fR...
270 A PNG image graphing the named run-time statistic over time.
271 The details of names and parameters are undocumented;
272 see
273 .B httpd.c
274 in the venti sources.
275 .TP
276 .B /log
277 A list of all debugging logs present in the server's memory.
278 .TP
279 .BI /log/ name
280 The contents of the debugging log with the given
281 .IR name .
282 .TP
283 .B /flushicache
284 Force venti to begin flushing the index cache to disk.
285 The request response will not be sent until the flush
286 has completed.
287 .TP
288 .B /flushdcache
289 Force venti to begin flushing the arena block cache to disk.
290 The request response will not be sent until the flush
291 has completed.
292 .PD
293 .PP
294 Requests for other files are served by consulting a
295 directory named in the configuration file
296 (see
297 .B webroot
298 below).
299 .SS Configuration File
300 A venti configuration file
301 enumerates the various index sections and
302 arenas that constitute a venti system.
303 The components are indicated by the name of the file, typically
304 a disk partition, in which they reside. The configuration
305 file is the only location that file names are used. Internally,
306 venti uses the names assigned when the components were formatted
307 with
308 .I fmtarenas
309 or
310 .I fmtisect
311 (see
312 .IR venti-fmt (8)).
313 In particular, only the configuration needs to be
314 changed if a component is moved to a different file.
315 .PP
316 The configuration file consists of lines in the form described below.
317 Lines starting with
318 .B #
319 are comments.
320 .TP
321 .BI index " name
322 Names the index for the system.
323 .TP
324 .BI arenas " file
325 .I File
326 is an arena partition, formatted using
327 .IR fmtarenas .
328 .TP
329 .BI isect " file
330 .I File
331 is an index section, formatted using
332 .IR fmtisect .
333 .TP
334 .BI bloom " file
335 .I File
336 is a bloom filter, formatted using
337 .IR fmtbloom .
338 .PD
339 .PP
340 After formatting a venti system using
341 .IR fmtindex ,
342 the order of arenas and index sections should not be changed.
343 Additional arenas can be appended to the configuration;
344 run
345 .I fmtindex
346 with the
347 .B -a
348 flag to update the index.
349 .PP
350 The configuration file also holds configuration parameters
351 for the venti server itself.
352 These are:
353 .TF httpaddr netaddr
354 .TP
355 .BI mem " size
356 lump cache size
357 .TP
358 .BI bcmem " size
359 block cache size
360 .TP
361 .BI icmem " size
362 index cache size
363 .TP
364 .BI addr " netaddr
365 network address to announce venti service
366 (default
367 .BR tcp!*!venti )
368 .TP
369 .BI httpaddr " netaddr
370 network address to announce HTTP service
371 (default
372 .BR tcp!*!http )
373 .TP
374 .B queuewrites
375 queue writes in memory
376 (default is not to queue)
377 .TP
378 .BI webroot " dir
379 directory tree containing files for HTTP server
380 to consult for unrecognized URLs
381 .PD
382 .PP
383 The units for the various cache sizes above can be specified by appending a
384 .LR k ,
385 .LR m ,
386 or
387 .LR g
388 (case-insensitive)
389 to indicate kilobytes, megabytes, or gigabytes respectively.
390 .PP
391 The
392 .I file
393 name in the configuration lines above can be of the form
394 .IB file : lo - hi
395 to specify a range of the file.
396 .I Lo
397 and
398 .I hi
399 are specified in bytes but can have the usual
400 .BI k ,
401 .BI m ,
402 or
403 .B g
404 suffixes.
405 Either
406 .I lo
407 or
408 .I hi
409 may be omitted.
410 This notation eliminates the need to
411 partition raw disks on non-Plan 9 systems.
412 .SS Command Line
413 Many of the options to Venti duplicate parameters that
414 can be specified in the configuration file.
415 The command line options override those found in a
416 configuration file.
417 Additional options are:
418 .TP
419 .BI -c " config
420 The server configuration file
421 (default
422 .BR venti.conf )
423 .TP
424 .B -d
425 Produce various debugging information on standard error.
426 Implies
427 .BR -s .
428 .TP
429 .B -L
430 Enable logging. By default all logging is disabled.
431 Logging slows server operation considerably.
432 .TP
433 .B -s
434 Do not run in the background.
435 Normally,
436 the foreground process will exit once the Venti server
437 is initialized and ready for connections.
438 .PD
439 .SH EXAMPLE
440 A simple configuration:
441 .IP
442 .EX
443 % cat venti.conf
444 index main
445 isect /tmp/disks/isect0
446 isect /tmp/disks/isect1
447 arenas /tmp/disks/arenas
448 bloom /tmp/disks/bloom
449 mem 10M
450 bcmem 20M
451 icmem 30M
453 .EE
454 .PP
455 Format the index sections, the arena partition, and
456 finally the main index:
457 .IP
458 .EX
459 % venti/fmtisect isect0. /tmp/disks/isect0 &
460 % venti/fmtisect isect1. /tmp/disks/isect1 &
461 % venti/fmtarenas arenas0. /tmp/disks/arenas &
462 % venti/fmtbloom /tmp/disks/bloom &
463 % wait
464 % venti/fmtindex venti.conf
466 .EE
467 .PP
468 Start the server and check the storage statistics:
469 .IP
470 .EX
471 % venti/venti
472 % hget http://$sysname/storage
473 .EE
474 .SH SOURCE
475 .B \*9/src/cmd/venti/srv
476 .SH "SEE ALSO"
477 .IR venti (1),
478 .IR venti (3),
479 .IR venti (7),
480 .IR venti-backup (8)
481 .IR venti-fmt (8)
482 .br
483 Sean Quinlan and Sean Dorward,
484 ``Venti: a new approach to archival storage'',
485 .I "Usenix Conference on File and Storage Technologies" ,
486 2002.
487 .SH BUGS
488 Setting up a venti server is too complicated.
489 .PP
490 Venti should not require the user to decide how to
491 partition its memory usage.
492 .PP
493 Users of shells other than
494 .IR rc (1)
495 will not be able to use the program names shown.
496 One solution is to define
497 .B "V=$PLAN9/bin/venti"
498 and then substitute
499 .B $V/
500 for
501 .B venti/
502 in the paths above.