3 venti \- archival storage server
41 is a SHA1-addressed archival storage server.
44 for a full introduction to the system.
45 This page documents the structure and operation of the server.
47 A venti server requires multiple disks or disk partitions,
48 each of which must be properly formatted before the server
51 The venti server maintains three disk structures, typically
52 stored on raw disk partitions:
55 which holds, in sequential order,
56 the contents of every block written to the server;
59 which helps locate a block in the data log given its score;
62 a concise summary of which scores are present in the index.
63 The data log is the primary storage.
64 To improve the robustness, it should be stored on
65 a device that provides RAID functionality.
66 The index and the bloom filter are optimizations
67 employed to access the data log efficiently and can be rebuilt
70 The data log is logically split into sections called
72 typically sized for easy offline backup
74 A data log may comprise many disks, each storing
77 .IR "arena partitions" .
78 Arena partitions are filled in the order given in the configuration.
80 The index is logically split into block-sized pieces called
82 each of which is responsible for a particular range of scores.
83 An index may be split across many disks, each storing many buckets.
85 .IR "index sections" .
87 The index must be sized so that no bucket is full.
88 When a bucket fills, the server must be shut down and
89 the index made larger.
90 Since scores appear random, each bucket will contain
91 approximately the same number of entries.
92 Index entries are 40 bytes long. Assuming that a typical block
93 being written to the server is 8192 bytes and compresses to 4096
94 bytes, the active index is expected to be about 1% of
96 Storing smaller blocks increases the relative index footprint;
97 storing larger blocks decreases it.
98 To allow variation in both block size and the random distribution
99 of scores to buckets, the suggested index size is 5% of
102 The (optional) bloom filter is a large bitmap that is stored on disk but
103 also kept completely in memory while the venti server runs.
104 It helps the venti server efficiently detect scores that are
106 already stored in the index.
107 The bloom filter starts out zeroed.
108 Each score recorded in the bloom filter is hashed to choose
110 bits to set in the bloom filter.
111 A score is definitely not stored in the index of any of its
114 The bloom filter thus has two parameters:
117 and the total bitmap size
118 (maximum 512MB, 2\s-2\u32\d\s+2 bits).
120 The bloom filter should be sized so that
129 is the expected number of blocks stored on the server
132 is the bitmap size in bits.
133 The false positive rate of the bloom filter when sized
134 this way is approximately 2\s-2\u\-\fInblock\fR\d\s+2.
136 less than 10 are not very useful;
138 greater than 24 are probably a waste of memory.
148 it will derive an appropriate
151 Venti can make effective use of large amounts of memory
156 holds recently-accessed venti data blocks, which the server refers to as
158 The lump cache should be at least 1MB but can profitably be much larger.
159 The lump cache can be thought of as the level-1 cache:
160 read requests handled by the lump cache can
165 holds recently-accessed
167 blocks from the arena partitions.
168 The block cache needs to be able to simultaneously hold two blocks
169 from each arena plus four blocks for the currently-filling arena.
170 The block cache can be thought of as the level-2 cache:
171 read requests handled by the block cache are slower than those
172 handled by the lump cache, since the lump data must be extracted
173 from the raw disk blocks and possibly decompressed, but no
174 disk accesses are necessary.
178 holds recently-accessed or prefetched
180 The index cache needs to be able to hold index entries
181 for three or four arenas, at least, in order for prefetching
182 to work properly. Each index entry is 50 bytes.
183 Assuming 500MB arenas of
184 128,000 blocks that are 4096 bytes each after compression,
185 the minimum index cache size is about 6MB.
186 The index cache can be thought of as the level-3 cache:
187 read requests handled by the index cache must still go
188 to disk to fetch the arena blocks, but the costly random
189 access to the index is avoided.
191 The size of the index cache determines how long venti
192 can sustain its `burst' write throughput, during which time
193 the only disk accesses on the critical path
194 are sequential writes to the arena partitions.
195 For example, if you want to be able to sustain 10MB/s
196 for an hour, you need enough index cache to hold entries
197 for 36GB of blocks. Assuming 8192-byte blocks,
198 you need room for almost five million index entries.
199 Since index entries are 50 bytes each, you need 250MB
201 If the background index update process can make a single
202 pass through the index in an hour, which is possible,
203 then you can sustain the 10MB/s indefinitely (at least until
204 the arenas are all filled).
208 requires memory equal to its size on disk,
211 A reasonable starting allocation is to
212 divide memory equally (in thirds) between
213 the bloom filter, the index cache, and the lump and block caches;
214 the third of memory allocated to the lump and block caches
215 should be split unevenly, with more (say, two thirds)
216 going to the block cache.
218 The venti server announces two network services, one
219 (conventionally TCP port
222 the venti protocol as described in
225 (conventionally TCP port
229 The venti web server provides the following
230 URLs for accessing status information:
235 A summary of the usage of the arenas and index sections.
242 Brief storage totals.
245 The current integer value of
249 whether or not to compress blocks
252 whether to write entries to the debugging logs;
254 whether to collect run-time statistics;
255 .BR icachesleeptime ,
256 the time in milliseconds between successive updates
257 of megabytes of the index cache;
258 .BR arenasumsleeptime ,
259 the time in milliseconds between reads while
260 checksumming an arena in the background.
261 The two sleep times should be (but are not) managed by venti;
262 they exist to provide more experience with their effects.
263 The other variables exist only for debugging and
264 performance measurement.
266 .BI /set/ variable / value
272 .BI /graph/ name / param / param / \fR...
273 A PNG image graphing the named run-time statistic over time.
274 The details of names and parameters are undocumented;
277 in the venti sources.
280 A list of all debugging logs present in the server's memory.
283 The contents of the debugging log with the given
287 Force venti to begin flushing the index cache to disk.
288 The request response will not be sent until the flush
292 Force venti to begin flushing the arena block cache to disk.
293 The request response will not be sent until the flush
297 Requests for other files are served by consulting a
298 directory named in the configuration file
302 .SS Configuration File
303 A venti configuration file
304 enumerates the various index sections and
305 arenas that constitute a venti system.
306 The components are indicated by the name of the file, typically
307 a disk partition, in which they reside. The configuration
308 file is the only location that file names are used. Internally,
309 venti uses the names assigned when the components were formatted
316 In particular, only the configuration needs to be
317 changed if a component is moved to a different file.
319 The configuration file consists of lines in the form described below.
323 .TF "\fLindex\fI name "
327 Names the index for the system.
331 is an arena partition, formatted using
336 is an index section, formatted using
341 is a bloom filter, formatted using
345 After formatting a venti system using
347 the order of arenas and index sections should not be changed.
348 Additional arenas can be appended to the configuration;
353 flag to update the index.
355 The configuration file also holds configuration parameters
356 for the venti server itself.
358 .TF "\fLhttpaddr\fI netaddr "
370 network address to announce venti service
374 .BI httpaddr " netaddr
375 network address to announce HTTP service
380 queue writes in memory
381 (default is not to queue)
384 directory tree containing files for
386 internal HTTP server to consult for unrecognized URLs
389 The units for the various cache sizes above can be specified by appending a
395 to indicate kilobytes, megabytes, or gigabytes respectively.
399 name in the configuration lines above can be of the form
401 to specify a range of the file.
405 are specified in bytes but can have the usual
416 This notation eliminates the need to
417 partition raw disks on non-Plan 9 systems.
419 Many of the options to Venti duplicate parameters that
420 can be specified in the configuration file.
421 The command line options override those found in a
423 Additional options are:
424 .TF "\fL-c\fI config"
428 The server configuration file
433 Produce various debugging information on standard error.
438 Enable logging. By default all logging is disabled.
439 Logging slows server operation considerably.
442 Allow only read access to the venti data.
445 Do not run in the background.
447 the foreground process will exit once the Venti server
448 is initialized and ready for connections.
451 A simple configuration:
456 isect /tmp/disks/isect0
457 isect /tmp/disks/isect1
458 arenas /tmp/disks/arenas
459 bloom /tmp/disks/bloom
466 Format the index sections, the arena partition,
467 the bloom filter, and
468 finally the main index:
471 % venti/fmtisect isect0. /tmp/disks/isect0
472 % venti/fmtisect isect1. /tmp/disks/isect1
473 % venti/fmtarenas arenas0. /tmp/disks/arenas &
474 % venti/fmtbloom /tmp/disks/bloom &
476 % venti/fmtindex venti.conf
480 Start the server and check the storage statistics:
484 % hget http://$sysname/storage
487 .B \*9/src/cmd/venti/srv
495 Sean Quinlan and Sean Dorward,
496 ``Venti: a new approach to archival storage'',
497 .I "Usenix Conference on File and Storage Technologies" ,
500 Setting up a venti server is too complicated.
502 Venti should not require the user to decide how to
503 partition its memory usage.
505 Users of shells other than
507 will not be able to use the program names shown.
508 One solution is to define
509 .B "V=$PLAN9/bin/venti"
