1 be7cbb4e 2005-07-12 devnull .TH VENTI.CONF 7
3 be7cbb4e 2005-07-12 devnull venti.conf \- venti configuration
4 be7cbb4e 2005-07-12 devnull .SH DESCRIPTION
5 be7cbb4e 2005-07-12 devnull Venti is a SHA1-addressed archival storage server.
7 be7cbb4e 2005-07-12 devnull .IR venti (7)
8 be7cbb4e 2005-07-12 devnull for a full introduction to the system.
9 be7cbb4e 2005-07-12 devnull This page documents the structure and operation of the server.
11 be7cbb4e 2005-07-12 devnull A venti server requires multiple disks or disk partitions,
12 be7cbb4e 2005-07-12 devnull each of which must be properly formatted before the server
13 be7cbb4e 2005-07-12 devnull can be run.
15 be7cbb4e 2005-07-12 devnull The venti server maintains three disk structures, typically
16 be7cbb4e 2005-07-12 devnull stored on raw disk partitions:
17 be7cbb4e 2005-07-12 devnull the append-only
18 be7cbb4e 2005-07-12 devnull .IR "data log" ,
19 be7cbb4e 2005-07-12 devnull which holds, in sequential order,
20 be7cbb4e 2005-07-12 devnull the contents of every block written to the server;
22 be7cbb4e 2005-07-12 devnull .IR index ,
23 be7cbb4e 2005-07-12 devnull which helps locate a block in the data log given its score;
24 be7cbb4e 2005-07-12 devnull and optionally the
25 be7cbb4e 2005-07-12 devnull .IR "bloom filter" ,
26 be7cbb4e 2005-07-12 devnull a concise summary of which scores are present in the index.
27 be7cbb4e 2005-07-12 devnull The data log is the primary storage.
28 be7cbb4e 2005-07-12 devnull To improve the robustness, it should be stored on
29 be7cbb4e 2005-07-12 devnull a device that provides RAID functionality.
30 be7cbb4e 2005-07-12 devnull The index and the bloom filter are optimizations
31 be7cbb4e 2005-07-12 devnull employed to access the data log efficiently and can be rebuilt
32 be7cbb4e 2005-07-12 devnull if lost or damaged.
34 be7cbb4e 2005-07-12 devnull The data log is logically split into sections called
35 be7cbb4e 2005-07-12 devnull .IR arenas ,
36 be7cbb4e 2005-07-12 devnull typically sized for easy offline backup
37 be7cbb4e 2005-07-12 devnull (e.g., 500MB).
38 be7cbb4e 2005-07-12 devnull A data log may comprise many disks, each storing
39 be7cbb4e 2005-07-12 devnull one or more arenas.
40 be7cbb4e 2005-07-12 devnull Such disks are called
41 be7cbb4e 2005-07-12 devnull .IR "arena partitions" .
42 be7cbb4e 2005-07-12 devnull Arena partitions are filled in the order given in the configuration.
44 be7cbb4e 2005-07-12 devnull The index is logically split into block-sized pieces called
45 be7cbb4e 2005-07-12 devnull .IR buckets ,
46 be7cbb4e 2005-07-12 devnull each of which is responsible for a particular range of scores.
47 be7cbb4e 2005-07-12 devnull An index may be split across many disks, each storing many buckets.
48 be7cbb4e 2005-07-12 devnull Such disks are called
49 be7cbb4e 2005-07-12 devnull .IR "index sections" .
51 be7cbb4e 2005-07-12 devnull The index must be sized so that no bucket is full.
52 be7cbb4e 2005-07-12 devnull When a bucket fills, the server must be shut down and
53 be7cbb4e 2005-07-12 devnull the index made larger.
54 be7cbb4e 2005-07-12 devnull Since scores appear random, each bucket will contain
55 be7cbb4e 2005-07-12 devnull approximately the same number of entries.
56 be7cbb4e 2005-07-12 devnull Index entries are 40 bytes long. Assuming that a typical block
57 be7cbb4e 2005-07-12 devnull being written to the server is 8192 bytes and compresses to 4096
58 be7cbb4e 2005-07-12 devnull bytes, the active index is expected to be about 1% of
59 be7cbb4e 2005-07-12 devnull the active data log.
60 be7cbb4e 2005-07-12 devnull Storing smaller blocks increases the relative index footprint;
61 be7cbb4e 2005-07-12 devnull storing larger blocks decreases it.
62 be7cbb4e 2005-07-12 devnull To allow variation in both block size and the random distribution
63 be7cbb4e 2005-07-12 devnull of scores to buckets, the suggested index size is 5% of
64 be7cbb4e 2005-07-12 devnull the active data log.
66 be7cbb4e 2005-07-12 devnull The (optional) bloom filter is a large bitmap that is stored on disk but
67 be7cbb4e 2005-07-12 devnull also kept completely in memory while the venti server runs.
68 be7cbb4e 2005-07-12 devnull It helps the venti server efficiently detect scores that are
70 be7cbb4e 2005-07-12 devnull already stored in the index.
71 be7cbb4e 2005-07-12 devnull The bloom filter starts out zeroed.
72 be7cbb4e 2005-07-12 devnull Each score recorded in the bloom filter is hashed to choose
74 be7cbb4e 2005-07-12 devnull bits to set in the bloom filter.
75 be7cbb4e 2005-07-12 devnull A score is definitely not stored in the index of any of its
77 be7cbb4e 2005-07-12 devnull bits are not set.
78 be7cbb4e 2005-07-12 devnull The bloom filter thus has two parameters:
80 be7cbb4e 2005-07-12 devnull (maximum 32)
81 be7cbb4e 2005-07-12 devnull and the total bitmap size
82 be7cbb4e 2005-07-12 devnull (maximum 512MB, 2\s-2\u32\d\s+2 bits).
84 be7cbb4e 2005-07-12 devnull The bloom filter should be sized so that
87 be7cbb4e 2005-07-12 devnull .I nblock
94 be7cbb4e 2005-07-12 devnull .I nblock
95 be7cbb4e 2005-07-12 devnull is the expected number of blocks stored on the server
98 be7cbb4e 2005-07-12 devnull is the bitmap size in bits.
99 be7cbb4e 2005-07-12 devnull The false positive rate of the bloom filter when sized
100 be7cbb4e 2005-07-12 devnull this way is approximately 2\s-2\u\-\fInblock\fR\d\s+2.
101 be7cbb4e 2005-07-12 devnull .I Nhash
102 be7cbb4e 2005-07-12 devnull less than 10 are not very useful;
103 be7cbb4e 2005-07-12 devnull .I nhash
104 be7cbb4e 2005-07-12 devnull greater than 24 are probably a waste of memory.
105 be7cbb4e 2005-07-12 devnull .I Fmtbloom
107 be7cbb4e 2005-07-12 devnull .IR venti-fmt (8))
108 be7cbb4e 2005-07-12 devnull can be given either
109 be7cbb4e 2005-07-12 devnull .I nhash
111 be7cbb4e 2005-07-12 devnull .IR nblock ;
112 be7cbb4e 2005-07-12 devnull if given
113 be7cbb4e 2005-07-12 devnull .IR nblock ,
114 be7cbb4e 2005-07-12 devnull it will derive an appropriate
115 be7cbb4e 2005-07-12 devnull .IR nhash .
116 be7cbb4e 2005-07-12 devnull .SS Memory
117 be7cbb4e 2005-07-12 devnull Venti can make effective use of large amounts of memory
118 be7cbb4e 2005-07-12 devnull for various caches.
121 be7cbb4e 2005-07-12 devnull .I "lump cache
122 be7cbb4e 2005-07-12 devnull holds recently-accessed venti data blocks, which the server refers to as
123 be7cbb4e 2005-07-12 devnull .IR lumps .
124 be7cbb4e 2005-07-12 devnull The lump cache should be at least 1MB but can profitably be much larger.
125 be7cbb4e 2005-07-12 devnull The lump cache can be thought of as the level-1 cache:
126 be7cbb4e 2005-07-12 devnull read requests handled by the lump cache can
127 be7cbb4e 2005-07-12 devnull be served instantly.
130 be7cbb4e 2005-07-12 devnull .I "block cache
131 be7cbb4e 2005-07-12 devnull holds recently-accessed
133 be7cbb4e 2005-07-12 devnull blocks from the arena partitions.
134 be7cbb4e 2005-07-12 devnull The block cache needs to be able to simultaneously hold two blocks
135 be7cbb4e 2005-07-12 devnull from each arena plus four blocks for the currently-filling arena.
136 be7cbb4e 2005-07-12 devnull The block cache can be thought of as the level-2 cache:
137 be7cbb4e 2005-07-12 devnull read requests handled by the block cache are slower than those
138 be7cbb4e 2005-07-12 devnull handled by the lump cache, since the lump data must be extracted
139 be7cbb4e 2005-07-12 devnull from the raw disk blocks and possibly decompressed, but no
140 be7cbb4e 2005-07-12 devnull disk accesses are necessary.
143 be7cbb4e 2005-07-12 devnull .I "index cache
144 be7cbb4e 2005-07-12 devnull holds recently-accessed or prefetched
145 be7cbb4e 2005-07-12 devnull index entries.
146 be7cbb4e 2005-07-12 devnull The index cache needs to be able to hold index entries
147 be7cbb4e 2005-07-12 devnull for three or four arenas, at least, in order for prefetching
148 be7cbb4e 2005-07-12 devnull to work properly. Each index entry is 50 bytes.
149 be7cbb4e 2005-07-12 devnull Assuming 500MB arenas of
150 be7cbb4e 2005-07-12 devnull 128,000 blocks that are 4096 bytes each after compression,
151 be7cbb4e 2005-07-12 devnull the minimum index cache size is about 6MB.
152 be7cbb4e 2005-07-12 devnull The index cache can be thought of as the level-3 cache:
153 be7cbb4e 2005-07-12 devnull read requests handled by the index cache must still go
154 be7cbb4e 2005-07-12 devnull to disk to fetch the arena blocks, but the costly random
155 be7cbb4e 2005-07-12 devnull access to the index is avoided.
157 be7cbb4e 2005-07-12 devnull The size of the index cache determines how long venti
158 be7cbb4e 2005-07-12 devnull can sustain its `burst' write throughput, during which time
159 be7cbb4e 2005-07-12 devnull the only disk accesses on the critical path
160 be7cbb4e 2005-07-12 devnull are sequential writes to the arena partitions.
161 be7cbb4e 2005-07-12 devnull For example, if you want to be able to sustain 10MB/s
162 be7cbb4e 2005-07-12 devnull for an hour, you need enough index cache to hold entries
163 be7cbb4e 2005-07-12 devnull for 36GB of blocks. Assuming 8192-byte blocks,
164 be7cbb4e 2005-07-12 devnull you need room for almost five million index entries.
165 be7cbb4e 2005-07-12 devnull Since index entries are 50 bytes each, you need 250MB
166 be7cbb4e 2005-07-12 devnull of index cache.
167 be7cbb4e 2005-07-12 devnull If the background index update process can make a single
168 be7cbb4e 2005-07-12 devnull pass through the index in an hour, which is possible,
169 be7cbb4e 2005-07-12 devnull then you can sustain the 10MB/s indefinitely (at least until
170 be7cbb4e 2005-07-12 devnull the arenas are all filled).
173 be7cbb4e 2005-07-12 devnull .I "bloom filter
174 be7cbb4e 2005-07-12 devnull requires memory equal to its size on disk,
175 be7cbb4e 2005-07-12 devnull as discussed above.
177 be7cbb4e 2005-07-12 devnull A reasonable starting allocation is to
178 be7cbb4e 2005-07-12 devnull divide memory equally (in thirds) between
179 be7cbb4e 2005-07-12 devnull the bloom filter, the index cache, and the lump and block caches;
180 be7cbb4e 2005-07-12 devnull the third of memory allocated to the lump and block caches
181 be7cbb4e 2005-07-12 devnull should be split unevenly, with more (say, two thirds)
182 be7cbb4e 2005-07-12 devnull going to the block cache.
183 be7cbb4e 2005-07-12 devnull .SS Network
184 be7cbb4e 2005-07-12 devnull The venti server announces two network services, one
185 be7cbb4e 2005-07-12 devnull (conventionally TCP port
186 be7cbb4e 2005-07-12 devnull .BR venti ,
187 be7cbb4e 2005-07-12 devnull 17034) serving
188 be7cbb4e 2005-07-12 devnull the venti protocol as described in
189 be7cbb4e 2005-07-12 devnull .IR venti (7),
190 be7cbb4e 2005-07-12 devnull and one serving HTTP
191 be7cbb4e 2005-07-12 devnull (conventionally TCP port
192 be7cbb4e 2005-07-12 devnull .BR venti ,
195 be7cbb4e 2005-07-12 devnull The venti web server provides the following
196 be7cbb4e 2005-07-12 devnull URLs for accessing status information:
198 be7cbb4e 2005-07-12 devnull .B /index
199 be7cbb4e 2005-07-12 devnull A summary of the usage of the arenas and index sections.
201 be7cbb4e 2005-07-12 devnull .B /xindex
202 be7cbb4e 2005-07-12 devnull An XML version of
203 be7cbb4e 2005-07-12 devnull .BR /index .
205 be7cbb4e 2005-07-12 devnull .B /storage
206 be7cbb4e 2005-07-12 devnull Brief storage totals.
208 be7cbb4e 2005-07-12 devnull .BI /set/ variable
209 be7cbb4e 2005-07-12 devnull The current integer value of
210 be7cbb4e 2005-07-12 devnull .IR variable .
211 be7cbb4e 2005-07-12 devnull Variables are:
212 be7cbb4e 2005-07-12 devnull .BR compress ,
213 be7cbb4e 2005-07-12 devnull whether or not to compress blocks
214 be7cbb4e 2005-07-12 devnull (for debugging);
215 be7cbb4e 2005-07-12 devnull .BR logging ,
216 be7cbb4e 2005-07-12 devnull whether to write entries to the debugging logs;
217 be7cbb4e 2005-07-12 devnull .BR stats ,
218 be7cbb4e 2005-07-12 devnull whether to collect run-time statistics;
219 be7cbb4e 2005-07-12 devnull .BR icachesleeptime ,
220 be7cbb4e 2005-07-12 devnull the time in milliseconds between successive updates
221 be7cbb4e 2005-07-12 devnull of megabytes of the index cache;
222 be7cbb4e 2005-07-12 devnull .BR arenasumsleeptime ,
223 be7cbb4e 2005-07-12 devnull the time in milliseconds between reads while
224 be7cbb4e 2005-07-12 devnull checksumming an arena in the background.
225 be7cbb4e 2005-07-12 devnull The two sleep times should be (but are not) managed by venti;
226 be7cbb4e 2005-07-12 devnull they exist to provide more experience with their effects.
227 be7cbb4e 2005-07-12 devnull The other variables exist only for debugging and
228 be7cbb4e 2005-07-12 devnull performance measurement.
230 be7cbb4e 2005-07-12 devnull .BI /set/ variable / value
232 be7cbb4e 2005-07-12 devnull .I variable
234 be7cbb4e 2005-07-12 devnull .IR value .
236 be7cbb4e 2005-07-12 devnull .BI /graph/ name / param / param / \fR...
237 be7cbb4e 2005-07-12 devnull A PNG image graphing the named run-time statistic over time.
238 be7cbb4e 2005-07-12 devnull The details of names and parameters are undocumented;
240 be7cbb4e 2005-07-12 devnull .B httpd.c
241 be7cbb4e 2005-07-12 devnull in the venti sources.
244 be7cbb4e 2005-07-12 devnull A list of all debugging logs present in the server's memory.
246 be7cbb4e 2005-07-12 devnull .BI /log/ name
247 be7cbb4e 2005-07-12 devnull The contents of the debugging log with the given
248 be7cbb4e 2005-07-12 devnull .IR name .
250 be7cbb4e 2005-07-12 devnull .B /flushicache
251 be7cbb4e 2005-07-12 devnull Force venti to begin flushing the index cache to disk.
252 be7cbb4e 2005-07-12 devnull The request response will not be sent until the flush
253 be7cbb4e 2005-07-12 devnull has completed.
255 be7cbb4e 2005-07-12 devnull .B /flushdcache
256 be7cbb4e 2005-07-12 devnull Force venti to begin flushing the arena block cache to disk.
257 be7cbb4e 2005-07-12 devnull The request response will not be sent until the flush
258 be7cbb4e 2005-07-12 devnull has completed.
261 be7cbb4e 2005-07-12 devnull Requests for other files are served by consulting a
262 be7cbb4e 2005-07-12 devnull directory named in the configuration file
264 be7cbb4e 2005-07-12 devnull .B webroot
266 be7cbb4e 2005-07-12 devnull .SS Configuration File
267 be7cbb4e 2005-07-12 devnull A venti configuration file
268 be7cbb4e 2005-07-12 devnull enumerates the various index sections and
269 be7cbb4e 2005-07-12 devnull arenas that constitute a venti system.
270 be7cbb4e 2005-07-12 devnull The components are indicated by the name of the file, typically
271 be7cbb4e 2005-07-12 devnull a disk partition, in which they reside. The configuration
272 be7cbb4e 2005-07-12 devnull file is the only location that file names are used. Internally,
273 be7cbb4e 2005-07-12 devnull venti uses the names assigned when the components were formatted
275 be7cbb4e 2005-07-12 devnull .I fmtarenas
277 be7cbb4e 2005-07-12 devnull .I fmtisect
279 be7cbb4e 2005-07-12 devnull .IR venti-fmt (8)).
280 be7cbb4e 2005-07-12 devnull In particular, only the configuration needs to be
281 be7cbb4e 2005-07-12 devnull changed if a component is moved to a different file.
283 be7cbb4e 2005-07-12 devnull The configuration file consists of lines in the form described below.
284 be7cbb4e 2005-07-12 devnull Lines starting with
286 be7cbb4e 2005-07-12 devnull are comments.
288 be7cbb4e 2005-07-12 devnull .BI index " name
289 be7cbb4e 2005-07-12 devnull Names the index for the system.
291 be7cbb4e 2005-07-12 devnull .BI arenas " file
293 be7cbb4e 2005-07-12 devnull is an arena partition, formatted using
294 be7cbb4e 2005-07-12 devnull .IR fmtarenas .
296 be7cbb4e 2005-07-12 devnull .BI isect " file
298 be7cbb4e 2005-07-12 devnull is an index section, formatted using
299 be7cbb4e 2005-07-12 devnull .IR fmtisect .
301 be7cbb4e 2005-07-12 devnull After formatting a venti system using
302 be7cbb4e 2005-07-12 devnull .IR fmtindex ,
303 be7cbb4e 2005-07-12 devnull the order of arenas and index sections should not be changed.
304 be7cbb4e 2005-07-12 devnull Additional arenas can be appended to the configuration;
306 be7cbb4e 2005-07-12 devnull .I fmtindex
307 be7cbb4e 2005-07-12 devnull with the
309 be7cbb4e 2005-07-12 devnull flag to update the index.
311 be7cbb4e 2005-07-12 devnull The configuration file also holds configuration parameters
312 be7cbb4e 2005-07-12 devnull for the venti server itself.
313 be7cbb4e 2005-07-12 devnull These are:
314 be7cbb4e 2005-07-12 devnull .TF httpaddr netaddr
316 be7cbb4e 2005-07-12 devnull .BI mem " size
317 be7cbb4e 2005-07-12 devnull lump cache size
319 be7cbb4e 2005-07-12 devnull .BI bcmem " size
320 be7cbb4e 2005-07-12 devnull block cache size
322 be7cbb4e 2005-07-12 devnull .BI icmem " size
323 be7cbb4e 2005-07-12 devnull index cache size
325 be7cbb4e 2005-07-12 devnull .BI addr " netaddr
326 be7cbb4e 2005-07-12 devnull network address to announce venti service
327 be7cbb4e 2005-07-12 devnull (default
328 be7cbb4e 2005-07-12 devnull .BR tcp!*!venti )
330 be7cbb4e 2005-07-12 devnull .BI httpaddr " netaddr
331 be7cbb4e 2005-07-12 devnull network address to announce HTTP service
332 be7cbb4e 2005-07-12 devnull (default
333 be7cbb4e 2005-07-12 devnull .BR tcp!*!http )
335 be7cbb4e 2005-07-12 devnull .B queuewrites
336 be7cbb4e 2005-07-12 devnull queue writes in memory
337 be7cbb4e 2005-07-12 devnull (default is not to queue)
339 be7cbb4e 2005-07-12 devnull See the server description in
340 be7cbb4e 2005-07-12 devnull .IR venti (8)
341 be7cbb4e 2005-07-12 devnull for explanations of these variables.
342 be7cbb4e 2005-07-12 devnull .SH EXAMPLE
345 be7cbb4e 2005-07-12 devnull index main
346 be7cbb4e 2005-07-12 devnull isect /tmp/disks/isect0
347 be7cbb4e 2005-07-12 devnull isect /tmp/disks/isect1
348 be7cbb4e 2005-07-12 devnull arenas /tmp/disks/arenas
350 be7cbb4e 2005-07-12 devnull bcmem 20M
351 be7cbb4e 2005-07-12 devnull icmem 30M
353 be7cbb4e 2005-07-12 devnull .SH "SEE ALSO"
354 be7cbb4e 2005-07-12 devnull .IR venti (8),
355 be7cbb4e 2005-07-12 devnull .IR venti-fmt (8)
356 be7cbb4e 2005-07-12 devnull .SH BUGS
357 be7cbb4e 2005-07-12 devnull Setting up a venti server is too complicated.
359 be7cbb4e 2005-07-12 devnull Venti should not require the user to decide how to
360 be7cbb4e 2005-07-12 devnull partition its memory usage.
