Blame


1 bdf5b5cd 2005-07-13 devnull .TH VENTI 8
2 bdf5b5cd 2005-07-13 devnull .SH NAME
3 3aec33fe 2005-07-18 devnull venti \- archival storage server
4 3aec33fe 2005-07-18 devnull .SH SYNOPSIS
5 3aec33fe 2005-07-18 devnull .B venti/venti
6 3aec33fe 2005-07-18 devnull [
7 3aec33fe 2005-07-18 devnull .B -Ldsw
8 3aec33fe 2005-07-18 devnull ]
9 3aec33fe 2005-07-18 devnull [
10 3aec33fe 2005-07-18 devnull .B -a
11 3aec33fe 2005-07-18 devnull .I address
12 3aec33fe 2005-07-18 devnull ]
13 3aec33fe 2005-07-18 devnull [
14 3aec33fe 2005-07-18 devnull .B -B
15 3aec33fe 2005-07-18 devnull .I blockcachesize
16 3aec33fe 2005-07-18 devnull ]
17 3aec33fe 2005-07-18 devnull [
18 3aec33fe 2005-07-18 devnull .B -c
19 3aec33fe 2005-07-18 devnull .I config
20 3aec33fe 2005-07-18 devnull ]
21 3aec33fe 2005-07-18 devnull .PP
22 3aec33fe 2005-07-18 devnull .B " "
23 3aec33fe 2005-07-18 devnull [
24 3aec33fe 2005-07-18 devnull .B -C
25 3aec33fe 2005-07-18 devnull .I lumpcachesize
26 3aec33fe 2005-07-18 devnull ]
27 3aec33fe 2005-07-18 devnull [
28 3aec33fe 2005-07-18 devnull .B -h
29 3aec33fe 2005-07-18 devnull .I httpaddress
30 3aec33fe 2005-07-18 devnull ]
31 3aec33fe 2005-07-18 devnull [
32 3aec33fe 2005-07-18 devnull .B -I
33 3aec33fe 2005-07-18 devnull .I indexcachesize
34 3aec33fe 2005-07-18 devnull ]
35 3aec33fe 2005-07-18 devnull [
36 3aec33fe 2005-07-18 devnull .B -W
37 3aec33fe 2005-07-18 devnull .I webroot
38 3aec33fe 2005-07-18 devnull ]
39 bdf5b5cd 2005-07-13 devnull .SH DESCRIPTION
40 bdf5b5cd 2005-07-13 devnull Venti is a SHA1-addressed archival storage server.
41 bdf5b5cd 2005-07-13 devnull See
42 bdf5b5cd 2005-07-13 devnull .IR venti (7)
43 bdf5b5cd 2005-07-13 devnull for a full introduction to the system.
44 bdf5b5cd 2005-07-13 devnull This page documents the structure and operation of the server.
45 bdf5b5cd 2005-07-13 devnull .PP
46 bdf5b5cd 2005-07-13 devnull A venti server requires multiple disks or disk partitions,
47 bdf5b5cd 2005-07-13 devnull each of which must be properly formatted before the server
48 bdf5b5cd 2005-07-13 devnull can be run.
49 bdf5b5cd 2005-07-13 devnull .SS Disk
50 bdf5b5cd 2005-07-13 devnull The venti server maintains three disk structures, typically
51 bdf5b5cd 2005-07-13 devnull stored on raw disk partitions:
52 bdf5b5cd 2005-07-13 devnull the append-only
53 bdf5b5cd 2005-07-13 devnull .IR "data log" ,
54 bdf5b5cd 2005-07-13 devnull which holds, in sequential order,
55 bdf5b5cd 2005-07-13 devnull the contents of every block written to the server;
56 bdf5b5cd 2005-07-13 devnull the
57 bdf5b5cd 2005-07-13 devnull .IR index ,
58 bdf5b5cd 2005-07-13 devnull which helps locate a block in the data log given its score;
59 bdf5b5cd 2005-07-13 devnull and optionally the
60 bdf5b5cd 2005-07-13 devnull .IR "bloom filter" ,
61 bdf5b5cd 2005-07-13 devnull a concise summary of which scores are present in the index.
62 bdf5b5cd 2005-07-13 devnull The data log is the primary storage.
63 bdf5b5cd 2005-07-13 devnull To improve the robustness, it should be stored on
64 bdf5b5cd 2005-07-13 devnull a device that provides RAID functionality.
65 bdf5b5cd 2005-07-13 devnull The index and the bloom filter are optimizations
66 bdf5b5cd 2005-07-13 devnull employed to access the data log efficiently and can be rebuilt
67 bdf5b5cd 2005-07-13 devnull if lost or damaged.
68 bdf5b5cd 2005-07-13 devnull .PP
69 bdf5b5cd 2005-07-13 devnull The data log is logically split into sections called
70 bdf5b5cd 2005-07-13 devnull .IR arenas ,
71 bdf5b5cd 2005-07-13 devnull typically sized for easy offline backup
72 bdf5b5cd 2005-07-13 devnull (e.g., 500MB).
73 bdf5b5cd 2005-07-13 devnull A data log may comprise many disks, each storing
74 bdf5b5cd 2005-07-13 devnull one or more arenas.
75 bdf5b5cd 2005-07-13 devnull Such disks are called
76 bdf5b5cd 2005-07-13 devnull .IR "arena partitions" .
77 bdf5b5cd 2005-07-13 devnull Arena partitions are filled in the order given in the configuration.
78 bdf5b5cd 2005-07-13 devnull .PP
79 bdf5b5cd 2005-07-13 devnull The index is logically split into block-sized pieces called
80 bdf5b5cd 2005-07-13 devnull .IR buckets ,
81 bdf5b5cd 2005-07-13 devnull each of which is responsible for a particular range of scores.
82 bdf5b5cd 2005-07-13 devnull An index may be split across many disks, each storing many buckets.
83 bdf5b5cd 2005-07-13 devnull Such disks are called
84 bdf5b5cd 2005-07-13 devnull .IR "index sections" .
85 bdf5b5cd 2005-07-13 devnull .PP
86 bdf5b5cd 2005-07-13 devnull The index must be sized so that no bucket is full.
87 bdf5b5cd 2005-07-13 devnull When a bucket fills, the server must be shut down and
88 bdf5b5cd 2005-07-13 devnull the index made larger.
89 bdf5b5cd 2005-07-13 devnull Since scores appear random, each bucket will contain
90 bdf5b5cd 2005-07-13 devnull approximately the same number of entries.
91 bdf5b5cd 2005-07-13 devnull Index entries are 40 bytes long. Assuming that a typical block
92 bdf5b5cd 2005-07-13 devnull being written to the server is 8192 bytes and compresses to 4096
93 bdf5b5cd 2005-07-13 devnull bytes, the active index is expected to be about 1% of
94 bdf5b5cd 2005-07-13 devnull the active data log.
95 bdf5b5cd 2005-07-13 devnull Storing smaller blocks increases the relative index footprint;
96 bdf5b5cd 2005-07-13 devnull storing larger blocks decreases it.
97 bdf5b5cd 2005-07-13 devnull To allow variation in both block size and the random distribution
98 bdf5b5cd 2005-07-13 devnull of scores to buckets, the suggested index size is 5% of
99 bdf5b5cd 2005-07-13 devnull the active data log.
100 bdf5b5cd 2005-07-13 devnull .PP
101 bdf5b5cd 2005-07-13 devnull The (optional) bloom filter is a large bitmap that is stored on disk but
102 bdf5b5cd 2005-07-13 devnull also kept completely in memory while the venti server runs.
103 bdf5b5cd 2005-07-13 devnull It helps the venti server efficiently detect scores that are
104 bdf5b5cd 2005-07-13 devnull .I not
105 bdf5b5cd 2005-07-13 devnull already stored in the index.
106 bdf5b5cd 2005-07-13 devnull The bloom filter starts out zeroed.
107 bdf5b5cd 2005-07-13 devnull Each score recorded in the bloom filter is hashed to choose
108 bdf5b5cd 2005-07-13 devnull .I nhash
109 bdf5b5cd 2005-07-13 devnull bits to set in the bloom filter.
110 bdf5b5cd 2005-07-13 devnull A score is definitely not stored in the index of any of its
111 bdf5b5cd 2005-07-13 devnull .I nhash
112 bdf5b5cd 2005-07-13 devnull bits are not set.
113 bdf5b5cd 2005-07-13 devnull The bloom filter thus has two parameters:
114 bdf5b5cd 2005-07-13 devnull .I nhash
115 bdf5b5cd 2005-07-13 devnull (maximum 32)
116 bdf5b5cd 2005-07-13 devnull and the total bitmap size
117 bdf5b5cd 2005-07-13 devnull (maximum 512MB, 2\s-2\u32\d\s+2 bits).
118 bdf5b5cd 2005-07-13 devnull .PP
119 bdf5b5cd 2005-07-13 devnull The bloom filter should be sized so that
120 bdf5b5cd 2005-07-13 devnull .I nhash
121 3aec33fe 2005-07-18 devnull \(mu
122 bdf5b5cd 2005-07-13 devnull .I nblock
123 bdf5b5cd 2005-07-13 devnull \(<=
124 3aec33fe 2005-07-18 devnull 0.7 \(mu
125 bdf5b5cd 2005-07-13 devnull .IR b ,
126 bdf5b5cd 2005-07-13 devnull where
127 bdf5b5cd 2005-07-13 devnull .I nblock
128 bdf5b5cd 2005-07-13 devnull is the expected number of blocks stored on the server
129 bdf5b5cd 2005-07-13 devnull and
130 bdf5b5cd 2005-07-13 devnull .I b
131 bdf5b5cd 2005-07-13 devnull is the bitmap size in bits.
132 bdf5b5cd 2005-07-13 devnull The false positive rate of the bloom filter when sized
133 bdf5b5cd 2005-07-13 devnull this way is approximately 2\s-2\u\-\fInblock\fR\d\s+2.
134 bdf5b5cd 2005-07-13 devnull .I Nhash
135 bdf5b5cd 2005-07-13 devnull less than 10 are not very useful;
136 bdf5b5cd 2005-07-13 devnull .I nhash
137 bdf5b5cd 2005-07-13 devnull greater than 24 are probably a waste of memory.
138 bdf5b5cd 2005-07-13 devnull .I Fmtbloom
139 bdf5b5cd 2005-07-13 devnull (see
140 bdf5b5cd 2005-07-13 devnull .IR venti-fmt (8))
141 bdf5b5cd 2005-07-13 devnull can be given either
142 bdf5b5cd 2005-07-13 devnull .I nhash
143 bdf5b5cd 2005-07-13 devnull or
144 bdf5b5cd 2005-07-13 devnull .IR nblock ;
145 bdf5b5cd 2005-07-13 devnull if given
146 bdf5b5cd 2005-07-13 devnull .IR nblock ,
147 bdf5b5cd 2005-07-13 devnull it will derive an appropriate
148 bdf5b5cd 2005-07-13 devnull .IR nhash .
149 bdf5b5cd 2005-07-13 devnull .SS Memory
150 bdf5b5cd 2005-07-13 devnull Venti can make effective use of large amounts of memory
151 bdf5b5cd 2005-07-13 devnull for various caches.
152 bdf5b5cd 2005-07-13 devnull .PP
153 bdf5b5cd 2005-07-13 devnull The
154 bdf5b5cd 2005-07-13 devnull .I "lump cache
155 bdf5b5cd 2005-07-13 devnull holds recently-accessed venti data blocks, which the server refers to as
156 bdf5b5cd 2005-07-13 devnull .IR lumps .
157 bdf5b5cd 2005-07-13 devnull The lump cache should be at least 1MB but can profitably be much larger.
158 bdf5b5cd 2005-07-13 devnull The lump cache can be thought of as the level-1 cache:
159 bdf5b5cd 2005-07-13 devnull read requests handled by the lump cache can
160 bdf5b5cd 2005-07-13 devnull be served instantly.
161 bdf5b5cd 2005-07-13 devnull .PP
162 bdf5b5cd 2005-07-13 devnull The
163 bdf5b5cd 2005-07-13 devnull .I "block cache
164 bdf5b5cd 2005-07-13 devnull holds recently-accessed
165 bdf5b5cd 2005-07-13 devnull .I disk
166 bdf5b5cd 2005-07-13 devnull blocks from the arena partitions.
167 bdf5b5cd 2005-07-13 devnull The block cache needs to be able to simultaneously hold two blocks
168 bdf5b5cd 2005-07-13 devnull from each arena plus four blocks for the currently-filling arena.
169 bdf5b5cd 2005-07-13 devnull The block cache can be thought of as the level-2 cache:
170 bdf5b5cd 2005-07-13 devnull read requests handled by the block cache are slower than those
171 bdf5b5cd 2005-07-13 devnull handled by the lump cache, since the lump data must be extracted
172 bdf5b5cd 2005-07-13 devnull from the raw disk blocks and possibly decompressed, but no
173 bdf5b5cd 2005-07-13 devnull disk accesses are necessary.
174 bdf5b5cd 2005-07-13 devnull .PP
175 bdf5b5cd 2005-07-13 devnull The
176 bdf5b5cd 2005-07-13 devnull .I "index cache
177 bdf5b5cd 2005-07-13 devnull holds recently-accessed or prefetched
178 bdf5b5cd 2005-07-13 devnull index entries.
179 bdf5b5cd 2005-07-13 devnull The index cache needs to be able to hold index entries
180 bdf5b5cd 2005-07-13 devnull for three or four arenas, at least, in order for prefetching
181 bdf5b5cd 2005-07-13 devnull to work properly. Each index entry is 50 bytes.
182 bdf5b5cd 2005-07-13 devnull Assuming 500MB arenas of
183 bdf5b5cd 2005-07-13 devnull 128,000 blocks that are 4096 bytes each after compression,
184 bdf5b5cd 2005-07-13 devnull the minimum index cache size is about 6MB.
185 bdf5b5cd 2005-07-13 devnull The index cache can be thought of as the level-3 cache:
186 bdf5b5cd 2005-07-13 devnull read requests handled by the index cache must still go
187 bdf5b5cd 2005-07-13 devnull to disk to fetch the arena blocks, but the costly random
188 bdf5b5cd 2005-07-13 devnull access to the index is avoided.
189 bdf5b5cd 2005-07-13 devnull .PP
190 bdf5b5cd 2005-07-13 devnull The size of the index cache determines how long venti
191 bdf5b5cd 2005-07-13 devnull can sustain its `burst' write throughput, during which time
192 bdf5b5cd 2005-07-13 devnull the only disk accesses on the critical path
193 bdf5b5cd 2005-07-13 devnull are sequential writes to the arena partitions.
194 bdf5b5cd 2005-07-13 devnull For example, if you want to be able to sustain 10MB/s
195 bdf5b5cd 2005-07-13 devnull for an hour, you need enough index cache to hold entries
196 bdf5b5cd 2005-07-13 devnull for 36GB of blocks. Assuming 8192-byte blocks,
197 bdf5b5cd 2005-07-13 devnull you need room for almost five million index entries.
198 bdf5b5cd 2005-07-13 devnull Since index entries are 50 bytes each, you need 250MB
199 bdf5b5cd 2005-07-13 devnull of index cache.
200 bdf5b5cd 2005-07-13 devnull If the background index update process can make a single
201 bdf5b5cd 2005-07-13 devnull pass through the index in an hour, which is possible,
202 bdf5b5cd 2005-07-13 devnull then you can sustain the 10MB/s indefinitely (at least until
203 bdf5b5cd 2005-07-13 devnull the arenas are all filled).
204 bdf5b5cd 2005-07-13 devnull .PP
205 bdf5b5cd 2005-07-13 devnull The
206 bdf5b5cd 2005-07-13 devnull .I "bloom filter
207 bdf5b5cd 2005-07-13 devnull requires memory equal to its size on disk,
208 bdf5b5cd 2005-07-13 devnull as discussed above.
209 bdf5b5cd 2005-07-13 devnull .PP
210 bdf5b5cd 2005-07-13 devnull A reasonable starting allocation is to
211 bdf5b5cd 2005-07-13 devnull divide memory equally (in thirds) between
212 bdf5b5cd 2005-07-13 devnull the bloom filter, the index cache, and the lump and block caches;
213 bdf5b5cd 2005-07-13 devnull the third of memory allocated to the lump and block caches
214 bdf5b5cd 2005-07-13 devnull should be split unevenly, with more (say, two thirds)
215 bdf5b5cd 2005-07-13 devnull going to the block cache.
216 bdf5b5cd 2005-07-13 devnull .SS Network
217 bdf5b5cd 2005-07-13 devnull The venti server announces two network services, one
218 bdf5b5cd 2005-07-13 devnull (conventionally TCP port
219 bdf5b5cd 2005-07-13 devnull .BR venti ,
220 bdf5b5cd 2005-07-13 devnull 17034) serving
221 bdf5b5cd 2005-07-13 devnull the venti protocol as described in
222 bdf5b5cd 2005-07-13 devnull .IR venti (7),
223 bdf5b5cd 2005-07-13 devnull and one serving HTTP
224 bdf5b5cd 2005-07-13 devnull (conventionally TCP port
225 bdf5b5cd 2005-07-13 devnull .BR venti ,
226 bdf5b5cd 2005-07-13 devnull 80).
227 bdf5b5cd 2005-07-13 devnull .PP
228 bdf5b5cd 2005-07-13 devnull The venti web server provides the following
229 bdf5b5cd 2005-07-13 devnull URLs for accessing status information:
230 bdf5b5cd 2005-07-13 devnull .TP
231 bdf5b5cd 2005-07-13 devnull .B /index
232 bdf5b5cd 2005-07-13 devnull A summary of the usage of the arenas and index sections.
233 bdf5b5cd 2005-07-13 devnull .TP
234 bdf5b5cd 2005-07-13 devnull .B /xindex
235 bdf5b5cd 2005-07-13 devnull An XML version of
236 bdf5b5cd 2005-07-13 devnull .BR /index .
237 bdf5b5cd 2005-07-13 devnull .TP
238 bdf5b5cd 2005-07-13 devnull .B /storage
239 bdf5b5cd 2005-07-13 devnull Brief storage totals.
240 bdf5b5cd 2005-07-13 devnull .TP
241 bdf5b5cd 2005-07-13 devnull .BI /set/ variable
242 bdf5b5cd 2005-07-13 devnull The current integer value of
243 bdf5b5cd 2005-07-13 devnull .IR variable .
244 bdf5b5cd 2005-07-13 devnull Variables are:
245 bdf5b5cd 2005-07-13 devnull .BR compress ,
246 bdf5b5cd 2005-07-13 devnull whether or not to compress blocks
247 bdf5b5cd 2005-07-13 devnull (for debugging);
248 bdf5b5cd 2005-07-13 devnull .BR logging ,
249 bdf5b5cd 2005-07-13 devnull whether to write entries to the debugging logs;
250 bdf5b5cd 2005-07-13 devnull .BR stats ,
251 bdf5b5cd 2005-07-13 devnull whether to collect run-time statistics;
252 bdf5b5cd 2005-07-13 devnull .BR icachesleeptime ,
253 bdf5b5cd 2005-07-13 devnull the time in milliseconds between successive updates
254 bdf5b5cd 2005-07-13 devnull of megabytes of the index cache;
255 bdf5b5cd 2005-07-13 devnull .BR arenasumsleeptime ,
256 bdf5b5cd 2005-07-13 devnull the time in milliseconds between reads while
257 bdf5b5cd 2005-07-13 devnull checksumming an arena in the background.
258 bdf5b5cd 2005-07-13 devnull The two sleep times should be (but are not) managed by venti;
259 bdf5b5cd 2005-07-13 devnull they exist to provide more experience with their effects.
260 bdf5b5cd 2005-07-13 devnull The other variables exist only for debugging and
261 bdf5b5cd 2005-07-13 devnull performance measurement.
262 bdf5b5cd 2005-07-13 devnull .TP
263 bdf5b5cd 2005-07-13 devnull .BI /set/ variable / value
264 bdf5b5cd 2005-07-13 devnull Set
265 bdf5b5cd 2005-07-13 devnull .I variable
266 bdf5b5cd 2005-07-13 devnull to
267 bdf5b5cd 2005-07-13 devnull .IR value .
268 bdf5b5cd 2005-07-13 devnull .TP
269 bdf5b5cd 2005-07-13 devnull .BI /graph/ name / param / param / \fR...
270 bdf5b5cd 2005-07-13 devnull A PNG image graphing the named run-time statistic over time.
271 bdf5b5cd 2005-07-13 devnull The details of names and parameters are undocumented;
272 bdf5b5cd 2005-07-13 devnull see
273 bdf5b5cd 2005-07-13 devnull .B httpd.c
274 bdf5b5cd 2005-07-13 devnull in the venti sources.
275 bdf5b5cd 2005-07-13 devnull .TP
276 bdf5b5cd 2005-07-13 devnull .B /log
277 bdf5b5cd 2005-07-13 devnull A list of all debugging logs present in the server's memory.
278 bdf5b5cd 2005-07-13 devnull .TP
279 bdf5b5cd 2005-07-13 devnull .BI /log/ name
280 bdf5b5cd 2005-07-13 devnull The contents of the debugging log with the given
281 bdf5b5cd 2005-07-13 devnull .IR name .
282 bdf5b5cd 2005-07-13 devnull .TP
283 bdf5b5cd 2005-07-13 devnull .B /flushicache
284 bdf5b5cd 2005-07-13 devnull Force venti to begin flushing the index cache to disk.
285 bdf5b5cd 2005-07-13 devnull The request response will not be sent until the flush
286 bdf5b5cd 2005-07-13 devnull has completed.
287 bdf5b5cd 2005-07-13 devnull .TP
288 bdf5b5cd 2005-07-13 devnull .B /flushdcache
289 bdf5b5cd 2005-07-13 devnull Force venti to begin flushing the arena block cache to disk.
290 bdf5b5cd 2005-07-13 devnull The request response will not be sent until the flush
291 bdf5b5cd 2005-07-13 devnull has completed.
292 bdf5b5cd 2005-07-13 devnull .PD
293 bdf5b5cd 2005-07-13 devnull .PP
294 bdf5b5cd 2005-07-13 devnull Requests for other files are served by consulting a
295 bdf5b5cd 2005-07-13 devnull directory named in the configuration file
296 bdf5b5cd 2005-07-13 devnull (see
297 bdf5b5cd 2005-07-13 devnull .B webroot
298 bdf5b5cd 2005-07-13 devnull below).
299 bdf5b5cd 2005-07-13 devnull .SS Configuration File
300 bdf5b5cd 2005-07-13 devnull A venti configuration file
301 bdf5b5cd 2005-07-13 devnull enumerates the various index sections and
302 bdf5b5cd 2005-07-13 devnull arenas that constitute a venti system.
303 bdf5b5cd 2005-07-13 devnull The components are indicated by the name of the file, typically
304 bdf5b5cd 2005-07-13 devnull a disk partition, in which they reside. The configuration
305 bdf5b5cd 2005-07-13 devnull file is the only location that file names are used. Internally,
306 bdf5b5cd 2005-07-13 devnull venti uses the names assigned when the components were formatted
307 bdf5b5cd 2005-07-13 devnull with
308 bdf5b5cd 2005-07-13 devnull .I fmtarenas
309 bdf5b5cd 2005-07-13 devnull or
310 bdf5b5cd 2005-07-13 devnull .I fmtisect
311 bdf5b5cd 2005-07-13 devnull (see
312 bdf5b5cd 2005-07-13 devnull .IR venti-fmt (8)).
313 bdf5b5cd 2005-07-13 devnull In particular, only the configuration needs to be
314 bdf5b5cd 2005-07-13 devnull changed if a component is moved to a different file.
315 bdf5b5cd 2005-07-13 devnull .PP
316 bdf5b5cd 2005-07-13 devnull The configuration file consists of lines in the form described below.
317 bdf5b5cd 2005-07-13 devnull Lines starting with
318 bdf5b5cd 2005-07-13 devnull .B #
319 bdf5b5cd 2005-07-13 devnull are comments.
320 bdf5b5cd 2005-07-13 devnull .TP
321 bdf5b5cd 2005-07-13 devnull .BI index " name
322 bdf5b5cd 2005-07-13 devnull Names the index for the system.
323 bdf5b5cd 2005-07-13 devnull .TP
324 bdf5b5cd 2005-07-13 devnull .BI arenas " file
325 bdf5b5cd 2005-07-13 devnull .I File
326 bdf5b5cd 2005-07-13 devnull is an arena partition, formatted using
327 bdf5b5cd 2005-07-13 devnull .IR fmtarenas .
328 bdf5b5cd 2005-07-13 devnull .TP
329 bdf5b5cd 2005-07-13 devnull .BI isect " file
330 bdf5b5cd 2005-07-13 devnull .I File
331 bdf5b5cd 2005-07-13 devnull is an index section, formatted using
332 bdf5b5cd 2005-07-13 devnull .IR fmtisect .
333 eb5fbb97 2005-08-11 devnull .TP
334 eb5fbb97 2005-08-11 devnull .BI bloom " file
335 eb5fbb97 2005-08-11 devnull .I File
336 eb5fbb97 2005-08-11 devnull is a bloom filter, formatted using
337 eb5fbb97 2005-08-11 devnull .IR fmtbloom .
338 06466bd0 2005-07-24 devnull .PD
339 bdf5b5cd 2005-07-13 devnull .PP
340 bdf5b5cd 2005-07-13 devnull After formatting a venti system using
341 bdf5b5cd 2005-07-13 devnull .IR fmtindex ,
342 bdf5b5cd 2005-07-13 devnull the order of arenas and index sections should not be changed.
343 bdf5b5cd 2005-07-13 devnull Additional arenas can be appended to the configuration;
344 bdf5b5cd 2005-07-13 devnull run
345 bdf5b5cd 2005-07-13 devnull .I fmtindex
346 bdf5b5cd 2005-07-13 devnull with the
347 bdf5b5cd 2005-07-13 devnull .B -a
348 bdf5b5cd 2005-07-13 devnull flag to update the index.
349 bdf5b5cd 2005-07-13 devnull .PP
350 bdf5b5cd 2005-07-13 devnull The configuration file also holds configuration parameters
351 bdf5b5cd 2005-07-13 devnull for the venti server itself.
352 bdf5b5cd 2005-07-13 devnull These are:
353 bdf5b5cd 2005-07-13 devnull .TF httpaddr netaddr
354 bdf5b5cd 2005-07-13 devnull .TP
355 bdf5b5cd 2005-07-13 devnull .BI mem " size
356 bdf5b5cd 2005-07-13 devnull lump cache size
357 bdf5b5cd 2005-07-13 devnull .TP
358 bdf5b5cd 2005-07-13 devnull .BI bcmem " size
359 bdf5b5cd 2005-07-13 devnull block cache size
360 bdf5b5cd 2005-07-13 devnull .TP
361 bdf5b5cd 2005-07-13 devnull .BI icmem " size
362 bdf5b5cd 2005-07-13 devnull index cache size
363 bdf5b5cd 2005-07-13 devnull .TP
364 bdf5b5cd 2005-07-13 devnull .BI addr " netaddr
365 bdf5b5cd 2005-07-13 devnull network address to announce venti service
366 bdf5b5cd 2005-07-13 devnull (default
367 bdf5b5cd 2005-07-13 devnull .BR tcp!*!venti )
368 bdf5b5cd 2005-07-13 devnull .TP
369 bdf5b5cd 2005-07-13 devnull .BI httpaddr " netaddr
370 bdf5b5cd 2005-07-13 devnull network address to announce HTTP service
371 bdf5b5cd 2005-07-13 devnull (default
372 bdf5b5cd 2005-07-13 devnull .BR tcp!*!http )
373 bdf5b5cd 2005-07-13 devnull .TP
374 bdf5b5cd 2005-07-13 devnull .B queuewrites
375 bdf5b5cd 2005-07-13 devnull queue writes in memory
376 bdf5b5cd 2005-07-13 devnull (default is not to queue)
377 bdf5b5cd 2005-07-13 devnull .TP
378 bdf5b5cd 2005-07-13 devnull .BI webroot " dir
379 bdf5b5cd 2005-07-13 devnull directory tree containing files for HTTP server
380 bdf5b5cd 2005-07-13 devnull to consult for unrecognized URLs
381 bdf5b5cd 2005-07-13 devnull .PD
382 bdf5b5cd 2005-07-13 devnull .PP
383 bdf5b5cd 2005-07-13 devnull The units for the various cache sizes above can be specified by appending a
384 bdf5b5cd 2005-07-13 devnull .LR k ,
385 bdf5b5cd 2005-07-13 devnull .LR m ,
386 bdf5b5cd 2005-07-13 devnull or
387 bdf5b5cd 2005-07-13 devnull .LR g
388 bdf5b5cd 2005-07-13 devnull (case-insensitive)
389 bdf5b5cd 2005-07-13 devnull to indicate kilobytes, megabytes, or gigabytes respectively.
390 06466bd0 2005-07-24 devnull .PP
391 06466bd0 2005-07-24 devnull The
392 06466bd0 2005-07-24 devnull .I file
393 06466bd0 2005-07-24 devnull name in the configuration lines above can be of the form
394 06466bd0 2005-07-24 devnull .IB file : lo - hi
395 06466bd0 2005-07-24 devnull to specify a range of the file.
396 06466bd0 2005-07-24 devnull .I Lo
397 06466bd0 2005-07-24 devnull and
398 06466bd0 2005-07-24 devnull .I hi
399 06466bd0 2005-07-24 devnull are specified in bytes but can have the usual
400 06466bd0 2005-07-24 devnull .BI k ,
401 06466bd0 2005-07-24 devnull .BI m ,
402 06466bd0 2005-07-24 devnull or
403 06466bd0 2005-07-24 devnull .B g
404 06466bd0 2005-07-24 devnull suffixes.
405 06466bd0 2005-07-24 devnull Either
406 06466bd0 2005-07-24 devnull .I lo
407 06466bd0 2005-07-24 devnull or
408 06466bd0 2005-07-24 devnull .I hi
409 06466bd0 2005-07-24 devnull may be omitted.
410 06466bd0 2005-07-24 devnull This notation eliminates the need to
411 06466bd0 2005-07-24 devnull partition raw disks on non-Plan 9 systems.
412 bdf5b5cd 2005-07-13 devnull .SS Command Line
413 3aec33fe 2005-07-18 devnull Many of the options to Venti duplicate parameters that
414 3aec33fe 2005-07-18 devnull can be specified in the configuration file.
415 3aec33fe 2005-07-18 devnull The command line options override those found in a
416 3aec33fe 2005-07-18 devnull configuration file.
417 3aec33fe 2005-07-18 devnull Additional options are:
418 bdf5b5cd 2005-07-13 devnull .TP
419 bdf5b5cd 2005-07-13 devnull .BI -c " config
420 bdf5b5cd 2005-07-13 devnull The server configuration file
421 bdf5b5cd 2005-07-13 devnull (default
422 bdf5b5cd 2005-07-13 devnull .BR venti.conf )
423 bdf5b5cd 2005-07-13 devnull .TP
424 bdf5b5cd 2005-07-13 devnull .B -d
425 bdf5b5cd 2005-07-13 devnull Produce various debugging information on standard error.
426 bdf5b5cd 2005-07-13 devnull Implies
427 bdf5b5cd 2005-07-13 devnull .BR -s .
428 bdf5b5cd 2005-07-13 devnull .TP
429 bdf5b5cd 2005-07-13 devnull .B -L
430 bdf5b5cd 2005-07-13 devnull Enable logging. By default all logging is disabled.
431 bdf5b5cd 2005-07-13 devnull Logging slows server operation considerably.
432 bdf5b5cd 2005-07-13 devnull .TP
433 bdf5b5cd 2005-07-13 devnull .B -s
434 bdf5b5cd 2005-07-13 devnull Do not run in the background.
435 bdf5b5cd 2005-07-13 devnull Normally,
436 bdf5b5cd 2005-07-13 devnull the foreground process will exit once the Venti server
437 bdf5b5cd 2005-07-13 devnull is initialized and ready for connections.
438 bdf5b5cd 2005-07-13 devnull .PD
439 bdf5b5cd 2005-07-13 devnull .SH EXAMPLE
440 bdf5b5cd 2005-07-13 devnull A simple configuration:
441 bdf5b5cd 2005-07-13 devnull .IP
442 bdf5b5cd 2005-07-13 devnull .EX
443 bdf5b5cd 2005-07-13 devnull % cat venti.conf
444 bdf5b5cd 2005-07-13 devnull index main
445 bdf5b5cd 2005-07-13 devnull isect /tmp/disks/isect0
446 bdf5b5cd 2005-07-13 devnull isect /tmp/disks/isect1
447 bdf5b5cd 2005-07-13 devnull arenas /tmp/disks/arenas
448 eb5fbb97 2005-08-11 devnull bloom /tmp/disks/bloom
449 bdf5b5cd 2005-07-13 devnull mem 10M
450 bdf5b5cd 2005-07-13 devnull bcmem 20M
451 bdf5b5cd 2005-07-13 devnull icmem 30M
452 bdf5b5cd 2005-07-13 devnull %
453 bdf5b5cd 2005-07-13 devnull .EE
454 bdf5b5cd 2005-07-13 devnull .PP
455 bdf5b5cd 2005-07-13 devnull Format the index sections, the arena partition, and
456 bdf5b5cd 2005-07-13 devnull finally the main index:
457 bdf5b5cd 2005-07-13 devnull .IP
458 bdf5b5cd 2005-07-13 devnull .EX
459 bdf5b5cd 2005-07-13 devnull % venti/fmtisect isect0. /tmp/disks/isect0 &
460 bdf5b5cd 2005-07-13 devnull % venti/fmtisect isect1. /tmp/disks/isect1 &
461 bdf5b5cd 2005-07-13 devnull % venti/fmtarenas arenas0. /tmp/disks/arenas &
462 eb5fbb97 2005-08-11 devnull % venti/fmtbloom /tmp/disks/bloom &
463 bdf5b5cd 2005-07-13 devnull % wait
464 bdf5b5cd 2005-07-13 devnull % venti/fmtindex venti.conf
465 bdf5b5cd 2005-07-13 devnull %
466 bdf5b5cd 2005-07-13 devnull .EE
467 bdf5b5cd 2005-07-13 devnull .PP
468 bdf5b5cd 2005-07-13 devnull Start the server and check the storage statistics:
469 bdf5b5cd 2005-07-13 devnull .IP
470 bdf5b5cd 2005-07-13 devnull .EX
471 bdf5b5cd 2005-07-13 devnull % venti/venti
472 bdf5b5cd 2005-07-13 devnull % hget http://$sysname/storage
473 bdf5b5cd 2005-07-13 devnull .EE
474 3aec33fe 2005-07-18 devnull .SH SOURCE
475 3aec33fe 2005-07-18 devnull .B \*9/src/cmd/venti/srv
476 bdf5b5cd 2005-07-13 devnull .SH "SEE ALSO"
477 bdf5b5cd 2005-07-13 devnull .IR venti (1),
478 bdf5b5cd 2005-07-13 devnull .IR venti (3),
479 bdf5b5cd 2005-07-13 devnull .IR venti (7),
480 bdf5b5cd 2005-07-13 devnull .IR venti-backup (8)
481 bdf5b5cd 2005-07-13 devnull .IR venti-fmt (8)
482 bdf5b5cd 2005-07-13 devnull .br
483 bdf5b5cd 2005-07-13 devnull Sean Quinlan and Sean Dorward,
484 bdf5b5cd 2005-07-13 devnull ``Venti: a new approach to archival storage'',
485 bdf5b5cd 2005-07-13 devnull .I "Usenix Conference on File and Storage Technologies" ,
486 bdf5b5cd 2005-07-13 devnull 2002.
487 bdf5b5cd 2005-07-13 devnull .SH BUGS
488 bdf5b5cd 2005-07-13 devnull Setting up a venti server is too complicated.
489 bdf5b5cd 2005-07-13 devnull .PP
490 bdf5b5cd 2005-07-13 devnull Venti should not require the user to decide how to
491 bdf5b5cd 2005-07-13 devnull partition its memory usage.
492 3aec33fe 2005-07-18 devnull .PP
493 3aec33fe 2005-07-18 devnull Users of shells other than
494 3aec33fe 2005-07-18 devnull .IR rc (1)
495 3aec33fe 2005-07-18 devnull will not be able to use the program names shown.
496 3aec33fe 2005-07-18 devnull One solution is to define
497 3aec33fe 2005-07-18 devnull .B "V=$PLAN9/bin/venti"
498 3aec33fe 2005-07-18 devnull and then substitute
499 3aec33fe 2005-07-18 devnull .B $V/
500 3aec33fe 2005-07-18 devnull for
501 3aec33fe 2005-07-18 devnull .B venti/
502 3aec33fe 2005-07-18 devnull in the paths above.