3 venti \- archival storage server
5 Venti is a block storage server intended for archival data.
6 In a Venti server, the SHA1 hash of a block's contents acts
7 as the block identifier for read and write operations.
8 This approach enforces a write-once policy, preventing
9 accidental or malicious destruction of data. In addition,
10 duplicate copies of a block are coalesced, reducing the
11 consumption of storage and simplifying the implementation
14 This manual page documents the basic concepts of
15 block storage using Venti as well as the Venti network protocol.
18 documents some simple clients.
24 are more complex clients.
27 describes a C library interface for accessing
28 Venti servers and manipulating Venti data structures.
31 describes the Venti server configuration file.
34 describes the programs used to run a Venti server.
37 The SHA1 hash that identifies a block is called its
39 The score of the zero-length block is called the
42 Scores may have an optional
44 prefix, typically used to
45 describe the format of the data.
52 uses prefixes corresponding to the file system
57 .SS "Files and Directories
58 Venti accepts blocks up to 56 kilobytes in size.
59 By convention, Venti clients use hash trees of blocks to
60 represent arbitrary-size data
62 The data to be stored is split into fixed-size
63 blocks and written to the server, producing a list
65 The resulting list of scores is split into fixed-size pointer
66 blocks (using only an integral number of scores per block)
67 and written to the server, producing a smaller list
69 The process continues, eventually ending with the
70 score for the hash tree's top-most block.
71 Each file stored this way is summarized by
74 structure recording the top-most score, the depth
75 of the tree, the data block size, and the pointer block size.
78 structures can be concatenated
79 and stored as a special file called a
82 manner, arbitrary trees of files can be constructed
85 Scores passed between programs conventionally refer
88 blocks, which contain descriptive information
89 as well as the score of a block containing a small number
93 To allow programs to traverse these structures without
94 needing to understand their higher-level meanings,
95 Venti tags each block with a type. The types are:
99 VtDataType 000 \f1data\fL
100 VtDataType+1 001 \fRscores of \fPVtDataType\fR blocks\fL
101 VtDataType+2 002 \fRscores of \fPVtDataType+1\fR blocks\fL
103 VtDirType 010 VtEntry\fR structures\fL
104 VtDirType+1 011 \fRscores of \fLVtDirType\fR blocks\fL
105 VtDirType+2 012 \fRscores of \fLVtDirType+1\fR blocks\fL
107 VtRootType 020 VtRoot\fR structure\fL
110 The octal numbers listed are the type numbers used
111 by the commands below.
112 (For historical reasons, the type numbers used on
113 disk and on the wire are different from the above.
114 They do not distinguish
120 To avoid storing the same short data blocks padded with
121 differing numbers of zeros, Venti clients working with fixed-size
122 blocks conventionally
123 `zero truncate' the blocks before writing them to the server.
124 For example, if a 1024-byte data block contains the
126 .RB ` hello " " world '
127 followed by 1013 zero bytes,
128 a client would store only the 11-byte block.
129 When the client later read the block from the server,
130 it would append zeros to the end as necessary to
131 reach the expected size.
133 When truncating pointer blocks
134 .RB ( VtDataType+ \fIn
138 trailing zero scores are removed
139 instead of trailing zero bytes.
141 Because of the truncation convention,
142 any file consisting entirely of zero bytes,
143 no matter what the length, will be represented by the zero score:
144 the data blocks contain all zeros and are thus truncated
145 to the empty block, and the pointer blocks contain all zero scores
146 and are thus also truncated to the empty block,
147 and so on up the hash tree.
149 A Venti session begins when a
151 connects to the network address served by a Venti
153 the conventional address is
154 .BI tcp! server !venti
158 Both client and server begin by sending a version
160 .BI venti- versions - comment \en \fR.
163 field is a list of acceptable versions separated by
165 The protocol described here is version
167 The client is responsible for choosing a common
168 version and sending it in the
170 message, described below.
172 After the initial version exchange, the client transmits
175 to the server, which subsequently returns
179 The combined act of transmitting (receiving) a request
180 of a particular type, and receiving (transmitting) its reply
185 Each message consists of a sequence of bytes.
186 Two-byte fields hold unsigned integers represented
187 in big-endian order (most significant byte first).
188 Data items of variable lengths are represented by
189 a one-byte field specifying a count,
194 Text strings are represented similarly,
195 using a two-byte count with
196 the text itself stored as a UTF-8 encoded sequence
197 of Unicode characters (see
203 counts the bytes of UTF-8 data, which include no final
207 character is illegal in text strings in the Venti protocol.
208 The maximum string length in Venti is 1024 bytes.
210 Each Venti message begins with a two-byte size field
211 specifying the length in bytes of the message,
212 not including the length field itself.
213 The next byte is the message type, one of the constants
214 in the enumeration in the include file
216 The next byte is an identifying
218 used to match responses with requests.
219 The remaining bytes are parameters of different sizes.
220 In the message descriptions, the number of bytes in a field
221 is given in brackets after the field name.
226 is not a constant represents a variable-length parameter:
230 bytes of data forming the
246 is the last field in the message represents a
247 variable-length field that comprises all remaining
248 bytes in the message.
250 All Venti RPC messages are prefixed with a field
252 giving the length of the message that follows
256 The message bodies are:
257 .ta \w'\fLVtTgoodbye 'u
320 Each T-message has a one-byte
322 field, chosen and used by the client to identify the message.
323 The server will echo the request's
326 Clients should arrange that no two outstanding
327 messages have the same tag field so that responses
328 can be distinguished.
330 The type of an R-message will either be one greater than
331 the type of the corresponding T-message or
333 indicating that the request failed.
334 In the latter case, the
336 field contains a string describing the reason for failure.
338 Venti connections must begin with a
343 message contains the protocol
345 that the client has chosen to use.
351 could be used to add authentication, encryption,
352 and compression to the Venti session
353 but are currently ignored.
360 response are similarly ignored.
365 fields are intended to be the identity
366 of the client and server but, given the lack of
367 authentication, should be treated only as advisory.
372 transaction during the session.
376 message has no effect and
377 is used mainly for debugging.
378 Servers should respond immediately to pings.
382 message requests a block with the given
392 to convert a block type enumeration value
397 used on disk and in the protocol.
400 field specifies the maximum expected size
404 in the reply is the block's contents.
408 message writes a new block of the given
413 The response includes the
415 to use to read the block,
416 which should be the SHA1 hash of
419 The Venti server may buffer written blocks in memory,
420 waiting until after responding to the
422 message before writing them to
424 The server will delay the response to a
426 message until after all blocks in earlier
428 messages have been written to permanent storage.
432 message ends a session. There is no
436 message, the server terminates up the connection.
442 Sean Quinlan and Sean Dorward,
443 ``Venti: a new approach to archival storage'',
444 .I "Usenix Conference on File and Storage Technologies" ,
