1 => /post/extracting-from-zips.gmi Part two: “Extracting files from zips”
3 => //git.omarpolo.com/zip-utils/ The code for the whole series; see ‘zipls.c’ for this post in particular.
5 Edit 2021/08/20: some edits to improve the code and the commentary.
6 Edit 2021/08/21: stricter while condition for ‘ls’ and added links to the code
8 Disclaimer: before today I didn’t knew anything about how zip files are structured, so take everything here with a huge grain of salt. The good news is that the code I wrote seems to be coherent with what I’ve read online and to actually work against some zips files I had around.
10 Background: I’d like to add support for gempubs to Telescope, the Gemini client I’m writing. gempubs are basically a directory of text/gemini files plus other assets (metadata.txt and images presumably) all zipped in a single archive.
12 => https://codeberg.org/oppenlab/gempub gempub: a new eBook format based on text/gemini
13 => //telescope.omarpolo.com Telescope
15 There are a lot of libraries to handle zip files, but I decided to give it a shot a writing something from scratch. After all, I don’t need to edit zips or do fancy stuff, I only need to read files from the archive, that’s all.
17 To start, in this entry we’ll only see how to dump the list of files in a zip archive. Maybe future entries will deal with more zip stuff.
19 From what I’ve gathered from APPNOTE.TXT and other sources, a zip file is a sequence of file “records” (a header followed by the file content) and a trailing “central directory” that holds the information about all the files.
21 => https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT APPNOTE.TXT
22 => https://users.cs.jmu.edu/buchhofp/forensics/formats/pkzip.html The structure of a PKZip file
23 => https://en.wikipedia.org/wiki/ZIP_(file_format) ZIP (Wikipedia)
25 Having the central directory at the end of the file instead that at the beginning seems to be a choice to waste people time^W^W^W allow embedding zips into other file formats, such as GIFs or EXE. I guess in some cases this may be an invaluable property, I just fail to see where, but anyway.
27 Edit 2021/08/20: Another advantage of having the central directory at the end is that is probably possible to build up a zip on-the-fly, maybe outputting to standard output or to a similar non-seekable device, without having to build all the zip in memory first.
29 One may think that it’s possible to scan a zip by reading these “records”, but it’s not the case unfortunately: the only source of truth for the actual files stored in the archive is the central directory. Applications that modify the zip may reuse or leave dummy file headers around, especially if they delete or replace files.
31 To aggravate the situation, it’s not obvious how to find the start of the central directory. Zip are truly wonderful, huh? I guess that adding a trailing 4-byte offset that points to the start of the central directory wouldn’t be bad, but we’re a bit too late.
33 The central directory is a sequence of record that identifies the files in the archive followed by a digital signature, two ZIP64 fields and the end of the central directory record. I still haven’t wrapped my head around the digital signature and the ZIP64 fields, but they don’t seem necessary to access the list of files.
35 The last part of the central directory, the end record, contains a handy pointer to the start of the content directory. Unfortunately, it also contains a trailing variable-width comment area that complicate things a bit.
37 But enough with the talks, let’s jump to the code. Since Telescope is written in C, the small toy program object of this entry will also be written in C. The main function is pretty straightforward:
41 main(int argc, char **argv)
48 errx(1, "missing file to inspect");
50 if ((fd = open(argv[1], O_RDONLY)) == -1)
51 err(1, "open %s", argv[1]);
53 zip = map_file(fd, &len);
54 if ((cd = find_central_directory(zip, len)) == NULL)
55 errx(1, "can't find central directory");
66 I think it would be easier for us to just mmap(2) the file into memory rather than moving back and forward by means of lseek(2). map_file is a thin wrapper around mmap(2):
68 ```implementation of the map_file function
70 map_file(int fd, size_t *len)
75 if ((jump = lseek(fd, 0, SEEK_END)) == -1)
78 if (lseek(fd, 0, SEEK_SET) == -1)
81 if ((addr = mmap(NULL, jump, PROT_READ, MAP_PRIVATE, fd, 0))
90 Just as we were discussing before, to locate the central directory we must first locate the “end of central directory record”. Its structure is as follows (the numbers inside the brackets indicates the byte count)
92 ```structure of the end of central directory record
93 signature[4] disk_number[2] disk_cd_number[2] disk_entries[2]
94 total_entrie[2] central_directory_size[4] cd_offset[4]
95 comment_len[2] comment…
98 The signature is always “\x50\x4b\x05\x06”, which helps in finding the record. We still need to be careful, since I haven’t seen anywhere that the signature MUST NOT appear inside the comment.
100 To be sure that we’ve actually found the real start of the end record, there’s a explicit check: the comment length plus the size of the non-variable part of the header must be equal to how far we have travelled from the end of the file. Granted, this is not completely bulletproof, since a specially-crafted comment may appear like a proper end of central directory record, but I’m not sure what could we do better to protect against faulty files.
102 Side note: as always, I’m treating these files as untrusted and do all the possible checks. You don’t want a malformed file to crash your program, don’t you?
104 One last thing: I’m totally fine with a very light and sparse usage of gotos. In find_central_directory I’m using a ‘goto again’ when we find a false signature inside a comment. A while loop would also do that, but it’d be a bit uglier.
106 ```the find_central_directory procedure
108 find_central_directory(uint8_t *addr, size_t len)
115 * At -22 bytes from the end there is the end of the central
116 * directory assuming an empty comment. It's a sensible place
125 for (; p > addr; --p)
126 if (memcmp(p, "\x50\x4b\x05\x06", 4) == 0)
132 /* read comment length */
133 memcpy(&clen, p + 20, sizeof(clen));
134 clen = le16toh(clen);
136 /* false signature inside a comment? */
137 if (clen + 22 != end - p) {
142 /* read the offset for the central directory */
143 memcpy(&offset, p + 16, sizeof(offset));
144 offset = le32toh(offset);
146 if (addr + offset > p)
149 return addr + offset;
153 Edit 2021/08/20: there’s a space for a little optimisation: the end record MUST be in the last 64kb (plus some bytes), so for big files there’s no need to continue searching back until the start. Why 64kb? The comment length is a 16 bit integer, so the biggest end of record possible is 22 bytes plus 64kb of comment.
155 If everything went well, we’ve found the pointer to the start of the central directory. It’s made by a sequence of file header records:
158 signature[4] version[2] vers_needed[2] flags[2] compression[2]
159 mod_time[2] mod_date[2] crc32[4]
160 compressed_size[4] uncompressed_size[4]
161 filename_len[2] extra_field_len[2] file_comment_len[2]
162 disk_number[2] internal_attrs[2] offset[4]
163 filename… extra_field… file_comment…
166 The signature field is always "\x50\x4b\x01\x02", which is different from the end record and the other records fortunately. To list the files we just have to read the file headers record until we find one with a different signature:
168 ```ls: traverse the file headers and print the filenames
170 ls(uint8_t *zip, size_t len, uint8_t *cd)
173 uint16_t flen, xlen, clen;
175 char filename[PATH_MAX];
178 while (cd < end - 46 && memcmp(cd, "\x50\x4b\x01\x02", 4) == 0) {
179 memcpy(&flen, cd + 28, sizeof(flen));
180 memcpy(&xlen, cd + 28 + 2, sizeof(xlen));
181 memcpy(&clen, cd + 28 + 2 + 2, sizeof(xlen));
183 flen = le16toh(flen);
184 xlen = le16toh(xlen);
185 clen = le16toh(clen);
187 memcpy(&offset, cd + 42, sizeof(offset));
188 offset = le32toh(offset);
190 memset(filename, 0, sizeof(filename));
191 memcpy(filename, cd + 46, MIN(sizeof(filename)-1, flen));
193 printf("%s [%d]\n", filename, offset);
195 cd += 46 + flen + xlen + clen;
200 As always, there are some magic numbers hardcoded, a real program would probably have some constants defined, but for this simple toy program I’m fine with things as is. Also, note the pedantry in ensuring we don’t end up reading out-of-bounds in the while condition, I don’t want faulty zip files to cause invalid memory access.
202 Now, to compile it and run:
205 % cc zipls.c -o zipls && ./zipls star_maker_olaf_stapledon.gpub
207 chapter_1_1_the_starting_point.gmi [2957]
208 chapter_1_2_earth_among_the_stars.gmi [6932]
209 chapter_2_1_interstellar_travel.gmi [11041]
210 chapter_3_1_on_the_other_earth.gmi [20382]
216 To conclude this entry, one of the things that I’m still not sure about is the endiannes of the numbers. I’m guessing they should be little endian, but it’s always that or only because the zip files were produced on a little endian machine?
218 Edit 2021/08/20: The majority of the number are stored in little-endian. There are some exception, so check the documentation, but is mostly for fields like the MSDOS-like time and date and stuff like that. The code was updated with the calls to leXYtoh() from ‘endian.h’.
220 Otherwise I’m pretty happy with the result. In a short time I went from knowing nothing about zips to being able to at least inspect them, using only the C standard library (well, assuming POSIX). I’ll leave the files decoding for a next time.
