op public repos

Blame

Date:: Sun Oct 23 00:37:19 2022 UTC
Message:: introduce gotd(8), a Git repository server reachable via ssh(1) This is an initial barebones implementation which provides the absolute minimum of functionality required to serve got(1) and git(1) clients. Basic fetch/send functionality has been tested and seems to work here, but this server is not yet expected to be stable. More testing is welcome. See the man pages for setup instructions. The current design uses one reader and one writer process per repository, which will have to be extended to N readers and N writers in the future. At startup, each process will chroot(2) into its assigned repository. This works because gotd(8) can only be started as root, and will then fork+exec, chroot, and privdrop. At present the parent process runs with the following pledge(2) promises: "stdio rpath wpath cpath proc getpw sendfd recvfd fattr flock unix unveil" The parent is the only process able to modify the repository in a way that becomes visible to Git clients. The parent uses unveil(2) to restrict its view of the filesystem to /tmp and the repositories listed in the configuration file gotd.conf(5). Per-repository chroot(2) processes use "stdio rpath sendfd recvfd". The writer defers to the parent for modifying references in the repository to point at newly uploaded commits. The reader is fine without such help, because Git repositories can be read without having to create any lock-files. gotd(8) requires a dedicated user ID, which should own repositories on the filesystem, and a separate secondary group, which should not have filesystem-level repository access, and must be allowed access to the gotd(8) socket. To obtain Git repository access, users must be members of this secondary group, and must have their login shell set to gotsh(1). gotsh(1) connects to the gotd(8) socket and speaks Git-protocol towards the client on the other end of the SSH connection. gotsh(1) is not an interactive command shell. At present, authenticated clients are granted read/write access to all repositories and all references (except for the "refs/got/" and the "refs/remotes/" namespaces, which are already being protected from modification). While complicated access control mechanism are not a design goal, making it possible to safely offer anonymous Git repository access over ssh(1) is on the road map.
Actions:: History | Blob | Raw File
   1 e6bcace5 2021-06-22 stsp /*
   2 e6bcace5 2021-06-22 stsp  * Copyright (c) 2020 Ori Bernstein
   3 e6bcace5 2021-06-22 stsp  * Copyright (c) 2021 Stefan Sperling <stsp@openbsd.org>
   4 e6bcace5 2021-06-22 stsp  *
   5 e6bcace5 2021-06-22 stsp  * Permission to use, copy, modify, and distribute this software for any
   6 e6bcace5 2021-06-22 stsp  * purpose with or without fee is hereby granted, provided that the above
   7 e6bcace5 2021-06-22 stsp  * copyright notice and this permission notice appear in all copies.
   8 e6bcace5 2021-06-22 stsp  *
   9 e6bcace5 2021-06-22 stsp  * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
  10 e6bcace5 2021-06-22 stsp  * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
  11 e6bcace5 2021-06-22 stsp  * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
  12 e6bcace5 2021-06-22 stsp  * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
  13 e6bcace5 2021-06-22 stsp  * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
  14 e6bcace5 2021-06-22 stsp  * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
  15 e6bcace5 2021-06-22 stsp  * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
  16 e6bcace5 2021-06-22 stsp  */
  17 e6bcace5 2021-06-22 stsp 
  18 08736cf9 2021-06-23 stsp #include <sys/types.h>
  19 e6bcace5 2021-06-22 stsp #include <sys/queue.h>
  20 f8b19efd 2021-10-13 stsp #include <sys/tree.h>
  21 08736cf9 2021-06-23 stsp #include <sys/uio.h>
  22 e6bcace5 2021-06-22 stsp #include <sys/stat.h>
  23 211cfef0 2022-01-05 stsp #include <sys/time.h>
  24 e93fb944 2022-05-10 stsp #include <sys/mman.h>
  25 e6bcace5 2021-06-22 stsp 
  26 e3f86256 2022-02-18 naddy #include <endian.h>
  27 e93fb944 2022-05-10 stsp #include <errno.h>
  28 08736cf9 2021-06-23 stsp #include <stdint.h>
  29 05118f5a 2021-06-22 stsp #include <imsg.h>
  30 756050ac 2022-07-19 op #include <inttypes.h>
  31 13b2bc37 2022-10-23 stsp #include <poll.h>
  32 e6bcace5 2021-06-22 stsp #include <stdio.h>
  33 e6bcace5 2021-06-22 stsp #include <stdlib.h>
  34 e6bcace5 2021-06-22 stsp #include <string.h>
  35 e6bcace5 2021-06-22 stsp #include <sha1.h>
  36 211cfef0 2022-01-05 stsp #include <time.h>
  37 bfc73a47 2022-03-19 naddy #include <unistd.h>
  38 e6bcace5 2021-06-22 stsp #include <limits.h>
  39 e6bcace5 2021-06-22 stsp #include <zlib.h>
  40 e6bcace5 2021-06-22 stsp 
  41 e6bcace5 2021-06-22 stsp #include "got_error.h"
  42 e6bcace5 2021-06-22 stsp #include "got_cancel.h"
  43 e6bcace5 2021-06-22 stsp #include "got_object.h"
  44 f8b19efd 2021-10-13 stsp #include "got_path.h"
  45 05118f5a 2021-06-22 stsp #include "got_reference.h"
  46 05118f5a 2021-06-22 stsp #include "got_repository_admin.h"
  47 e6bcace5 2021-06-22 stsp 
  48 e6bcace5 2021-06-22 stsp #include "got_lib_deltify.h"
  49 e6bcace5 2021-06-22 stsp #include "got_lib_delta.h"
  50 e6bcace5 2021-06-22 stsp #include "got_lib_object.h"
  51 e6bcace5 2021-06-22 stsp #include "got_lib_object_idset.h"
  52 05118f5a 2021-06-22 stsp #include "got_lib_object_cache.h"
  53 e6bcace5 2021-06-22 stsp #include "got_lib_deflate.h"
  54 cae60ab8 2022-10-18 stsp #include "got_lib_ratelimit.h"
  55 e6bcace5 2021-06-22 stsp #include "got_lib_pack.h"
  56 17cfdba6 2022-05-20 op #include "got_lib_pack_create.h"
  57 05118f5a 2021-06-22 stsp #include "got_lib_repository.h"
  58 2d9e6abf 2022-05-04 stsp #include "got_lib_inflate.h"
  59 13b2bc37 2022-10-23 stsp #include "got_lib_poll.h"
  60 e6bcace5 2021-06-22 stsp 
  61 f8174ca5 2022-05-20 stsp #include "murmurhash2.h"
  62 f8174ca5 2022-05-20 stsp 
  63 74881701 2021-10-15 stsp #ifndef MIN
  64 74881701 2021-10-15 stsp #define	MIN(_a,_b) ((_a) < (_b) ? (_a) : (_b))
  65 74881701 2021-10-15 stsp #endif
  66 74881701 2021-10-15 stsp 
  67 e6bcace5 2021-06-22 stsp #ifndef MAX
  68 e6bcace5 2021-06-22 stsp #define	MAX(_a,_b) ((_a) > (_b) ? (_a) : (_b))
  69 e6bcace5 2021-06-22 stsp #endif
  70 e6bcace5 2021-06-22 stsp 
  71 70f8f24d 2022-04-14 stsp #ifndef nitems
  72 70f8f24d 2022-04-14 stsp #define nitems(_a)	(sizeof((_a)) / sizeof((_a)[0]))
  73 70f8f24d 2022-04-14 stsp #endif
  74 e6bcace5 2021-06-22 stsp 
  75 e6bcace5 2021-06-22 stsp static const struct got_error *
  76 e6bcace5 2021-06-22 stsp alloc_meta(struct got_pack_meta **new, struct got_object_id *id,
  77 d6a28ffe 2022-05-20 op     const char *path, int obj_type, time_t mtime, uint32_t seed)
  78 e6bcace5 2021-06-22 stsp {
  79 e6bcace5 2021-06-22 stsp 	struct got_pack_meta *m;
  80 e6bcace5 2021-06-22 stsp 
  81 e6bcace5 2021-06-22 stsp 	*new = NULL;
  82 e6bcace5 2021-06-22 stsp 
  83 e6bcace5 2021-06-22 stsp 	m = calloc(1, sizeof(*m));
  84 e6bcace5 2021-06-22 stsp 	if (m == NULL)
  85 1d19226a 2021-10-13 stsp 		return got_error_from_errno("calloc");
  86 e6bcace5 2021-06-22 stsp 
  87 e6bcace5 2021-06-22 stsp 	memcpy(&m->id, id, sizeof(m->id));
  88 e6bcace5 2021-06-22 stsp 
  89 d6a28ffe 2022-05-20 op 	m->path_hash = murmurhash2(path, strlen(path), seed);
  90 e6bcace5 2021-06-22 stsp 	m->obj_type = obj_type;
  91 e6bcace5 2021-06-22 stsp 	m->mtime = mtime;
  92 e6bcace5 2021-06-22 stsp 	*new = m;
  93 e6bcace5 2021-06-22 stsp 	return NULL;
  94 e6bcace5 2021-06-22 stsp }
  95 e6bcace5 2021-06-22 stsp 
  96 e6bcace5 2021-06-22 stsp static void
  97 e6bcace5 2021-06-22 stsp clear_meta(struct got_pack_meta *meta)
  98 e6bcace5 2021-06-22 stsp {
  99 e6bcace5 2021-06-22 stsp 	if (meta == NULL)
 100 e6bcace5 2021-06-22 stsp 		return;
 101 f8174ca5 2022-05-20 stsp 	meta->path_hash = 0;
 102 5060d5a1 2022-01-10 stsp 	free(meta->delta_buf);
 103 5060d5a1 2022-01-10 stsp 	meta->delta_buf = NULL;
 104 67fd6849 2022-02-13 stsp 	free(meta->base_obj_id);
 105 67fd6849 2022-02-13 stsp 	meta->base_obj_id = NULL;
 106 411cbec1 2022-05-20 stsp 	meta->reused_delta_offset = 0;
 107 e6bcace5 2021-06-22 stsp }
 108 e6bcace5 2021-06-22 stsp 
 109 e6bcace5 2021-06-22 stsp static void
 110 e6bcace5 2021-06-22 stsp free_nmeta(struct got_pack_meta **meta, int nmeta)
 111 e6bcace5 2021-06-22 stsp {
 112 e6bcace5 2021-06-22 stsp 	int i;
 113 e6bcace5 2021-06-22 stsp 
 114 e6bcace5 2021-06-22 stsp 	for (i = 0; i < nmeta; i++)
 115 e6bcace5 2021-06-22 stsp 		clear_meta(meta[i]);
 116 e6bcace5 2021-06-22 stsp 	free(meta);
 117 e6bcace5 2021-06-22 stsp }
 118 e6bcace5 2021-06-22 stsp 
 119 e6bcace5 2021-06-22 stsp static int
 120 e6bcace5 2021-06-22 stsp delta_order_cmp(const void *pa, const void *pb)
 121 e6bcace5 2021-06-22 stsp {
 122 e6bcace5 2021-06-22 stsp 	struct got_pack_meta *a, *b;
 123 e6bcace5 2021-06-22 stsp 
 124 e6bcace5 2021-06-22 stsp 	a = *(struct got_pack_meta **)pa;
 125 e6bcace5 2021-06-22 stsp 	b = *(struct got_pack_meta **)pb;
 126 e6bcace5 2021-06-22 stsp 
 127 e6bcace5 2021-06-22 stsp 	if (a->obj_type != b->obj_type)
 128 e6bcace5 2021-06-22 stsp 		return a->obj_type - b->obj_type;
 129 f8174ca5 2022-05-20 stsp 	if (a->path_hash < b->path_hash)
 130 f8174ca5 2022-05-20 stsp 		return -1;
 131 f8174ca5 2022-05-20 stsp 	if (a->path_hash > b->path_hash)
 132 f8174ca5 2022-05-20 stsp 		return 1;
 133 611e8e31 2022-05-01 stsp 	if (a->mtime < b->mtime)
 134 611e8e31 2022-05-01 stsp 		return -1;
 135 611e8e31 2022-05-01 stsp 	if (a->mtime > b->mtime)
 136 611e8e31 2022-05-01 stsp 		return 1;
 137 e6bcace5 2021-06-22 stsp 	return got_object_id_cmp(&a->id, &b->id);
 138 e6bcace5 2021-06-22 stsp }
 139 e6bcace5 2021-06-22 stsp 
 140 5060d5a1 2022-01-10 stsp static off_t
 141 e6bcace5 2021-06-22 stsp delta_size(struct got_delta_instruction *deltas, int ndeltas)
 142 e6bcace5 2021-06-22 stsp {
 143 5060d5a1 2022-01-10 stsp 	int i;
 144 5060d5a1 2022-01-10 stsp 	off_t size = 32;
 145 e6bcace5 2021-06-22 stsp 	for (i = 0; i < ndeltas; i++) {
 146 e6bcace5 2021-06-22 stsp 		if (deltas[i].copy)
 147 e6bcace5 2021-06-22 stsp 			size += GOT_DELTA_SIZE_SHIFT;
 148 e6bcace5 2021-06-22 stsp 		else
 149 e6bcace5 2021-06-22 stsp 			size += deltas[i].len + 1;
 150 e6bcace5 2021-06-22 stsp 	}
 151 e6bcace5 2021-06-22 stsp 	return size;
 152 5060d5a1 2022-01-10 stsp }
 153 5060d5a1 2022-01-10 stsp 
 154 5060d5a1 2022-01-10 stsp static const struct got_error *
 155 5060d5a1 2022-01-10 stsp append(unsigned char **p, size_t *len, off_t *sz, void *seg, int nseg)
 156 5060d5a1 2022-01-10 stsp {
 157 5060d5a1 2022-01-10 stsp 	char *n;
 158 5060d5a1 2022-01-10 stsp 
 159 5060d5a1 2022-01-10 stsp 	if (*len + nseg >= *sz) {
 160 5060d5a1 2022-01-10 stsp 		while (*len + nseg >= *sz)
 161 5060d5a1 2022-01-10 stsp 			*sz += *sz / 2;
 162 5060d5a1 2022-01-10 stsp 		n = realloc(*p, *sz);
 163 5060d5a1 2022-01-10 stsp 		if (n == NULL)
 164 5060d5a1 2022-01-10 stsp 			return got_error_from_errno("realloc");
 165 5060d5a1 2022-01-10 stsp 		*p = n;
 166 5060d5a1 2022-01-10 stsp 	}
 167 5060d5a1 2022-01-10 stsp 	memcpy(*p + *len, seg, nseg);
 168 5060d5a1 2022-01-10 stsp 	*len += nseg;
 169 5060d5a1 2022-01-10 stsp 	return NULL;
 170 5060d5a1 2022-01-10 stsp }
 171 5060d5a1 2022-01-10 stsp 
 172 5060d5a1 2022-01-10 stsp static const struct got_error *
 173 5060d5a1 2022-01-10 stsp encode_delta_in_mem(struct got_pack_meta *m, struct got_raw_object *o,
 174 5060d5a1 2022-01-10 stsp     struct got_delta_instruction *deltas, int ndeltas,
 175 5060d5a1 2022-01-10 stsp     off_t delta_size, off_t base_size)
 176 5060d5a1 2022-01-10 stsp {
 177 5060d5a1 2022-01-10 stsp 	const struct got_error *err;
 178 5060d5a1 2022-01-10 stsp 	unsigned char buf[16], *bp;
 179 5060d5a1 2022-01-10 stsp 	int i, j;
 180 2d9e6abf 2022-05-04 stsp 	size_t len = 0, compressed_len;
 181 2d9e6abf 2022-05-04 stsp 	off_t bufsize = delta_size;
 182 5060d5a1 2022-01-10 stsp 	off_t n;
 183 5060d5a1 2022-01-10 stsp 	struct got_delta_instruction *d;
 184 2d9e6abf 2022-05-04 stsp 	uint8_t *delta_buf;
 185 5060d5a1 2022-01-10 stsp 
 186 2d9e6abf 2022-05-04 stsp 	delta_buf = malloc(bufsize);
 187 2d9e6abf 2022-05-04 stsp 	if (delta_buf == NULL)
 188 2d9e6abf 2022-05-04 stsp 		return got_error_from_errno("malloc");
 189 5060d5a1 2022-01-10 stsp 
 190 5060d5a1 2022-01-10 stsp 	/* base object size */
 191 5060d5a1 2022-01-10 stsp 	buf[0] = base_size & GOT_DELTA_SIZE_VAL_MASK;
 192 5060d5a1 2022-01-10 stsp 	n = base_size >> GOT_DELTA_SIZE_SHIFT;
 193 5060d5a1 2022-01-10 stsp 	for (i = 1; n > 0; i++) {
 194 5060d5a1 2022-01-10 stsp 		buf[i - 1] |= GOT_DELTA_SIZE_MORE;
 195 5060d5a1 2022-01-10 stsp 		buf[i] = n & GOT_DELTA_SIZE_VAL_MASK;
 196 5060d5a1 2022-01-10 stsp 		n >>= GOT_DELTA_SIZE_SHIFT;
 197 5060d5a1 2022-01-10 stsp 	}
 198 2d9e6abf 2022-05-04 stsp 	err = append(&delta_buf, &len, &bufsize, buf, i);
 199 5060d5a1 2022-01-10 stsp 	if (err)
 200 2d9e6abf 2022-05-04 stsp 		goto done;
 201 5060d5a1 2022-01-10 stsp 
 202 5060d5a1 2022-01-10 stsp 	/* target object size */
 203 5060d5a1 2022-01-10 stsp 	buf[0] = o->size & GOT_DELTA_SIZE_VAL_MASK;
 204 5060d5a1 2022-01-10 stsp 	n = o->size >> GOT_DELTA_SIZE_SHIFT;
 205 5060d5a1 2022-01-10 stsp 	for (i = 1; n > 0; i++) {
 206 5060d5a1 2022-01-10 stsp 		buf[i - 1] |= GOT_DELTA_SIZE_MORE;
 207 5060d5a1 2022-01-10 stsp 		buf[i] = n & GOT_DELTA_SIZE_VAL_MASK;
 208 5060d5a1 2022-01-10 stsp 		n >>= GOT_DELTA_SIZE_SHIFT;
 209 5060d5a1 2022-01-10 stsp 	}
 210 2d9e6abf 2022-05-04 stsp 	err = append(&delta_buf, &len, &bufsize, buf, i);
 211 5060d5a1 2022-01-10 stsp 	if (err)
 212 2d9e6abf 2022-05-04 stsp 		goto done;
 213 5060d5a1 2022-01-10 stsp 
 214 5060d5a1 2022-01-10 stsp 	for (j = 0; j < ndeltas; j++) {
 215 5060d5a1 2022-01-10 stsp 		d = &deltas[j];
 216 5060d5a1 2022-01-10 stsp 		if (d->copy) {
 217 5060d5a1 2022-01-10 stsp 			n = d->offset;
 218 5060d5a1 2022-01-10 stsp 			bp = &buf[1];
 219 5060d5a1 2022-01-10 stsp 			buf[0] = GOT_DELTA_BASE_COPY;
 220 5060d5a1 2022-01-10 stsp 			for (i = 0; i < 4; i++) {
 221 5060d5a1 2022-01-10 stsp 				/* DELTA_COPY_OFF1 ... DELTA_COPY_OFF4 */
 222 5060d5a1 2022-01-10 stsp 				buf[0] |= 1 << i;
 223 5060d5a1 2022-01-10 stsp 				*bp++ = n & 0xff;
 224 5060d5a1 2022-01-10 stsp 				n >>= 8;
 225 5060d5a1 2022-01-10 stsp 				if (n == 0)
 226 5060d5a1 2022-01-10 stsp 					break;
 227 5060d5a1 2022-01-10 stsp 			}
 228 5060d5a1 2022-01-10 stsp 
 229 5060d5a1 2022-01-10 stsp 			n = d->len;
 230 5060d5a1 2022-01-10 stsp 			if (n != GOT_DELTA_COPY_DEFAULT_LEN) {
 231 5060d5a1 2022-01-10 stsp 				/* DELTA_COPY_LEN1 ... DELTA_COPY_LEN3 */
 232 5060d5a1 2022-01-10 stsp 				for (i = 0; i < 3 && n > 0; i++) {
 233 5060d5a1 2022-01-10 stsp 					buf[0] |= 1 << (i + 4);
 234 5060d5a1 2022-01-10 stsp 					*bp++ = n & 0xff;
 235 5060d5a1 2022-01-10 stsp 					n >>= 8;
 236 5060d5a1 2022-01-10 stsp 				}
 237 5060d5a1 2022-01-10 stsp 			}
 238 2d9e6abf 2022-05-04 stsp 			err = append(&delta_buf, &len, &bufsize,
 239 5060d5a1 2022-01-10 stsp 			    buf, bp - buf);
 240 5060d5a1 2022-01-10 stsp 			if (err)
 241 2d9e6abf 2022-05-04 stsp 				goto done;
 242 5060d5a1 2022-01-10 stsp 		} else if (o->f == NULL) {
 243 5060d5a1 2022-01-10 stsp 			n = 0;
 244 5060d5a1 2022-01-10 stsp 			while (n != d->len) {
 245 5060d5a1 2022-01-10 stsp 				buf[0] = (d->len - n < 127) ? d->len - n : 127;
 246 2d9e6abf 2022-05-04 stsp 				err = append(&delta_buf, &len, &bufsize,
 247 5060d5a1 2022-01-10 stsp 				    buf, 1);
 248 5060d5a1 2022-01-10 stsp 				if (err)
 249 2d9e6abf 2022-05-04 stsp 					goto done;
 250 2d9e6abf 2022-05-04 stsp 				err = append(&delta_buf, &len, &bufsize,
 251 5060d5a1 2022-01-10 stsp 				    o->data + o->hdrlen + d->offset + n,
 252 5060d5a1 2022-01-10 stsp 				    buf[0]);
 253 5060d5a1 2022-01-10 stsp 				if (err)
 254 2d9e6abf 2022-05-04 stsp 					goto done;
 255 5060d5a1 2022-01-10 stsp 				n += buf[0];
 256 5060d5a1 2022-01-10 stsp 			}
 257 5060d5a1 2022-01-10 stsp 		} else {
 258 5060d5a1 2022-01-10 stsp 			char content[128];
 259 5060d5a1 2022-01-10 stsp 			size_t r;
 260 2d9e6abf 2022-05-04 stsp 			if (fseeko(o->f, o->hdrlen + d->offset, SEEK_SET) == -1) {
 261 2d9e6abf 2022-05-04 stsp 				err = got_error_from_errno("fseeko");
 262 2d9e6abf 2022-05-04 stsp 				goto done;
 263 2d9e6abf 2022-05-04 stsp 			}
 264 5060d5a1 2022-01-10 stsp 			n = 0;
 265 5060d5a1 2022-01-10 stsp 			while (n != d->len) {
 266 5060d5a1 2022-01-10 stsp 				buf[0] = (d->len - n < 127) ? d->len - n : 127;
 267 2d9e6abf 2022-05-04 stsp 				err = append(&delta_buf, &len, &bufsize,
 268 5060d5a1 2022-01-10 stsp 				    buf, 1);
 269 5060d5a1 2022-01-10 stsp 				if (err)
 270 2d9e6abf 2022-05-04 stsp 					goto done;
 271 5060d5a1 2022-01-10 stsp 				r = fread(content, 1, buf[0], o->f);
 272 2d9e6abf 2022-05-04 stsp 				if (r != buf[0]) {
 273 2d9e6abf 2022-05-04 stsp 					err = got_ferror(o->f, GOT_ERR_IO);
 274 2d9e6abf 2022-05-04 stsp 					goto done;
 275 2d9e6abf 2022-05-04 stsp 				}
 276 2d9e6abf 2022-05-04 stsp 				err = append(&delta_buf, &len, &bufsize,
 277 5060d5a1 2022-01-10 stsp 				    content, buf[0]);
 278 5060d5a1 2022-01-10 stsp 				if (err)
 279 2d9e6abf 2022-05-04 stsp 					goto done;
 280 5060d5a1 2022-01-10 stsp 				n += buf[0];
 281 5060d5a1 2022-01-10 stsp 			}
 282 5060d5a1 2022-01-10 stsp 		}
 283 5060d5a1 2022-01-10 stsp 	}
 284 5060d5a1 2022-01-10 stsp 
 285 2d9e6abf 2022-05-04 stsp 	err = got_deflate_to_mem_mmap(&m->delta_buf, &compressed_len,
 286 2d9e6abf 2022-05-04 stsp 	    NULL, NULL, delta_buf, 0, len);
 287 2d9e6abf 2022-05-04 stsp 	if (err)
 288 2d9e6abf 2022-05-04 stsp 		goto done;
 289 2d9e6abf 2022-05-04 stsp 
 290 5060d5a1 2022-01-10 stsp 	m->delta_len = len;
 291 2d9e6abf 2022-05-04 stsp 	m->delta_compressed_len = compressed_len;
 292 2d9e6abf 2022-05-04 stsp done:
 293 2d9e6abf 2022-05-04 stsp 	free(delta_buf);
 294 2d9e6abf 2022-05-04 stsp 	return err;
 295 e6bcace5 2021-06-22 stsp }
 296 e6bcace5 2021-06-22 stsp 
 297 74881701 2021-10-15 stsp static const struct got_error *
 298 74881701 2021-10-15 stsp encode_delta(struct got_pack_meta *m, struct got_raw_object *o,
 299 74881701 2021-10-15 stsp     struct got_delta_instruction *deltas, int ndeltas,
 300 a319ca8c 2021-10-15 stsp     off_t base_size, FILE *f)
 301 a319ca8c 2021-10-15 stsp {
 302 2d9e6abf 2022-05-04 stsp 	const struct got_error *err;
 303 a319ca8c 2021-10-15 stsp 	unsigned char buf[16], *bp;
 304 a319ca8c 2021-10-15 stsp 	int i, j;
 305 a319ca8c 2021-10-15 stsp 	off_t n;
 306 2d9e6abf 2022-05-04 stsp 	struct got_deflate_buf zb;
 307 a319ca8c 2021-10-15 stsp 	struct got_delta_instruction *d;
 308 2d9e6abf 2022-05-04 stsp 	off_t delta_len = 0, compressed_len = 0;
 309 a319ca8c 2021-10-15 stsp 
 310 2d9e6abf 2022-05-04 stsp 	err = got_deflate_init(&zb, NULL, GOT_DEFLATE_BUFSIZE);
 311 2d9e6abf 2022-05-04 stsp 	if (err)
 312 2d9e6abf 2022-05-04 stsp 		return err;
 313 2d9e6abf 2022-05-04 stsp 
 314 a319ca8c 2021-10-15 stsp 	/* base object size */
 315 a319ca8c 2021-10-15 stsp 	buf[0] = base_size & GOT_DELTA_SIZE_VAL_MASK;
 316 a319ca8c 2021-10-15 stsp 	n = base_size >> GOT_DELTA_SIZE_SHIFT;
 317 a319ca8c 2021-10-15 stsp 	for (i = 1; n > 0; i++) {
 318 a319ca8c 2021-10-15 stsp 		buf[i - 1] |= GOT_DELTA_SIZE_MORE;
 319 a319ca8c 2021-10-15 stsp 		buf[i] = n & GOT_DELTA_SIZE_VAL_MASK;
 320 a319ca8c 2021-10-15 stsp 		n >>= GOT_DELTA_SIZE_SHIFT;
 321 a319ca8c 2021-10-15 stsp 	}
 322 a319ca8c 2021-10-15 stsp 
 323 2d9e6abf 2022-05-04 stsp 	err = got_deflate_append_to_file_mmap(&zb, &compressed_len,
 324 2d9e6abf 2022-05-04 stsp 	    buf, 0, i, f, NULL);
 325 2d9e6abf 2022-05-04 stsp 	if (err)
 326 2d9e6abf 2022-05-04 stsp 		goto done;
 327 2d9e6abf 2022-05-04 stsp 	delta_len += i;
 328 2d9e6abf 2022-05-04 stsp 
 329 a319ca8c 2021-10-15 stsp 	/* target object size */
 330 a319ca8c 2021-10-15 stsp 	buf[0] = o->size & GOT_DELTA_SIZE_VAL_MASK;
 331 a319ca8c 2021-10-15 stsp 	n = o->size >> GOT_DELTA_SIZE_SHIFT;
 332 a319ca8c 2021-10-15 stsp 	for (i = 1; n > 0; i++) {
 333 a319ca8c 2021-10-15 stsp 		buf[i - 1] |= GOT_DELTA_SIZE_MORE;
 334 a319ca8c 2021-10-15 stsp 		buf[i] = n & GOT_DELTA_SIZE_VAL_MASK;
 335 a319ca8c 2021-10-15 stsp 		n >>= GOT_DELTA_SIZE_SHIFT;
 336 a319ca8c 2021-10-15 stsp 	}
 337 a319ca8c 2021-10-15 stsp 
 338 2d9e6abf 2022-05-04 stsp 	err = got_deflate_append_to_file_mmap(&zb, &compressed_len,
 339 2d9e6abf 2022-05-04 stsp 	    buf, 0, i, f, NULL);
 340 2d9e6abf 2022-05-04 stsp 	if (err)
 341 2d9e6abf 2022-05-04 stsp 		goto done;
 342 2d9e6abf 2022-05-04 stsp 	delta_len += i;
 343 2d9e6abf 2022-05-04 stsp 
 344 a319ca8c 2021-10-15 stsp 	for (j = 0; j < ndeltas; j++) {
 345 a319ca8c 2021-10-15 stsp 		d = &deltas[j];
 346 a319ca8c 2021-10-15 stsp 		if (d->copy) {
 347 a319ca8c 2021-10-15 stsp 			n = d->offset;
 348 a319ca8c 2021-10-15 stsp 			bp = &buf[1];
 349 a319ca8c 2021-10-15 stsp 			buf[0] = GOT_DELTA_BASE_COPY;
 350 a319ca8c 2021-10-15 stsp 			for (i = 0; i < 4; i++) {
 351 a319ca8c 2021-10-15 stsp 				/* DELTA_COPY_OFF1 ... DELTA_COPY_OFF4 */
 352 a319ca8c 2021-10-15 stsp 				buf[0] |= 1 << i;
 353 a319ca8c 2021-10-15 stsp 				*bp++ = n & 0xff;
 354 a319ca8c 2021-10-15 stsp 				n >>= 8;
 355 a319ca8c 2021-10-15 stsp 				if (n == 0)
 356 a319ca8c 2021-10-15 stsp 					break;
 357 a319ca8c 2021-10-15 stsp 			}
 358 a319ca8c 2021-10-15 stsp 			n = d->len;
 359 a319ca8c 2021-10-15 stsp 			if (n != GOT_DELTA_COPY_DEFAULT_LEN) {
 360 a319ca8c 2021-10-15 stsp 				/* DELTA_COPY_LEN1 ... DELTA_COPY_LEN3 */
 361 a319ca8c 2021-10-15 stsp 				for (i = 0; i < 3 && n > 0; i++) {
 362 a319ca8c 2021-10-15 stsp 					buf[0] |= 1 << (i + 4);
 363 a319ca8c 2021-10-15 stsp 					*bp++ = n & 0xff;
 364 a319ca8c 2021-10-15 stsp 					n >>= 8;
 365 a319ca8c 2021-10-15 stsp 				}
 366 a319ca8c 2021-10-15 stsp 			}
 367 2d9e6abf 2022-05-04 stsp 			err = got_deflate_append_to_file_mmap(&zb,
 368 2d9e6abf 2022-05-04 stsp 			    &compressed_len, buf, 0, bp - buf, f, NULL);
 369 2d9e6abf 2022-05-04 stsp 			if (err)
 370 2d9e6abf 2022-05-04 stsp 				goto done;
 371 2d9e6abf 2022-05-04 stsp 			delta_len += (bp - buf);
 372 64a8571e 2022-01-07 stsp 		} else if (o->f == NULL) {
 373 64a8571e 2022-01-07 stsp 			n = 0;
 374 64a8571e 2022-01-07 stsp 			while (n != d->len) {
 375 64a8571e 2022-01-07 stsp 				buf[0] = (d->len - n < 127) ? d->len - n : 127;
 376 2d9e6abf 2022-05-04 stsp 				err = got_deflate_append_to_file_mmap(&zb,
 377 2d9e6abf 2022-05-04 stsp 				    &compressed_len, buf, 0, 1, f, NULL);
 378 2d9e6abf 2022-05-04 stsp 				if (err)
 379 2d9e6abf 2022-05-04 stsp 					goto done;
 380 2d9e6abf 2022-05-04 stsp 				delta_len++;
 381 2d9e6abf 2022-05-04 stsp 				err = got_deflate_append_to_file_mmap(&zb,
 382 2d9e6abf 2022-05-04 stsp 				    &compressed_len,
 383 2d9e6abf 2022-05-04 stsp 				    o->data + o->hdrlen + d->offset + n, 0,
 384 2d9e6abf 2022-05-04 stsp 				    buf[0], f, NULL);
 385 2d9e6abf 2022-05-04 stsp 				if (err)
 386 2d9e6abf 2022-05-04 stsp 					goto done;
 387 2d9e6abf 2022-05-04 stsp 				delta_len += buf[0];
 388 64a8571e 2022-01-07 stsp 				n += buf[0];
 389 64a8571e 2022-01-07 stsp 			}
 390 a319ca8c 2021-10-15 stsp 		} else {
 391 a319ca8c 2021-10-15 stsp 			char content[128];
 392 a319ca8c 2021-10-15 stsp 			size_t r;
 393 2d9e6abf 2022-05-04 stsp 			if (fseeko(o->f, o->hdrlen + d->offset, SEEK_SET) == -1) {
 394 2d9e6abf 2022-05-04 stsp 				err = got_error_from_errno("fseeko");
 395 2d9e6abf 2022-05-04 stsp 				goto done;
 396 2d9e6abf 2022-05-04 stsp 			}
 397 a319ca8c 2021-10-15 stsp 			n = 0;
 398 a319ca8c 2021-10-15 stsp 			while (n != d->len) {
 399 a319ca8c 2021-10-15 stsp 				buf[0] = (d->len - n < 127) ? d->len - n : 127;
 400 2d9e6abf 2022-05-04 stsp 				err = got_deflate_append_to_file_mmap(&zb,
 401 2d9e6abf 2022-05-04 stsp 				    &compressed_len, buf, 0, 1, f, NULL);
 402 2d9e6abf 2022-05-04 stsp 				if (err)
 403 2d9e6abf 2022-05-04 stsp 					goto done;
 404 2d9e6abf 2022-05-04 stsp 				delta_len++;
 405 a319ca8c 2021-10-15 stsp 				r = fread(content, 1, buf[0], o->f);
 406 2d9e6abf 2022-05-04 stsp 				if (r != buf[0]) {
 407 2d9e6abf 2022-05-04 stsp 					err = got_ferror(o->f, GOT_ERR_IO);
 408 2d9e6abf 2022-05-04 stsp 					goto done;
 409 2d9e6abf 2022-05-04 stsp 				}
 410 2d9e6abf 2022-05-04 stsp 				err = got_deflate_append_to_file_mmap(&zb,
 411 2d9e6abf 2022-05-04 stsp 				    &compressed_len, content, 0, buf[0], f,
 412 2d9e6abf 2022-05-04 stsp 				    NULL);
 413 2d9e6abf 2022-05-04 stsp 				if (err)
 414 2d9e6abf 2022-05-04 stsp 					goto done;
 415 2d9e6abf 2022-05-04 stsp 				delta_len += buf[0];
 416 a319ca8c 2021-10-15 stsp 				n += buf[0];
 417 a319ca8c 2021-10-15 stsp 			}
 418 a319ca8c 2021-10-15 stsp 		}
 419 a319ca8c 2021-10-15 stsp 	}
 420 e6bcace5 2021-06-22 stsp 
 421 2d9e6abf 2022-05-04 stsp 	err = got_deflate_flush(&zb, f, NULL, &compressed_len);
 422 2d9e6abf 2022-05-04 stsp 	if (err)
 423 2d9e6abf 2022-05-04 stsp 		goto done;
 424 2d9e6abf 2022-05-04 stsp 
 425 2d9e6abf 2022-05-04 stsp 	/* sanity check */
 426 2d9e6abf 2022-05-04 stsp 	if (compressed_len != ftello(f) - m->delta_offset) {
 427 2d9e6abf 2022-05-04 stsp 		err = got_error(GOT_ERR_COMPRESSION);
 428 2d9e6abf 2022-05-04 stsp 		goto done;
 429 2d9e6abf 2022-05-04 stsp 	}
 430 2d9e6abf 2022-05-04 stsp 
 431 2d9e6abf 2022-05-04 stsp 	m->delta_len = delta_len;
 432 2d9e6abf 2022-05-04 stsp 	m->delta_compressed_len = compressed_len;
 433 2d9e6abf 2022-05-04 stsp done:
 434 2d9e6abf 2022-05-04 stsp 	got_deflate_end(&zb);
 435 2d9e6abf 2022-05-04 stsp 	return err;
 436 a319ca8c 2021-10-15 stsp }
 437 211cfef0 2022-01-05 stsp 
 438 301e83b3 2022-10-16 stsp const struct got_error *
 439 301e83b3 2022-10-16 stsp got_pack_report_progress(got_pack_progress_cb progress_cb, void *progress_arg,
 440 b8af7c06 2022-03-15 stsp     struct got_ratelimit *rl, int ncolored, int nfound, int ntrees,
 441 b8af7c06 2022-03-15 stsp     off_t packfile_size, int ncommits, int nobj_total, int obj_deltify,
 442 b8af7c06 2022-03-15 stsp     int nobj_written)
 443 211cfef0 2022-01-05 stsp {
 444 211cfef0 2022-01-05 stsp 	const struct got_error *err;
 445 211cfef0 2022-01-05 stsp 	int elapsed;
 446 211cfef0 2022-01-05 stsp 
 447 211cfef0 2022-01-05 stsp 	if (progress_cb == NULL)
 448 211cfef0 2022-01-05 stsp 		return NULL;
 449 211cfef0 2022-01-05 stsp 
 450 211cfef0 2022-01-05 stsp 	err = got_ratelimit_check(&elapsed, rl);
 451 211cfef0 2022-01-05 stsp 	if (err || !elapsed)
 452 211cfef0 2022-01-05 stsp 		return err;
 453 a319ca8c 2021-10-15 stsp 
 454 b8af7c06 2022-03-15 stsp 	return progress_cb(progress_arg, ncolored, nfound, ntrees,
 455 b8af7c06 2022-03-15 stsp 	    packfile_size, ncommits, nobj_total, obj_deltify, nobj_written);
 456 211cfef0 2022-01-05 stsp }
 457 a319ca8c 2021-10-15 stsp 
 458 301e83b3 2022-10-16 stsp const struct got_error *
 459 301e83b3 2022-10-16 stsp got_pack_add_meta(struct got_pack_meta *m, struct got_pack_metavec *v)
 460 67fd6849 2022-02-13 stsp {
 461 67fd6849 2022-02-13 stsp 	if (v->nmeta == v->metasz){
 462 67fd6849 2022-02-13 stsp 		size_t newsize = 2 * v->metasz;
 463 67fd6849 2022-02-13 stsp 		struct got_pack_meta **new;
 464 67fd6849 2022-02-13 stsp 		new = reallocarray(v->meta, newsize, sizeof(*new));
 465 67fd6849 2022-02-13 stsp 		if (new == NULL)
 466 67fd6849 2022-02-13 stsp 			return got_error_from_errno("reallocarray");
 467 67fd6849 2022-02-13 stsp 		v->meta = new;
 468 5e91dae4 2022-08-30 stsp 		v->metasz = newsize;
 469 67fd6849 2022-02-13 stsp 	}
 470 67fd6849 2022-02-13 stsp 
 471 67fd6849 2022-02-13 stsp 	v->meta[v->nmeta++] = m;
 472 67fd6849 2022-02-13 stsp 	return NULL;
 473 67fd6849 2022-02-13 stsp }
 474 67fd6849 2022-02-13 stsp 
 475 301e83b3 2022-10-16 stsp const struct got_error *
 476 301e83b3 2022-10-16 stsp got_pack_find_pack_for_reuse(struct got_packidx **best_packidx,
 477 67fd6849 2022-02-13 stsp     struct got_repository *repo)
 478 67fd6849 2022-02-13 stsp {
 479 9b576444 2022-03-14 stsp 	const struct got_error *err = NULL;
 480 67fd6849 2022-02-13 stsp 	struct got_pathlist_entry *pe;
 481 67fd6849 2022-02-13 stsp 	const char *best_packidx_path = NULL;
 482 67fd6849 2022-02-13 stsp 	int nobj_max = 0;
 483 67fd6849 2022-02-13 stsp 
 484 67fd6849 2022-02-13 stsp 	*best_packidx = NULL;
 485 67fd6849 2022-02-13 stsp 
 486 9b576444 2022-03-14 stsp 	TAILQ_FOREACH(pe, &repo->packidx_paths, entry) {
 487 67fd6849 2022-02-13 stsp 		const char *path_packidx = pe->path;
 488 67fd6849 2022-02-13 stsp 		struct got_packidx *packidx;
 489 67fd6849 2022-02-13 stsp 		int nobj;
 490 67fd6849 2022-02-13 stsp 
 491 67fd6849 2022-02-13 stsp 		err = got_repo_get_packidx(&packidx, path_packidx, repo);
 492 67fd6849 2022-02-13 stsp 		if (err)
 493 67fd6849 2022-02-13 stsp 			break;
 494 67fd6849 2022-02-13 stsp 
 495 67fd6849 2022-02-13 stsp 		nobj = be32toh(packidx->hdr.fanout_table[0xff]);
 496 67fd6849 2022-02-13 stsp 		if (nobj > nobj_max) {
 497 67fd6849 2022-02-13 stsp 			best_packidx_path = path_packidx;
 498 67fd6849 2022-02-13 stsp 			nobj_max = nobj;
 499 67fd6849 2022-02-13 stsp 		}
 500 67fd6849 2022-02-13 stsp 	}
 501 67fd6849 2022-02-13 stsp 
 502 67fd6849 2022-02-13 stsp 	if (best_packidx_path) {
 503 67fd6849 2022-02-13 stsp 		err = got_repo_get_packidx(best_packidx, best_packidx_path,
 504 67fd6849 2022-02-13 stsp 		    repo);
 505 67fd6849 2022-02-13 stsp 	}
 506 67fd6849 2022-02-13 stsp 
 507 67fd6849 2022-02-13 stsp 	return err;
 508 67fd6849 2022-02-13 stsp }
 509 67fd6849 2022-02-13 stsp 
 510 301e83b3 2022-10-16 stsp const struct got_error *
 511 301e83b3 2022-10-16 stsp got_pack_cache_pack_for_packidx(struct got_pack **pack,
 512 301e83b3 2022-10-16 stsp     struct got_packidx *packidx, struct got_repository *repo)
 513 3d589bee 2022-06-25 stsp {
 514 61af9b21 2022-06-28 stsp 	const struct got_error *err;
 515 3d589bee 2022-06-25 stsp 	char *path_packfile = NULL;
 516 3d589bee 2022-06-25 stsp 
 517 3d589bee 2022-06-25 stsp 	err = got_packidx_get_packfile_path(&path_packfile,
 518 3d589bee 2022-06-25 stsp 	    packidx->path_packidx);
 519 3d589bee 2022-06-25 stsp 	if (err)
 520 3d589bee 2022-06-25 stsp 		return err;
 521 3d589bee 2022-06-25 stsp 
 522 3d589bee 2022-06-25 stsp 	*pack = got_repo_get_cached_pack(repo, path_packfile);
 523 3d589bee 2022-06-25 stsp 	if (*pack == NULL) {
 524 3d589bee 2022-06-25 stsp 		err = got_repo_cache_pack(pack, repo, path_packfile, packidx);
 525 3d589bee 2022-06-25 stsp 		if (err)
 526 3d589bee 2022-06-25 stsp 			goto done;
 527 3d589bee 2022-06-25 stsp 	}
 528 61af9b21 2022-06-28 stsp done:
 529 61af9b21 2022-06-28 stsp 	free(path_packfile);
 530 67fd6849 2022-02-13 stsp 	return err;
 531 67fd6849 2022-02-13 stsp }
 532 67fd6849 2022-02-13 stsp 
 533 67fd6849 2022-02-13 stsp static const struct got_error *
 534 b8af7c06 2022-03-15 stsp pick_deltas(struct got_pack_meta **meta, int nmeta, int ncolored,
 535 b8af7c06 2022-03-15 stsp     int nfound, int ntrees, int ncommits, int nreused, FILE *delta_cache,
 536 b8af7c06 2022-03-15 stsp     struct got_repository *repo,
 537 05118f5a 2021-06-22 stsp     got_pack_progress_cb progress_cb, void *progress_arg,
 538 211cfef0 2022-01-05 stsp     struct got_ratelimit *rl, got_cancel_cb cancel_cb, void *cancel_arg)
 539 e6bcace5 2021-06-22 stsp {
 540 e6bcace5 2021-06-22 stsp 	const struct got_error *err = NULL;
 541 e6bcace5 2021-06-22 stsp 	struct got_pack_meta *m = NULL, *base = NULL;
 542 e6bcace5 2021-06-22 stsp 	struct got_raw_object *raw = NULL, *base_raw = NULL;
 543 74881701 2021-10-15 stsp 	struct got_delta_instruction *deltas = NULL, *best_deltas = NULL;
 544 5060d5a1 2022-01-10 stsp 	int i, j, ndeltas, best_ndeltas;
 545 5060d5a1 2022-01-10 stsp 	off_t size, best_size;
 546 4f4d853e 2021-10-24 stsp 	const int max_base_candidates = 3;
 547 402a5ec1 2022-01-10 stsp 	size_t delta_memsize = 0;
 548 adb4bbb2 2022-05-20 stsp 	const size_t max_delta_memsize = 4 * GOT_DELTA_RESULT_SIZE_CACHED_MAX;
 549 cc7a354a 2021-10-15 stsp 	int outfd = -1;
 550 d6a28ffe 2022-05-20 op 	uint32_t delta_seed;
 551 d6a28ffe 2022-05-20 op 
 552 d6a28ffe 2022-05-20 op 	delta_seed = arc4random();
 553 e6bcace5 2021-06-22 stsp 
 554 e6bcace5 2021-06-22 stsp 	qsort(meta, nmeta, sizeof(struct got_pack_meta *), delta_order_cmp);
 555 e6bcace5 2021-06-22 stsp 	for (i = 0; i < nmeta; i++) {
 556 e6bcace5 2021-06-22 stsp 		if (cancel_cb) {
 557 e6bcace5 2021-06-22 stsp 			err = (*cancel_cb)(cancel_arg);
 558 e6bcace5 2021-06-22 stsp 			if (err)
 559 e6bcace5 2021-06-22 stsp 				break;
 560 e6bcace5 2021-06-22 stsp 		}
 561 301e83b3 2022-10-16 stsp 		err = got_pack_report_progress(progress_cb, progress_arg, rl,
 562 b8af7c06 2022-03-15 stsp 		    ncolored, nfound, ntrees, 0L, ncommits, nreused + nmeta,
 563 b8af7c06 2022-03-15 stsp 		    nreused + i, 0);
 564 211cfef0 2022-01-05 stsp 		if (err)
 565 211cfef0 2022-01-05 stsp 			goto done;
 566 e6bcace5 2021-06-22 stsp 		m = meta[i];
 567 e6bcace5 2021-06-22 stsp 
 568 e6bcace5 2021-06-22 stsp 		if (m->obj_type == GOT_OBJ_TYPE_COMMIT ||
 569 e6bcace5 2021-06-22 stsp 		    m->obj_type == GOT_OBJ_TYPE_TAG)
 570 e6bcace5 2021-06-22 stsp 			continue;
 571 e6bcace5 2021-06-22 stsp 
 572 94dac27c 2021-10-15 stsp 		err = got_object_raw_open(&raw, &outfd, repo, &m->id);
 573 e6bcace5 2021-06-22 stsp 		if (err)
 574 e6bcace5 2021-06-22 stsp 			goto done;
 575 600b755e 2021-10-14 stsp 		m->size = raw->size;
 576 e6bcace5 2021-06-22 stsp 
 577 64a8571e 2022-01-07 stsp 		if (raw->f == NULL) {
 578 64a8571e 2022-01-07 stsp 			err = got_deltify_init_mem(&m->dtab, raw->data,
 579 d6a28ffe 2022-05-20 op 			    raw->hdrlen, raw->size + raw->hdrlen, delta_seed);
 580 64a8571e 2022-01-07 stsp 		} else {
 581 64a8571e 2022-01-07 stsp 			err = got_deltify_init(&m->dtab, raw->f, raw->hdrlen,
 582 d6a28ffe 2022-05-20 op 			    raw->size + raw->hdrlen, delta_seed);
 583 64a8571e 2022-01-07 stsp 		}
 584 e6bcace5 2021-06-22 stsp 		if (err)
 585 e6bcace5 2021-06-22 stsp 			goto done;
 586 e6bcace5 2021-06-22 stsp 
 587 e6bcace5 2021-06-22 stsp 		if (i > max_base_candidates) {
 588 e6bcace5 2021-06-22 stsp 			struct got_pack_meta *n = NULL;
 589 e6bcace5 2021-06-22 stsp 			n = meta[i - (max_base_candidates + 1)];
 590 e6bcace5 2021-06-22 stsp 			got_deltify_free(n->dtab);
 591 e6bcace5 2021-06-22 stsp 			n->dtab = NULL;
 592 e6bcace5 2021-06-22 stsp 		}
 593 e6bcace5 2021-06-22 stsp 
 594 74881701 2021-10-15 stsp 		best_size = raw->size;
 595 74881701 2021-10-15 stsp 		best_ndeltas = 0;
 596 e6bcace5 2021-06-22 stsp 		for (j = MAX(0, i - max_base_candidates); j < i; j++) {
 597 e6bcace5 2021-06-22 stsp 			if (cancel_cb) {
 598 e6bcace5 2021-06-22 stsp 				err = (*cancel_cb)(cancel_arg);
 599 e6bcace5 2021-06-22 stsp 				if (err)
 600 e6bcace5 2021-06-22 stsp 					goto done;
 601 e6bcace5 2021-06-22 stsp 			}
 602 e6bcace5 2021-06-22 stsp 			base = meta[j];
 603 e6bcace5 2021-06-22 stsp 			/* long chains make unpacking slow, avoid such bases */
 604 22edbce7 2021-10-24 stsp 			if (base->nchain >= 128 ||
 605 e6bcace5 2021-06-22 stsp 			    base->obj_type != m->obj_type)
 606 e6bcace5 2021-06-22 stsp 				continue;
 607 e6bcace5 2021-06-22 stsp 
 608 d3c116bf 2021-10-15 stsp 			err = got_object_raw_open(&base_raw, &outfd, repo,
 609 94dac27c 2021-10-15 stsp 			    &base->id);
 610 e6bcace5 2021-06-22 stsp 			if (err)
 611 e6bcace5 2021-06-22 stsp 				goto done;
 612 67fd6849 2022-02-13 stsp 
 613 64a8571e 2022-01-07 stsp 			if (raw->f == NULL && base_raw->f == NULL) {
 614 64a8571e 2022-01-07 stsp 				err = got_deltify_mem_mem(&deltas, &ndeltas,
 615 64a8571e 2022-01-07 stsp 				    raw->data, raw->hdrlen,
 616 d6a28ffe 2022-05-20 op 				    raw->size + raw->hdrlen, delta_seed,
 617 64a8571e 2022-01-07 stsp 				    base->dtab, base_raw->data,
 618 64a8571e 2022-01-07 stsp 				    base_raw->hdrlen,
 619 64a8571e 2022-01-07 stsp 				    base_raw->size + base_raw->hdrlen);
 620 64a8571e 2022-01-07 stsp 			} else if (raw->f == NULL) {
 621 64a8571e 2022-01-07 stsp 				err = got_deltify_mem_file(&deltas, &ndeltas,
 622 64a8571e 2022-01-07 stsp 				    raw->data, raw->hdrlen,
 623 d6a28ffe 2022-05-20 op 				    raw->size + raw->hdrlen, delta_seed,
 624 64a8571e 2022-01-07 stsp 				    base->dtab, base_raw->f,
 625 64a8571e 2022-01-07 stsp 				    base_raw->hdrlen,
 626 64a8571e 2022-01-07 stsp 				    base_raw->size + base_raw->hdrlen);
 627 64a8571e 2022-01-07 stsp 			} else if (base_raw->f == NULL) {
 628 64a8571e 2022-01-07 stsp 				err = got_deltify_file_mem(&deltas, &ndeltas,
 629 64a8571e 2022-01-07 stsp 				    raw->f, raw->hdrlen,
 630 d6a28ffe 2022-05-20 op 				    raw->size + raw->hdrlen, delta_seed,
 631 64a8571e 2022-01-07 stsp 				    base->dtab, base_raw->data,
 632 64a8571e 2022-01-07 stsp 				    base_raw->hdrlen,
 633 64a8571e 2022-01-07 stsp 				    base_raw->size + base_raw->hdrlen);
 634 64a8571e 2022-01-07 stsp 			} else {
 635 64a8571e 2022-01-07 stsp 				err = got_deltify(&deltas, &ndeltas,
 636 64a8571e 2022-01-07 stsp 				    raw->f, raw->hdrlen,
 637 d6a28ffe 2022-05-20 op 				    raw->size + raw->hdrlen, delta_seed,
 638 64a8571e 2022-01-07 stsp 				    base->dtab, base_raw->f, base_raw->hdrlen,
 639 64a8571e 2022-01-07 stsp 				    base_raw->size + base_raw->hdrlen);
 640 64a8571e 2022-01-07 stsp 			}
 641 e6bcace5 2021-06-22 stsp 			got_object_raw_close(base_raw);
 642 e6bcace5 2021-06-22 stsp 			base_raw = NULL;
 643 e6bcace5 2021-06-22 stsp 			if (err)
 644 e6bcace5 2021-06-22 stsp 				goto done;
 645 e6bcace5 2021-06-22 stsp 
 646 e6bcace5 2021-06-22 stsp 			size = delta_size(deltas, ndeltas);
 647 74881701 2021-10-15 stsp 			if (size + 32 < best_size){
 648 e6bcace5 2021-06-22 stsp 				/*
 649 e6bcace5 2021-06-22 stsp 				 * if we already picked a best delta,
 650 e6bcace5 2021-06-22 stsp 				 * replace it.
 651 e6bcace5 2021-06-22 stsp 				 */
 652 74881701 2021-10-15 stsp 				best_size = size;
 653 74881701 2021-10-15 stsp 				free(best_deltas);
 654 74881701 2021-10-15 stsp 				best_deltas = deltas;
 655 74881701 2021-10-15 stsp 				best_ndeltas = ndeltas;
 656 74881701 2021-10-15 stsp 				deltas = NULL;
 657 e6bcace5 2021-06-22 stsp 				m->nchain = base->nchain + 1;
 658 e6bcace5 2021-06-22 stsp 				m->prev = base;
 659 e6bcace5 2021-06-22 stsp 				m->head = base->head;
 660 e6bcace5 2021-06-22 stsp 				if (m->head == NULL)
 661 e6bcace5 2021-06-22 stsp 					m->head = base;
 662 e6bcace5 2021-06-22 stsp 			} else {
 663 e6bcace5 2021-06-22 stsp 				free(deltas);
 664 e6bcace5 2021-06-22 stsp 				deltas = NULL;
 665 e6bcace5 2021-06-22 stsp 				ndeltas = 0;
 666 e6bcace5 2021-06-22 stsp 			}
 667 e6bcace5 2021-06-22 stsp 		}
 668 e6bcace5 2021-06-22 stsp 
 669 74881701 2021-10-15 stsp 		if (best_ndeltas > 0) {
 670 402a5ec1 2022-01-10 stsp 			if (best_size <= GOT_DELTA_RESULT_SIZE_CACHED_MAX &&
 671 402a5ec1 2022-01-10 stsp 			    delta_memsize + best_size <= max_delta_memsize) {
 672 402a5ec1 2022-01-10 stsp 				delta_memsize += best_size;
 673 5060d5a1 2022-01-10 stsp 				err = encode_delta_in_mem(m, raw, best_deltas,
 674 5060d5a1 2022-01-10 stsp 				    best_ndeltas, best_size, m->prev->size);
 675 5060d5a1 2022-01-10 stsp 			} else {
 676 5060d5a1 2022-01-10 stsp 				m->delta_offset = ftello(delta_cache);
 677 5060d5a1 2022-01-10 stsp 				err = encode_delta(m, raw, best_deltas,
 678 5060d5a1 2022-01-10 stsp 				    best_ndeltas, m->prev->size, delta_cache);
 679 5060d5a1 2022-01-10 stsp 			}
 680 74881701 2021-10-15 stsp 			free(best_deltas);
 681 74881701 2021-10-15 stsp 			best_deltas = NULL;
 682 74881701 2021-10-15 stsp 			best_ndeltas = 0;
 683 74881701 2021-10-15 stsp 			if (err)
 684 74881701 2021-10-15 stsp 				goto done;
 685 74881701 2021-10-15 stsp 		}
 686 74881701 2021-10-15 stsp 
 687 e6bcace5 2021-06-22 stsp 		got_object_raw_close(raw);
 688 e6bcace5 2021-06-22 stsp 		raw = NULL;
 689 e6bcace5 2021-06-22 stsp 	}
 690 e6bcace5 2021-06-22 stsp done:
 691 e6bcace5 2021-06-22 stsp 	for (i = MAX(0, nmeta - max_base_candidates); i < nmeta; i++) {
 692 e6bcace5 2021-06-22 stsp 		got_deltify_free(meta[i]->dtab);
 693 e6bcace5 2021-06-22 stsp 		meta[i]->dtab = NULL;
 694 e6bcace5 2021-06-22 stsp 	}
 695 e6bcace5 2021-06-22 stsp 	if (raw)
 696 e6bcace5 2021-06-22 stsp 		got_object_raw_close(raw);
 697 e6bcace5 2021-06-22 stsp 	if (base_raw)
 698 e6bcace5 2021-06-22 stsp 		got_object_raw_close(base_raw);
 699 cc7a354a 2021-10-15 stsp 	if (outfd != -1 && close(outfd) == -1 && err == NULL)
 700 cc7a354a 2021-10-15 stsp 		err = got_error_from_errno("close");
 701 74881701 2021-10-15 stsp 	free(deltas);
 702 74881701 2021-10-15 stsp 	free(best_deltas);
 703 e6bcace5 2021-06-22 stsp 	return err;
 704 e6bcace5 2021-06-22 stsp }
 705 e6bcace5 2021-06-22 stsp 
 706 e6bcace5 2021-06-22 stsp static const struct got_error *
 707 05118f5a 2021-06-22 stsp search_packidx(int *found, struct got_object_id *id,
 708 05118f5a 2021-06-22 stsp     struct got_repository *repo)
 709 05118f5a 2021-06-22 stsp {
 710 05118f5a 2021-06-22 stsp 	const struct got_error *err = NULL;
 711 05118f5a 2021-06-22 stsp 	struct got_packidx *packidx = NULL;
 712 05118f5a 2021-06-22 stsp 	int idx;
 713 05118f5a 2021-06-22 stsp 
 714 05118f5a 2021-06-22 stsp 	*found = 0;
 715 05118f5a 2021-06-22 stsp 
 716 05118f5a 2021-06-22 stsp 	err = got_repo_search_packidx(&packidx, &idx, repo, id);
 717 05118f5a 2021-06-22 stsp 	if (err == NULL)
 718 05118f5a 2021-06-22 stsp 		*found = 1; /* object is already packed */
 719 05118f5a 2021-06-22 stsp 	else if (err->code == GOT_ERR_NO_OBJ)
 720 05118f5a 2021-06-22 stsp 		err = NULL;
 721 05118f5a 2021-06-22 stsp 	return err;
 722 05118f5a 2021-06-22 stsp }
 723 05118f5a 2021-06-22 stsp 
 724 301e83b3 2022-10-16 stsp const struct got_error *
 725 301e83b3 2022-10-16 stsp got_pack_add_object(int want_meta, struct got_object_idset *idset,
 726 e6bcace5 2021-06-22 stsp     struct got_object_id *id, const char *path, int obj_type,
 727 d6a28ffe 2022-05-20 op     time_t mtime, uint32_t seed, int loose_obj_only,
 728 d6a28ffe 2022-05-20 op     struct got_repository *repo, int *ncolored, int *nfound, int *ntrees,
 729 6863cbf9 2022-03-21 stsp     got_pack_progress_cb progress_cb, void *progress_arg,
 730 6863cbf9 2022-03-21 stsp     struct got_ratelimit *rl)
 731 e6bcace5 2021-06-22 stsp {
 732 e6bcace5 2021-06-22 stsp 	const struct got_error *err;
 733 67fd6849 2022-02-13 stsp 	struct got_pack_meta *m = NULL;
 734 e6bcace5 2021-06-22 stsp 
 735 05118f5a 2021-06-22 stsp 	if (loose_obj_only) {
 736 05118f5a 2021-06-22 stsp 		int is_packed;
 737 05118f5a 2021-06-22 stsp 		err = search_packidx(&is_packed, id, repo);
 738 05118f5a 2021-06-22 stsp 		if (err)
 739 05118f5a 2021-06-22 stsp 			return err;
 740 cdeb891a 2022-03-21 stsp 		if (is_packed && want_meta)
 741 05118f5a 2021-06-22 stsp 			return NULL;
 742 05118f5a 2021-06-22 stsp 	}
 743 05118f5a 2021-06-22 stsp 
 744 67fd6849 2022-02-13 stsp 	if (want_meta) {
 745 d6a28ffe 2022-05-20 op 		err = alloc_meta(&m, id, path, obj_type, mtime, seed);
 746 6863cbf9 2022-03-21 stsp 		if (err)
 747 6863cbf9 2022-03-21 stsp 			return err;
 748 6863cbf9 2022-03-21 stsp 
 749 6863cbf9 2022-03-21 stsp 		(*nfound)++;
 750 301e83b3 2022-10-16 stsp 		err = got_pack_report_progress(progress_cb, progress_arg, rl,
 751 6863cbf9 2022-03-21 stsp 		    *ncolored, *nfound, *ntrees, 0L, 0, 0, 0, 0);
 752 bb6672b6 2022-04-14 tb 		if (err) {
 753 bb6672b6 2022-04-14 tb 			clear_meta(m);
 754 bb6672b6 2022-04-14 tb 			free(m);
 755 67fd6849 2022-02-13 stsp 			return err;
 756 bb6672b6 2022-04-14 tb 		}
 757 e6bcace5 2021-06-22 stsp 	}
 758 e6bcace5 2021-06-22 stsp 
 759 bb6672b6 2022-04-14 tb 	err = got_object_idset_add(idset, id, m);
 760 bb6672b6 2022-04-14 tb 	if (err) {
 761 bb6672b6 2022-04-14 tb 		clear_meta(m);
 762 bb6672b6 2022-04-14 tb 		free(m);
 763 bb6672b6 2022-04-14 tb 	}
 764 bb6672b6 2022-04-14 tb 	return err;
 765 e6bcace5 2021-06-22 stsp }
 766 e6bcace5 2021-06-22 stsp 
 767 301e83b3 2022-10-16 stsp const struct got_error *
 768 301e83b3 2022-10-16 stsp got_pack_load_tree_entries(struct got_object_id_queue *ids, int want_meta,
 769 2f8438b0 2022-05-04 stsp     struct got_object_idset *idset, struct got_object_idset *idset_exclude,
 770 0ab4c957 2022-06-13 stsp     struct got_tree_object *tree,
 771 d6a28ffe 2022-05-20 op     const char *dpath, time_t mtime, uint32_t seed, struct got_repository *repo,
 772 b8af7c06 2022-03-15 stsp     int loose_obj_only, int *ncolored, int *nfound, int *ntrees,
 773 b8af7c06 2022-03-15 stsp     got_pack_progress_cb progress_cb, void *progress_arg,
 774 b8af7c06 2022-03-15 stsp     struct got_ratelimit *rl, got_cancel_cb cancel_cb, void *cancel_arg)
 775 e6bcace5 2021-06-22 stsp {
 776 e6bcace5 2021-06-22 stsp 	const struct got_error *err;
 777 e6bcace5 2021-06-22 stsp 	char *p = NULL;
 778 e6bcace5 2021-06-22 stsp 	int i;
 779 b8af7c06 2022-03-15 stsp 
 780 b8af7c06 2022-03-15 stsp 	(*ntrees)++;
 781 301e83b3 2022-10-16 stsp 	err = got_pack_report_progress(progress_cb, progress_arg, rl,
 782 b8af7c06 2022-03-15 stsp 	    *ncolored, *nfound, *ntrees, 0L, 0, 0, 0, 0);
 783 e6bcace5 2021-06-22 stsp 	if (err)
 784 e6bcace5 2021-06-22 stsp 		return err;
 785 e6bcace5 2021-06-22 stsp 
 786 e6bcace5 2021-06-22 stsp 	for (i = 0; i < got_object_tree_get_nentries(tree); i++) {
 787 e6bcace5 2021-06-22 stsp 		struct got_tree_entry *e = got_object_tree_get_entry(tree, i);
 788 e6bcace5 2021-06-22 stsp 		struct got_object_id *id = got_tree_entry_get_id(e);
 789 e6bcace5 2021-06-22 stsp 		mode_t mode = got_tree_entry_get_mode(e);
 790 e6bcace5 2021-06-22 stsp 
 791 e6bcace5 2021-06-22 stsp 		if (cancel_cb) {
 792 e6bcace5 2021-06-22 stsp 			err = (*cancel_cb)(cancel_arg);
 793 e6bcace5 2021-06-22 stsp 			if (err)
 794 e6bcace5 2021-06-22 stsp 				break;
 795 e6bcace5 2021-06-22 stsp 		}
 796 e6bcace5 2021-06-22 stsp 
 797 3af9de88 2021-09-22 stsp 		if (got_object_tree_entry_is_submodule(e) ||
 798 2f8438b0 2022-05-04 stsp 		    got_object_idset_contains(idset, id) ||
 799 2f8438b0 2022-05-04 stsp 		    got_object_idset_contains(idset_exclude, id))
 800 cee6a7ea 2022-06-07 stsp 			continue;
 801 0ab4c957 2022-06-13 stsp 
 802 0ab4c957 2022-06-13 stsp 		/*
 803 0ab4c957 2022-06-13 stsp 		 * If got-read-pack is crawling trees for us then
 804 0ab4c957 2022-06-13 stsp 		 * we are only here to collect blob IDs.
 805 0ab4c957 2022-06-13 stsp 		 */
 806 0ab4c957 2022-06-13 stsp 		if (ids == NULL && S_ISDIR(mode))
 807 0ab4c957 2022-06-13 stsp 			continue;
 808 0ab4c957 2022-06-13 stsp 
 809 0ab4c957 2022-06-13 stsp 		if (asprintf(&p, "%s%s%s", dpath,
 810 0ab4c957 2022-06-13 stsp 		    got_path_is_root_dir(dpath) ? "" : "/",
 811 e6bcace5 2021-06-22 stsp 		    got_tree_entry_get_name(e)) == -1) {
 812 e6bcace5 2021-06-22 stsp 			err = got_error_from_errno("asprintf");
 813 e6bcace5 2021-06-22 stsp 			break;
 814 e6bcace5 2021-06-22 stsp 		}
 815 e6bcace5 2021-06-22 stsp 
 816 e6bcace5 2021-06-22 stsp 		if (S_ISDIR(mode)) {
 817 e6bcace5 2021-06-22 stsp 			struct got_object_qid *qid;
 818 e6bcace5 2021-06-22 stsp 			err = got_object_qid_alloc(&qid, id);
 819 e6bcace5 2021-06-22 stsp 			if (err)
 820 e6bcace5 2021-06-22 stsp 				break;
 821 3e6ceea0 2022-05-20 stsp 			qid->data = p;
 822 3e6ceea0 2022-05-20 stsp 			p = NULL;
 823 dbdddfee 2021-06-23 naddy 			STAILQ_INSERT_TAIL(ids, qid, entry);
 824 3af9de88 2021-09-22 stsp 		} else if (S_ISREG(mode) || S_ISLNK(mode)) {
 825 301e83b3 2022-10-16 stsp 			err = got_pack_add_object(want_meta,
 826 2f8438b0 2022-05-04 stsp 			    want_meta ? idset : idset_exclude, id, p,
 827 d6a28ffe 2022-05-20 op 			    GOT_OBJ_TYPE_BLOB, mtime, seed, loose_obj_only,
 828 d6a28ffe 2022-05-20 op 			    repo, ncolored, nfound, ntrees,
 829 6863cbf9 2022-03-21 stsp 			    progress_cb, progress_arg, rl);
 830 b8af7c06 2022-03-15 stsp 			if (err)
 831 b8af7c06 2022-03-15 stsp 				break;
 832 3e6ceea0 2022-05-20 stsp 			free(p);
 833 3e6ceea0 2022-05-20 stsp 			p = NULL;
 834 3e6ceea0 2022-05-20 stsp 		} else {
 835 3e6ceea0 2022-05-20 stsp 			free(p);
 836 3e6ceea0 2022-05-20 stsp 			p = NULL;
 837 e6bcace5 2021-06-22 stsp 		}
 838 e6bcace5 2021-06-22 stsp 	}
 839 e6bcace5 2021-06-22 stsp 
 840 e6bcace5 2021-06-22 stsp 	free(p);
 841 e6bcace5 2021-06-22 stsp 	return err;
 842 e6bcace5 2021-06-22 stsp }
 843 e6bcace5 2021-06-22 stsp 
 844 301e83b3 2022-10-16 stsp const struct got_error *
 845 301e83b3 2022-10-16 stsp got_pack_load_tree(int want_meta, struct got_object_idset *idset,
 846 2f8438b0 2022-05-04 stsp     struct got_object_idset *idset_exclude,
 847 e6bcace5 2021-06-22 stsp     struct got_object_id *tree_id, const char *dpath, time_t mtime,
 848 d6a28ffe 2022-05-20 op     uint32_t seed, struct got_repository *repo, int loose_obj_only,
 849 b8af7c06 2022-03-15 stsp     int *ncolored, int *nfound, int *ntrees,
 850 b8af7c06 2022-03-15 stsp     got_pack_progress_cb progress_cb, void *progress_arg,
 851 b8af7c06 2022-03-15 stsp     struct got_ratelimit *rl, got_cancel_cb cancel_cb, void *cancel_arg)
 852 e6bcace5 2021-06-22 stsp {
 853 e6bcace5 2021-06-22 stsp 	const struct got_error *err = NULL;
 854 e6bcace5 2021-06-22 stsp 	struct got_object_id_queue tree_ids;
 855 e6bcace5 2021-06-22 stsp 	struct got_object_qid *qid;
 856 0ab4c957 2022-06-13 stsp 	struct got_tree_object *tree = NULL;
 857 e6bcace5 2021-06-22 stsp 
 858 2f8438b0 2022-05-04 stsp 	if (got_object_idset_contains(idset, tree_id) ||
 859 2f8438b0 2022-05-04 stsp 	    got_object_idset_contains(idset_exclude, tree_id))
 860 e6bcace5 2021-06-22 stsp 		return NULL;
 861 e6bcace5 2021-06-22 stsp 
 862 e6bcace5 2021-06-22 stsp 	err = got_object_qid_alloc(&qid, tree_id);
 863 e6bcace5 2021-06-22 stsp 	if (err)
 864 3e6ceea0 2022-05-20 stsp 		return err;
 865 3e6ceea0 2022-05-20 stsp 	qid->data = strdup(dpath);
 866 3e6ceea0 2022-05-20 stsp 	if (qid->data == NULL) {
 867 3e6ceea0 2022-05-20 stsp 		err = got_error_from_errno("strdup");
 868 3e6ceea0 2022-05-20 stsp 		got_object_qid_free(qid);
 869 e6bcace5 2021-06-22 stsp 		return err;
 870 3e6ceea0 2022-05-20 stsp 	}
 871 e6bcace5 2021-06-22 stsp 
 872 dbdddfee 2021-06-23 naddy 	STAILQ_INIT(&tree_ids);
 873 dbdddfee 2021-06-23 naddy 	STAILQ_INSERT_TAIL(&tree_ids, qid, entry);
 874 e6bcace5 2021-06-22 stsp 
 875 dbdddfee 2021-06-23 naddy 	while (!STAILQ_EMPTY(&tree_ids)) {
 876 3e6ceea0 2022-05-20 stsp 		const char *path;
 877 e6bcace5 2021-06-22 stsp 		if (cancel_cb) {
 878 e6bcace5 2021-06-22 stsp 			err = (*cancel_cb)(cancel_arg);
 879 e6bcace5 2021-06-22 stsp 			if (err)
 880 e6bcace5 2021-06-22 stsp 				break;
 881 e6bcace5 2021-06-22 stsp 		}
 882 e6bcace5 2021-06-22 stsp 
 883 dbdddfee 2021-06-23 naddy 		qid = STAILQ_FIRST(&tree_ids);
 884 dbdddfee 2021-06-23 naddy 		STAILQ_REMOVE_HEAD(&tree_ids, entry);
 885 3e6ceea0 2022-05-20 stsp 		path = qid->data;
 886 e6bcace5 2021-06-22 stsp 
 887 2f8438b0 2022-05-04 stsp 		if (got_object_idset_contains(idset, &qid->id) ||
 888 2f8438b0 2022-05-04 stsp 		    got_object_idset_contains(idset_exclude, &qid->id)) {
 889 3e6ceea0 2022-05-20 stsp 			free(qid->data);
 890 e6bcace5 2021-06-22 stsp 			got_object_qid_free(qid);
 891 e6bcace5 2021-06-22 stsp 			continue;
 892 e6bcace5 2021-06-22 stsp 		}
 893 e6bcace5 2021-06-22 stsp 
 894 301e83b3 2022-10-16 stsp 		err = got_pack_add_object(want_meta,
 895 301e83b3 2022-10-16 stsp 		    want_meta ? idset : idset_exclude,
 896 3e6ceea0 2022-05-20 stsp 		    &qid->id, path, GOT_OBJ_TYPE_TREE,
 897 d6a28ffe 2022-05-20 op 		    mtime, seed, loose_obj_only, repo,
 898 6863cbf9 2022-03-21 stsp 		    ncolored, nfound, ntrees, progress_cb, progress_arg, rl);
 899 e6bcace5 2021-06-22 stsp 		if (err) {
 900 3e6ceea0 2022-05-20 stsp 			free(qid->data);
 901 e6bcace5 2021-06-22 stsp 			got_object_qid_free(qid);
 902 e6bcace5 2021-06-22 stsp 			break;
 903 e6bcace5 2021-06-22 stsp 		}
 904 e6bcace5 2021-06-22 stsp 
 905 0ab4c957 2022-06-13 stsp 		err = got_object_open_as_tree(&tree, repo, &qid->id);
 906 0ab4c957 2022-06-13 stsp 		if (err) {
 907 0ab4c957 2022-06-13 stsp 			free(qid->data);
 908 0ab4c957 2022-06-13 stsp 			got_object_qid_free(qid);
 909 0ab4c957 2022-06-13 stsp 			break;
 910 0ab4c957 2022-06-13 stsp 		}
 911 0ab4c957 2022-06-13 stsp 
 912 301e83b3 2022-10-16 stsp 		err = got_pack_load_tree_entries(&tree_ids, want_meta, idset,
 913 0ab4c957 2022-06-13 stsp 		    idset_exclude, tree, path, mtime, seed, repo,
 914 0ab4c957 2022-06-13 stsp 		    loose_obj_only, ncolored, nfound, ntrees,
 915 0ab4c957 2022-06-13 stsp 		    progress_cb, progress_arg, rl,
 916 b8af7c06 2022-03-15 stsp 		    cancel_cb, cancel_arg);
 917 3e6ceea0 2022-05-20 stsp 		free(qid->data);
 918 e6bcace5 2021-06-22 stsp 		got_object_qid_free(qid);
 919 e6bcace5 2021-06-22 stsp 		if (err)
 920 e6bcace5 2021-06-22 stsp 			break;
 921 0ab4c957 2022-06-13 stsp 
 922 0ab4c957 2022-06-13 stsp 		got_object_tree_close(tree);
 923 0ab4c957 2022-06-13 stsp 		tree = NULL;
 924 e6bcace5 2021-06-22 stsp 	}
 925 e6bcace5 2021-06-22 stsp 
 926 3e6ceea0 2022-05-20 stsp 	STAILQ_FOREACH(qid, &tree_ids, entry)
 927 3e6ceea0 2022-05-20 stsp 		free(qid->data);
 928 e6bcace5 2021-06-22 stsp 	got_object_id_queue_free(&tree_ids);
 929 0ab4c957 2022-06-13 stsp 	if (tree)
 930 0ab4c957 2022-06-13 stsp 		got_object_tree_close(tree);
 931 e6bcace5 2021-06-22 stsp 	return err;
 932 e6bcace5 2021-06-22 stsp }
 933 e6bcace5 2021-06-22 stsp 
 934 e6bcace5 2021-06-22 stsp static const struct got_error *
 935 67fd6849 2022-02-13 stsp load_commit(int want_meta, struct got_object_idset *idset,
 936 2f8438b0 2022-05-04 stsp     struct got_object_idset *idset_exclude,
 937 d6a28ffe 2022-05-20 op     struct got_object_id *id, struct got_repository *repo, uint32_t seed,
 938 d6a28ffe 2022-05-20 op     int loose_obj_only, int *ncolored, int *nfound, int *ntrees,
 939 b8af7c06 2022-03-15 stsp     got_pack_progress_cb progress_cb, void *progress_arg,
 940 b8af7c06 2022-03-15 stsp     struct got_ratelimit *rl, got_cancel_cb cancel_cb, void *cancel_arg)
 941 e6bcace5 2021-06-22 stsp {
 942 e6bcace5 2021-06-22 stsp 	const struct got_error *err;
 943 e6bcace5 2021-06-22 stsp 	struct got_commit_object *commit;
 944 e6bcace5 2021-06-22 stsp 
 945 2f8438b0 2022-05-04 stsp 	if (got_object_idset_contains(idset, id) ||
 946 2f8438b0 2022-05-04 stsp 	    got_object_idset_contains(idset_exclude, id))
 947 e6bcace5 2021-06-22 stsp 		return NULL;
 948 05118f5a 2021-06-22 stsp 
 949 05118f5a 2021-06-22 stsp 	if (loose_obj_only) {
 950 05118f5a 2021-06-22 stsp 		int is_packed;
 951 05118f5a 2021-06-22 stsp 		err = search_packidx(&is_packed, id, repo);
 952 05118f5a 2021-06-22 stsp 		if (err)
 953 05118f5a 2021-06-22 stsp 			return err;
 954 cdeb891a 2022-03-21 stsp 		if (is_packed && want_meta)
 955 05118f5a 2021-06-22 stsp 			return NULL;
 956 05118f5a 2021-06-22 stsp 	}
 957 e6bcace5 2021-06-22 stsp 
 958 e6bcace5 2021-06-22 stsp 	err = got_object_open_as_commit(&commit, repo, id);
 959 e6bcace5 2021-06-22 stsp 	if (err)
 960 e6bcace5 2021-06-22 stsp 		return err;
 961 e6bcace5 2021-06-22 stsp 
 962 301e83b3 2022-10-16 stsp 	err = got_pack_add_object(want_meta,
 963 301e83b3 2022-10-16 stsp 	    want_meta ? idset : idset_exclude, id, "", GOT_OBJ_TYPE_COMMIT,
 964 d6a28ffe 2022-05-20 op 	    got_object_commit_get_committer_time(commit), seed,
 965 6863cbf9 2022-03-21 stsp 	    loose_obj_only, repo,
 966 6863cbf9 2022-03-21 stsp 	    ncolored, nfound, ntrees, progress_cb, progress_arg, rl);
 967 e6bcace5 2021-06-22 stsp 	if (err)
 968 e6bcace5 2021-06-22 stsp 		goto done;
 969 b8af7c06 2022-03-15 stsp 
 970 301e83b3 2022-10-16 stsp 	err = got_pack_load_tree(want_meta, idset, idset_exclude,
 971 2f8438b0 2022-05-04 stsp 	    got_object_commit_get_tree_id(commit),
 972 d6a28ffe 2022-05-20 op 	    "", got_object_commit_get_committer_time(commit), seed,
 973 b8af7c06 2022-03-15 stsp 	    repo, loose_obj_only, ncolored, nfound, ntrees,
 974 b8af7c06 2022-03-15 stsp 	    progress_cb, progress_arg, rl, cancel_cb, cancel_arg);
 975 e6bcace5 2021-06-22 stsp done:
 976 e6bcace5 2021-06-22 stsp 	got_object_commit_close(commit);
 977 e6bcace5 2021-06-22 stsp 	return err;
 978 e6bcace5 2021-06-22 stsp }
 979 e6bcace5 2021-06-22 stsp 
 980 05118f5a 2021-06-22 stsp static const struct got_error *
 981 67fd6849 2022-02-13 stsp load_tag(int want_meta, struct got_object_idset *idset,
 982 2f8438b0 2022-05-04 stsp     struct got_object_idset *idset_exclude,
 983 d6a28ffe 2022-05-20 op     struct got_object_id *id, struct got_repository *repo, uint32_t seed,
 984 d6a28ffe 2022-05-20 op     int loose_obj_only, int *ncolored, int *nfound, int *ntrees,
 985 b8af7c06 2022-03-15 stsp     got_pack_progress_cb progress_cb, void *progress_arg,
 986 b8af7c06 2022-03-15 stsp     struct got_ratelimit *rl, got_cancel_cb cancel_cb, void *cancel_arg)
 987 05118f5a 2021-06-22 stsp {
 988 05118f5a 2021-06-22 stsp 	const struct got_error *err;
 989 05118f5a 2021-06-22 stsp 	struct got_tag_object *tag = NULL;
 990 05118f5a 2021-06-22 stsp 
 991 2f8438b0 2022-05-04 stsp 	if (got_object_idset_contains(idset, id) ||
 992 2f8438b0 2022-05-04 stsp 	    got_object_idset_contains(idset_exclude, id))
 993 05118f5a 2021-06-22 stsp 		return NULL;
 994 05118f5a 2021-06-22 stsp 
 995 05118f5a 2021-06-22 stsp 	if (loose_obj_only) {
 996 05118f5a 2021-06-22 stsp 		int is_packed;
 997 05118f5a 2021-06-22 stsp 		err = search_packidx(&is_packed, id, repo);
 998 05118f5a 2021-06-22 stsp 		if (err)
 999 05118f5a 2021-06-22 stsp 			return err;
1000 cdeb891a 2022-03-21 stsp 		if (is_packed && want_meta)
1001 05118f5a 2021-06-22 stsp 			return NULL;
1002 05118f5a 2021-06-22 stsp 	}
1003 05118f5a 2021-06-22 stsp 
1004 05118f5a 2021-06-22 stsp 	err = got_object_open_as_tag(&tag, repo, id);
1005 05118f5a 2021-06-22 stsp 	if (err)
1006 05118f5a 2021-06-22 stsp 		return err;
1007 05118f5a 2021-06-22 stsp 
1008 301e83b3 2022-10-16 stsp 	err = got_pack_add_object(want_meta,
1009 301e83b3 2022-10-16 stsp 	    want_meta ? idset : idset_exclude, id, "", GOT_OBJ_TYPE_TAG,
1010 d6a28ffe 2022-05-20 op 	    got_object_tag_get_tagger_time(tag), seed, loose_obj_only, repo,
1011 6863cbf9 2022-03-21 stsp 	    ncolored, nfound, ntrees, progress_cb, progress_arg, rl);
1012 05118f5a 2021-06-22 stsp 	if (err)
1013 05118f5a 2021-06-22 stsp 		goto done;
1014 05118f5a 2021-06-22 stsp 
1015 05118f5a 2021-06-22 stsp 	switch (got_object_tag_get_object_type(tag)) {
1016 05118f5a 2021-06-22 stsp 	case GOT_OBJ_TYPE_COMMIT:
1017 2f8438b0 2022-05-04 stsp 		err = load_commit(want_meta, idset, idset_exclude,
1018 d6a28ffe 2022-05-20 op 		    got_object_tag_get_object_id(tag), repo, seed,
1019 d6a28ffe 2022-05-20 op 		    loose_obj_only, ncolored, nfound, ntrees,
1020 d6a28ffe 2022-05-20 op 		    progress_cb, progress_arg, rl, cancel_cb, cancel_arg);
1021 05118f5a 2021-06-22 stsp 		break;
1022 05118f5a 2021-06-22 stsp 	case GOT_OBJ_TYPE_TREE:
1023 301e83b3 2022-10-16 stsp 		err = got_pack_load_tree(want_meta, idset, idset_exclude,
1024 67fd6849 2022-02-13 stsp 		    got_object_tag_get_object_id(tag), "",
1025 d6a28ffe 2022-05-20 op 		    got_object_tag_get_tagger_time(tag), seed, repo,
1026 d6a28ffe 2022-05-20 op 		    loose_obj_only, ncolored, nfound, ntrees,
1027 d6a28ffe 2022-05-20 op 		    progress_cb, progress_arg, rl, cancel_cb, cancel_arg);
1028 05118f5a 2021-06-22 stsp 		break;
1029 05118f5a 2021-06-22 stsp 	default:
1030 05118f5a 2021-06-22 stsp 		break;
1031 05118f5a 2021-06-22 stsp 	}
1032 05118f5a 2021-06-22 stsp 
1033 05118f5a 2021-06-22 stsp done:
1034 05118f5a 2021-06-22 stsp 	got_object_tag_close(tag);
1035 05118f5a 2021-06-22 stsp 	return err;
1036 05118f5a 2021-06-22 stsp }
1037 05118f5a 2021-06-22 stsp 
1038 301e83b3 2022-10-16 stsp const struct got_error *
1039 301e83b3 2022-10-16 stsp got_pack_paint_commit(struct got_object_qid *qid, intptr_t color)
1040 e6bcace5 2021-06-22 stsp {
1041 61af9b21 2022-06-28 stsp 	if (color < 0 || color >= COLOR_MAX)
1042 70f8f24d 2022-04-14 stsp 		return got_error(GOT_ERR_RANGE);
1043 e6bcace5 2021-06-22 stsp 
1044 61af9b21 2022-06-28 stsp 	qid->data = (void *)color;
1045 e6bcace5 2021-06-22 stsp 	return NULL;
1046 e6bcace5 2021-06-22 stsp }
1047 e6bcace5 2021-06-22 stsp 
1048 301e83b3 2022-10-16 stsp const struct got_error *
1049 301e83b3 2022-10-16 stsp got_pack_queue_commit_id(struct got_object_id_queue *ids,
1050 301e83b3 2022-10-16 stsp     struct got_object_id *id, intptr_t color, struct got_repository *repo)
1051 e6bcace5 2021-06-22 stsp {
1052 70f8f24d 2022-04-14 stsp 	const struct got_error *err;
1053 e6bcace5 2021-06-22 stsp 	struct got_object_qid *qid;
1054 e6bcace5 2021-06-22 stsp 
1055 e6bcace5 2021-06-22 stsp 	err = got_object_qid_alloc(&qid, id);
1056 e6bcace5 2021-06-22 stsp 	if (err)
1057 e6bcace5 2021-06-22 stsp 		return err;
1058 e6bcace5 2021-06-22 stsp 
1059 70f8f24d 2022-04-14 stsp 	STAILQ_INSERT_TAIL(ids, qid, entry);
1060 301e83b3 2022-10-16 stsp 	return got_pack_paint_commit(qid, color);
1061 e6bcace5 2021-06-22 stsp }
1062 e6bcace5 2021-06-22 stsp 
1063 e6bcace5 2021-06-22 stsp struct append_id_arg {
1064 e6bcace5 2021-06-22 stsp 	struct got_object_id **array;
1065 e6bcace5 2021-06-22 stsp 	int idx;
1066 70f8f24d 2022-04-14 stsp 	struct got_object_idset *drop;
1067 70f8f24d 2022-04-14 stsp 	struct got_object_idset *skip;
1068 e6bcace5 2021-06-22 stsp };
1069 e6bcace5 2021-06-22 stsp 
1070 e6bcace5 2021-06-22 stsp static const struct got_error *
1071 e6bcace5 2021-06-22 stsp append_id(struct got_object_id *id, void *data, void *arg)
1072 e6bcace5 2021-06-22 stsp {
1073 e6bcace5 2021-06-22 stsp 	struct append_id_arg *a = arg;
1074 e6bcace5 2021-06-22 stsp 
1075 70f8f24d 2022-04-14 stsp 	if (got_object_idset_contains(a->skip, id) ||
1076 70f8f24d 2022-04-14 stsp 	    got_object_idset_contains(a->drop, id))
1077 70f8f24d 2022-04-14 stsp 		return NULL;
1078 70f8f24d 2022-04-14 stsp 
1079 70f8f24d 2022-04-14 stsp 	a->array[++a->idx] = got_object_id_dup(id);
1080 e6bcace5 2021-06-22 stsp 	if (a->array[a->idx] == NULL)
1081 e6bcace5 2021-06-22 stsp 		return got_error_from_errno("got_object_id_dup");
1082 e6bcace5 2021-06-22 stsp 
1083 e6bcace5 2021-06-22 stsp 	return NULL;
1084 29e0594f 2022-04-09 stsp }
1085 29e0594f 2022-04-09 stsp 
1086 29e0594f 2022-04-09 stsp static const struct got_error *
1087 61af9b21 2022-06-28 stsp queue_commit_or_tag_id(struct got_object_id *id, intptr_t color,
1088 29e0594f 2022-04-09 stsp     struct got_object_id_queue *ids, struct got_repository *repo)
1089 29e0594f 2022-04-09 stsp {
1090 29e0594f 2022-04-09 stsp 	const struct got_error *err;
1091 29e0594f 2022-04-09 stsp 	struct got_tag_object *tag = NULL;
1092 29e0594f 2022-04-09 stsp 	int obj_type;
1093 29e0594f 2022-04-09 stsp 
1094 29e0594f 2022-04-09 stsp 	err = got_object_get_type(&obj_type, repo, id);
1095 29e0594f 2022-04-09 stsp 	if (err)
1096 29e0594f 2022-04-09 stsp 		return err;
1097 29e0594f 2022-04-09 stsp 
1098 29e0594f 2022-04-09 stsp 	if (obj_type == GOT_OBJ_TYPE_TAG) {
1099 29e0594f 2022-04-09 stsp 		err = got_object_open_as_tag(&tag, repo, id);
1100 29e0594f 2022-04-09 stsp 		if (err)
1101 29e0594f 2022-04-09 stsp 			return err;
1102 29e0594f 2022-04-09 stsp 		obj_type = got_object_tag_get_object_type(tag);
1103 29e0594f 2022-04-09 stsp 		id = got_object_tag_get_object_id(tag);
1104 29e0594f 2022-04-09 stsp 	}
1105 29e0594f 2022-04-09 stsp 
1106 29e0594f 2022-04-09 stsp 	if (obj_type == GOT_OBJ_TYPE_COMMIT) {
1107 301e83b3 2022-10-16 stsp 		err = got_pack_queue_commit_id(ids, id, color, repo);
1108 29e0594f 2022-04-09 stsp 		if (err)
1109 29e0594f 2022-04-09 stsp 			goto done;
1110 29e0594f 2022-04-09 stsp 	}
1111 29e0594f 2022-04-09 stsp done:
1112 29e0594f 2022-04-09 stsp 	if (tag)
1113 29e0594f 2022-04-09 stsp 		got_object_tag_close(tag);
1114 61af9b21 2022-06-28 stsp 	return err;
1115 61af9b21 2022-06-28 stsp }
1116 61af9b21 2022-06-28 stsp 
1117 301e83b3 2022-10-16 stsp const struct got_error *
1118 301e83b3 2022-10-16 stsp got_pack_find_pack_for_commit_painting(struct got_packidx **best_packidx,
1119 61af9b21 2022-06-28 stsp     struct got_object_id_queue *ids, int nids, struct got_repository *repo)
1120 61af9b21 2022-06-28 stsp {
1121 61af9b21 2022-06-28 stsp 	const struct got_error *err = NULL;
1122 61af9b21 2022-06-28 stsp 	struct got_pathlist_entry *pe;
1123 61af9b21 2022-06-28 stsp 	const char *best_packidx_path = NULL;
1124 61af9b21 2022-06-28 stsp 	int nobj_max = 0;
1125 61af9b21 2022-06-28 stsp 	int ncommits_max = 0;
1126 61af9b21 2022-06-28 stsp 
1127 61af9b21 2022-06-28 stsp 	*best_packidx = NULL;
1128 61af9b21 2022-06-28 stsp 
1129 61af9b21 2022-06-28 stsp 	/*
1130 61af9b21 2022-06-28 stsp 	 * Find the largest pack which contains at least some of the
1131 61af9b21 2022-06-28 stsp 	 * commits we are interested in.
1132 61af9b21 2022-06-28 stsp 	 */
1133 61af9b21 2022-06-28 stsp 	TAILQ_FOREACH(pe, &repo->packidx_paths, entry) {
1134 61af9b21 2022-06-28 stsp 		const char *path_packidx = pe->path;
1135 61af9b21 2022-06-28 stsp 		struct got_packidx *packidx;
1136 61af9b21 2022-06-28 stsp 		int nobj, idx, ncommits = 0;
1137 61af9b21 2022-06-28 stsp 		struct got_object_qid *qid;
1138 61af9b21 2022-06-28 stsp 
1139 61af9b21 2022-06-28 stsp 		err = got_repo_get_packidx(&packidx, path_packidx, repo);
1140 61af9b21 2022-06-28 stsp 		if (err)
1141 61af9b21 2022-06-28 stsp 			break;
1142 61af9b21 2022-06-28 stsp 
1143 61af9b21 2022-06-28 stsp 		nobj = be32toh(packidx->hdr.fanout_table[0xff]);
1144 61af9b21 2022-06-28 stsp 		if (nobj <= nobj_max)
1145 61af9b21 2022-06-28 stsp 			continue;
1146 61af9b21 2022-06-28 stsp 
1147 61af9b21 2022-06-28 stsp 		STAILQ_FOREACH(qid, ids, entry) {
1148 61af9b21 2022-06-28 stsp 			idx = got_packidx_get_object_idx(packidx, &qid->id);
1149 61af9b21 2022-06-28 stsp 			if (idx != -1)
1150 61af9b21 2022-06-28 stsp 				ncommits++;
1151 61af9b21 2022-06-28 stsp 		}
1152 61af9b21 2022-06-28 stsp 		if (ncommits > ncommits_max) {
1153 61af9b21 2022-06-28 stsp 			best_packidx_path = path_packidx;
1154 61af9b21 2022-06-28 stsp 			nobj_max = nobj;
1155 61af9b21 2022-06-28 stsp 			ncommits_max = ncommits;
1156 61af9b21 2022-06-28 stsp 		}
1157 61af9b21 2022-06-28 stsp 	}
1158 61af9b21 2022-06-28 stsp 
1159 61af9b21 2022-06-28 stsp 	if (best_packidx_path && err == NULL) {
1160 61af9b21 2022-06-28 stsp 		err = got_repo_get_packidx(best_packidx, best_packidx_path,
1161 61af9b21 2022-06-28 stsp 		    repo);
1162 61af9b21 2022-06-28 stsp 	}
1163 61af9b21 2022-06-28 stsp 
1164 61af9b21 2022-06-28 stsp 	return err;
1165 61af9b21 2022-06-28 stsp }
1166 61af9b21 2022-06-28 stsp 
1167 61af9b21 2022-06-28 stsp static const struct got_error *
1168 14dbbf48 2022-04-10 stsp findtwixt(struct got_object_id ***res, int *nres, int *ncolored,
1169 14dbbf48 2022-04-10 stsp     struct got_object_id **head, int nhead,
1170 14dbbf48 2022-04-10 stsp     struct got_object_id **tail, int ntail,
1171 14dbbf48 2022-04-10 stsp     struct got_repository *repo,
1172 14dbbf48 2022-04-10 stsp     got_pack_progress_cb progress_cb, void *progress_arg,
1173 14dbbf48 2022-04-10 stsp     struct got_ratelimit *rl, got_cancel_cb cancel_cb, void *cancel_arg)
1174 14dbbf48 2022-04-10 stsp {
1175 14dbbf48 2022-04-10 stsp 	const struct got_error *err = NULL;
1176 14dbbf48 2022-04-10 stsp 	struct got_object_id_queue ids;
1177 70f8f24d 2022-04-14 stsp 	struct got_object_idset *keep, *drop, *skip = NULL;
1178 14dbbf48 2022-04-10 stsp 	int i, nkeep;
1179 14dbbf48 2022-04-10 stsp 
1180 14dbbf48 2022-04-10 stsp 	STAILQ_INIT(&ids);
1181 14dbbf48 2022-04-10 stsp 	*res = NULL;
1182 14dbbf48 2022-04-10 stsp 	*nres = 0;
1183 14dbbf48 2022-04-10 stsp 	*ncolored = 0;
1184 14dbbf48 2022-04-10 stsp 
1185 14dbbf48 2022-04-10 stsp 	keep = got_object_idset_alloc();
1186 14dbbf48 2022-04-10 stsp 	if (keep == NULL)
1187 14dbbf48 2022-04-10 stsp 		return got_error_from_errno("got_object_idset_alloc");
1188 14dbbf48 2022-04-10 stsp 
1189 14dbbf48 2022-04-10 stsp 	drop = got_object_idset_alloc();
1190 14dbbf48 2022-04-10 stsp 	if (drop == NULL) {
1191 14dbbf48 2022-04-10 stsp 		err = got_error_from_errno("got_object_idset_alloc");
1192 14dbbf48 2022-04-10 stsp 		goto done;
1193 14dbbf48 2022-04-10 stsp 	}
1194 14dbbf48 2022-04-10 stsp 
1195 70f8f24d 2022-04-14 stsp 	skip = got_object_idset_alloc();
1196 70f8f24d 2022-04-14 stsp 	if (skip == NULL) {
1197 70f8f24d 2022-04-14 stsp 		err = got_error_from_errno("got_object_idset_alloc");
1198 70f8f24d 2022-04-14 stsp 		goto done;
1199 70f8f24d 2022-04-14 stsp 	}
1200 70f8f24d 2022-04-14 stsp 
1201 14dbbf48 2022-04-10 stsp 	for (i = 0; i < nhead; i++) {
1202 14dbbf48 2022-04-10 stsp 		struct got_object_id *id = head[i];
1203 14dbbf48 2022-04-10 stsp 		if (id == NULL)
1204 14dbbf48 2022-04-10 stsp 			continue;
1205 fbafdecf 2022-04-10 stsp 		err = queue_commit_or_tag_id(id, COLOR_KEEP, &ids, repo);
1206 14dbbf48 2022-04-10 stsp 		if (err)
1207 14dbbf48 2022-04-10 stsp 			goto done;
1208 5e91dae4 2022-08-30 stsp 	}
1209 14dbbf48 2022-04-10 stsp 
1210 14dbbf48 2022-04-10 stsp 	for (i = 0; i < ntail; i++) {
1211 14dbbf48 2022-04-10 stsp 		struct got_object_id *id = tail[i];
1212 14dbbf48 2022-04-10 stsp 		if (id == NULL)
1213 14dbbf48 2022-04-10 stsp 			continue;
1214 14dbbf48 2022-04-10 stsp 		err = queue_commit_or_tag_id(id, COLOR_DROP, &ids, repo);
1215 14dbbf48 2022-04-10 stsp 		if (err)
1216 14dbbf48 2022-04-10 stsp 			goto done;
1217 14dbbf48 2022-04-10 stsp 	}
1218 14dbbf48 2022-04-10 stsp 
1219 301e83b3 2022-10-16 stsp 	err = got_pack_paint_commits(ncolored, &ids, nhead + ntail,
1220 70f8f24d 2022-04-14 stsp 	    keep, drop, skip, repo, progress_cb, progress_arg, rl,
1221 70f8f24d 2022-04-14 stsp 	    cancel_cb, cancel_arg);
1222 14dbbf48 2022-04-10 stsp 	if (err)
1223 14dbbf48 2022-04-10 stsp 		goto done;
1224 14dbbf48 2022-04-10 stsp 
1225 e6bcace5 2021-06-22 stsp 	nkeep = got_object_idset_num_elements(keep);
1226 e6bcace5 2021-06-22 stsp 	if (nkeep > 0) {
1227 e6bcace5 2021-06-22 stsp 		struct append_id_arg arg;
1228 e6bcace5 2021-06-22 stsp 		arg.array = calloc(nkeep, sizeof(struct got_object_id *));
1229 e6bcace5 2021-06-22 stsp 		if (arg.array == NULL) {
1230 e6bcace5 2021-06-22 stsp 			err = got_error_from_errno("calloc");
1231 e6bcace5 2021-06-22 stsp 			goto done;
1232 e6bcace5 2021-06-22 stsp 		}
1233 70f8f24d 2022-04-14 stsp 		arg.idx = -1;
1234 70f8f24d 2022-04-14 stsp 		arg.skip = skip;
1235 70f8f24d 2022-04-14 stsp 		arg.drop = drop;
1236 e6bcace5 2021-06-22 stsp 		err = got_object_idset_for_each(keep, append_id, &arg);
1237 e6bcace5 2021-06-22 stsp 		if (err) {
1238 e6bcace5 2021-06-22 stsp 			free(arg.array);
1239 e6bcace5 2021-06-22 stsp 			goto done;
1240 e6bcace5 2021-06-22 stsp 		}
1241 e6bcace5 2021-06-22 stsp 		*res = arg.array;
1242 70f8f24d 2022-04-14 stsp 		*nres = arg.idx + 1;
1243 e6bcace5 2021-06-22 stsp 	}
1244 e6bcace5 2021-06-22 stsp done:
1245 e6bcace5 2021-06-22 stsp 	got_object_idset_free(keep);
1246 e6bcace5 2021-06-22 stsp 	got_object_idset_free(drop);
1247 70f8f24d 2022-04-14 stsp 	if (skip)
1248 70f8f24d 2022-04-14 stsp 		got_object_idset_free(skip);
1249 e6bcace5 2021-06-22 stsp 	got_object_id_queue_free(&ids);
1250 e6bcace5 2021-06-22 stsp 	return err;
1251 e6bcace5 2021-06-22 stsp }
1252 e6bcace5 2021-06-22 stsp 
1253 e6bcace5 2021-06-22 stsp static const struct got_error *
1254 0ab4c957 2022-06-13 stsp find_pack_for_enumeration(struct got_packidx **best_packidx,
1255 0ab4c957 2022-06-13 stsp     struct got_object_id **ids, int nids, struct got_repository *repo)
1256 0ab4c957 2022-06-13 stsp {
1257 0ab4c957 2022-06-13 stsp 	const struct got_error *err = NULL;
1258 0ab4c957 2022-06-13 stsp 	struct got_pathlist_entry *pe;
1259 0ab4c957 2022-06-13 stsp 	const char *best_packidx_path = NULL;
1260 0ab4c957 2022-06-13 stsp 	int nobj_max = 0;
1261 0ab4c957 2022-06-13 stsp 	int ncommits_max = 0;
1262 0ab4c957 2022-06-13 stsp 
1263 0ab4c957 2022-06-13 stsp 	*best_packidx = NULL;
1264 0ab4c957 2022-06-13 stsp 
1265 0ab4c957 2022-06-13 stsp 	/*
1266 0ab4c957 2022-06-13 stsp 	 * Find the largest pack which contains at least some of the
1267 0ab4c957 2022-06-13 stsp 	 * commits and tags we are interested in.
1268 0ab4c957 2022-06-13 stsp 	 */
1269 0ab4c957 2022-06-13 stsp 	TAILQ_FOREACH(pe, &repo->packidx_paths, entry) {
1270 0ab4c957 2022-06-13 stsp 		const char *path_packidx = pe->path;
1271 0ab4c957 2022-06-13 stsp 		struct got_packidx *packidx;
1272 0ab4c957 2022-06-13 stsp 		int nobj, i, idx, ncommits = 0;
1273 0ab4c957 2022-06-13 stsp 
1274 0ab4c957 2022-06-13 stsp 		err = got_repo_get_packidx(&packidx, path_packidx, repo);
1275 0ab4c957 2022-06-13 stsp 		if (err)
1276 0ab4c957 2022-06-13 stsp 			break;
1277 0ab4c957 2022-06-13 stsp 
1278 0ab4c957 2022-06-13 stsp 		nobj = be32toh(packidx->hdr.fanout_table[0xff]);
1279 0ab4c957 2022-06-13 stsp 		if (nobj <= nobj_max)
1280 0ab4c957 2022-06-13 stsp 			continue;
1281 0ab4c957 2022-06-13 stsp 
1282 0ab4c957 2022-06-13 stsp 		for (i = 0; i < nids; i++) {
1283 0ab4c957 2022-06-13 stsp 			idx = got_packidx_get_object_idx(packidx, ids[i]);
1284 0ab4c957 2022-06-13 stsp 			if (idx != -1)
1285 0ab4c957 2022-06-13 stsp 				ncommits++;
1286 0ab4c957 2022-06-13 stsp 		}
1287 0ab4c957 2022-06-13 stsp 		if (ncommits > ncommits_max) {
1288 0ab4c957 2022-06-13 stsp 			best_packidx_path = path_packidx;
1289 0ab4c957 2022-06-13 stsp 			nobj_max = nobj;
1290 0ab4c957 2022-06-13 stsp 			ncommits_max = ncommits;
1291 0ab4c957 2022-06-13 stsp 		}
1292 0ab4c957 2022-06-13 stsp 	}
1293 0ab4c957 2022-06-13 stsp 
1294 eb7b30a1 2022-06-13 stsp 	if (best_packidx_path && err == NULL) {
1295 0ab4c957 2022-06-13 stsp 		err = got_repo_get_packidx(best_packidx, best_packidx_path,
1296 0ab4c957 2022-06-13 stsp 		    repo);
1297 0ab4c957 2022-06-13 stsp 	}
1298 0ab4c957 2022-06-13 stsp 
1299 0ab4c957 2022-06-13 stsp 	return err;
1300 0ab4c957 2022-06-13 stsp }
1301 0ab4c957 2022-06-13 stsp 
1302 0ab4c957 2022-06-13 stsp static const struct got_error *
1303 b8af7c06 2022-03-15 stsp load_object_ids(int *ncolored, int *nfound, int *ntrees,
1304 b8af7c06 2022-03-15 stsp     struct got_object_idset *idset, struct got_object_id **theirs, int ntheirs,
1305 e6bcace5 2021-06-22 stsp     struct got_object_id **ours, int nours, struct got_repository *repo,
1306 d6a28ffe 2022-05-20 op     uint32_t seed, int loose_obj_only, got_pack_progress_cb progress_cb,
1307 d6a28ffe 2022-05-20 op     void *progress_arg, struct got_ratelimit *rl, got_cancel_cb cancel_cb,
1308 d6a28ffe 2022-05-20 op     void *cancel_arg)
1309 e6bcace5 2021-06-22 stsp {
1310 e6bcace5 2021-06-22 stsp 	const struct got_error *err = NULL;
1311 e6bcace5 2021-06-22 stsp 	struct got_object_id **ids = NULL;
1312 0ab4c957 2022-06-13 stsp 	struct got_packidx *packidx = NULL;
1313 db9b9b1c 2022-06-14 stsp 	int i, nobj = 0, obj_type, found_all_objects = 0;
1314 2f8438b0 2022-05-04 stsp 	struct got_object_idset *idset_exclude;
1315 2f8438b0 2022-05-04 stsp 
1316 2f8438b0 2022-05-04 stsp 	idset_exclude = got_object_idset_alloc();
1317 2f8438b0 2022-05-04 stsp 	if (idset_exclude == NULL)
1318 2f8438b0 2022-05-04 stsp 		return got_error_from_errno("got_object_idset_alloc");
1319 e6bcace5 2021-06-22 stsp 
1320 b8af7c06 2022-03-15 stsp 	*ncolored = 0;
1321 b8af7c06 2022-03-15 stsp 	*nfound = 0;
1322 b8af7c06 2022-03-15 stsp 	*ntrees = 0;
1323 b8af7c06 2022-03-15 stsp 
1324 b8af7c06 2022-03-15 stsp 	err = findtwixt(&ids, &nobj, ncolored, ours, nours, theirs, ntheirs,
1325 b8af7c06 2022-03-15 stsp 	    repo, progress_cb, progress_arg, rl, cancel_cb, cancel_arg);
1326 dc3fe1bf 2022-05-10 stsp 	if (err)
1327 e6bcace5 2021-06-22 stsp 		goto done;
1328 0ab4c957 2022-06-13 stsp 
1329 0ab4c957 2022-06-13 stsp 	err = find_pack_for_enumeration(&packidx, theirs, ntheirs, repo);
1330 0ab4c957 2022-06-13 stsp 	if (err)
1331 0ab4c957 2022-06-13 stsp 		goto done;
1332 0ab4c957 2022-06-13 stsp 	if (packidx) {
1333 301e83b3 2022-10-16 stsp 		err = got_pack_load_packed_object_ids(&found_all_objects,
1334 db9b9b1c 2022-06-14 stsp 		    theirs, ntheirs, NULL, 0, 0, seed, idset, idset_exclude,
1335 db9b9b1c 2022-06-14 stsp 		    loose_obj_only, repo, packidx, ncolored, nfound, ntrees,
1336 db9b9b1c 2022-06-14 stsp 		    progress_cb, progress_arg, rl, cancel_cb, cancel_arg);
1337 0ab4c957 2022-06-13 stsp 		if (err)
1338 0ab4c957 2022-06-13 stsp 			goto done;
1339 0ab4c957 2022-06-13 stsp 	}
1340 cee6a7ea 2022-06-07 stsp 
1341 e6bcace5 2021-06-22 stsp 	for (i = 0; i < ntheirs; i++) {
1342 05118f5a 2021-06-22 stsp 		struct got_object_id *id = theirs[i];
1343 05118f5a 2021-06-22 stsp 		if (id == NULL)
1344 05118f5a 2021-06-22 stsp 			continue;
1345 05118f5a 2021-06-22 stsp 		err = got_object_get_type(&obj_type, repo, id);
1346 05118f5a 2021-06-22 stsp 		if (err)
1347 05118f5a 2021-06-22 stsp 			return err;
1348 9d34261e 2022-04-07 stsp 		if (obj_type == GOT_OBJ_TYPE_COMMIT) {
1349 db9b9b1c 2022-06-14 stsp 			if (!found_all_objects) {
1350 db9b9b1c 2022-06-14 stsp 				err = load_commit(0, idset, idset_exclude,
1351 db9b9b1c 2022-06-14 stsp 				    id, repo, seed, loose_obj_only,
1352 db9b9b1c 2022-06-14 stsp 				    ncolored, nfound, ntrees,
1353 db9b9b1c 2022-06-14 stsp 				    progress_cb, progress_arg, rl,
1354 db9b9b1c 2022-06-14 stsp 				    cancel_cb, cancel_arg);
1355 db9b9b1c 2022-06-14 stsp 				if (err)
1356 db9b9b1c 2022-06-14 stsp 					goto done;
1357 db9b9b1c 2022-06-14 stsp 			}
1358 9d34261e 2022-04-07 stsp 		} else if (obj_type == GOT_OBJ_TYPE_TAG) {
1359 2f8438b0 2022-05-04 stsp 			err = load_tag(0, idset, idset_exclude, id, repo,
1360 d6a28ffe 2022-05-20 op 			    seed, loose_obj_only, ncolored, nfound, ntrees,
1361 9d34261e 2022-04-07 stsp 			    progress_cb, progress_arg, rl,
1362 9d34261e 2022-04-07 stsp 			    cancel_cb, cancel_arg);
1363 9d34261e 2022-04-07 stsp 			if (err)
1364 9d34261e 2022-04-07 stsp 				goto done;
1365 9d34261e 2022-04-07 stsp 		}
1366 05118f5a 2021-06-22 stsp 	}
1367 05118f5a 2021-06-22 stsp 
1368 db9b9b1c 2022-06-14 stsp 	found_all_objects = 0;
1369 0ab4c957 2022-06-13 stsp 	err = find_pack_for_enumeration(&packidx, ids, nobj, repo);
1370 0ab4c957 2022-06-13 stsp 	if (err)
1371 0ab4c957 2022-06-13 stsp 		goto done;
1372 0ab4c957 2022-06-13 stsp 	if (packidx) {
1373 301e83b3 2022-10-16 stsp 		err = got_pack_load_packed_object_ids(&found_all_objects, ids,
1374 db9b9b1c 2022-06-14 stsp 		    nobj, theirs, ntheirs, 1, seed, idset, idset_exclude,
1375 db9b9b1c 2022-06-14 stsp 		    loose_obj_only, repo, packidx, ncolored, nfound, ntrees,
1376 0ab4c957 2022-06-13 stsp 		    progress_cb, progress_arg, rl, cancel_cb, cancel_arg);
1377 0ab4c957 2022-06-13 stsp 		if (err)
1378 0ab4c957 2022-06-13 stsp 			goto done;
1379 0ab4c957 2022-06-13 stsp 	}
1380 0ab4c957 2022-06-13 stsp 
1381 db9b9b1c 2022-06-14 stsp 	if (!found_all_objects) {
1382 db9b9b1c 2022-06-14 stsp 		for (i = 0; i < nobj; i++) {
1383 db9b9b1c 2022-06-14 stsp 			err = load_commit(1, idset, idset_exclude, ids[i],
1384 db9b9b1c 2022-06-14 stsp 			    repo, seed, loose_obj_only, ncolored, nfound,
1385 db9b9b1c 2022-06-14 stsp 			    ntrees, progress_cb, progress_arg, rl,
1386 db9b9b1c 2022-06-14 stsp 			    cancel_cb, cancel_arg);
1387 db9b9b1c 2022-06-14 stsp 			if (err)
1388 db9b9b1c 2022-06-14 stsp 				goto done;
1389 db9b9b1c 2022-06-14 stsp 		}
1390 eca70f98 2021-09-03 stsp 	}
1391 eca70f98 2021-09-03 stsp 
1392 05118f5a 2021-06-22 stsp 	for (i = 0; i < nours; i++) {
1393 05118f5a 2021-06-22 stsp 		struct got_object_id *id = ours[i];
1394 67fd6849 2022-02-13 stsp 		struct got_pack_meta *m;
1395 05118f5a 2021-06-22 stsp 		if (id == NULL)
1396 05118f5a 2021-06-22 stsp 			continue;
1397 67fd6849 2022-02-13 stsp 		m = got_object_idset_get(idset, id);
1398 67fd6849 2022-02-13 stsp 		if (m == NULL) {
1399 07165b17 2021-07-01 stsp 			err = got_object_get_type(&obj_type, repo, id);
1400 07165b17 2021-07-01 stsp 			if (err)
1401 07165b17 2021-07-01 stsp 				goto done;
1402 07165b17 2021-07-01 stsp 		} else
1403 67fd6849 2022-02-13 stsp 			obj_type = m->obj_type;
1404 05118f5a 2021-06-22 stsp 		if (obj_type != GOT_OBJ_TYPE_TAG)
1405 05118f5a 2021-06-22 stsp 			continue;
1406 2f8438b0 2022-05-04 stsp 		err = load_tag(1, idset, idset_exclude, id, repo,
1407 d6a28ffe 2022-05-20 op 		    seed, loose_obj_only, ncolored, nfound, ntrees,
1408 2f8438b0 2022-05-04 stsp 		    progress_cb, progress_arg, rl, cancel_cb, cancel_arg);
1409 05118f5a 2021-06-22 stsp 		if (err)
1410 e6bcace5 2021-06-22 stsp 			goto done;
1411 e6bcace5 2021-06-22 stsp 	}
1412 e6bcace5 2021-06-22 stsp done:
1413 e6bcace5 2021-06-22 stsp 	for (i = 0; i < nobj; i++) {
1414 e6bcace5 2021-06-22 stsp 		free(ids[i]);
1415 e6bcace5 2021-06-22 stsp 	}
1416 e6bcace5 2021-06-22 stsp 	free(ids);
1417 2f8438b0 2022-05-04 stsp 	got_object_idset_free(idset_exclude);
1418 e6bcace5 2021-06-22 stsp 	return err;
1419 e6bcace5 2021-06-22 stsp }
1420 e6bcace5 2021-06-22 stsp 
1421 336075a4 2022-06-25 op static const struct got_error *
1422 894e4711 2022-10-15 stsp hwrite(int fd, const void *buf, off_t len, SHA1_CTX *ctx)
1423 e6bcace5 2021-06-22 stsp {
1424 e6bcace5 2021-06-22 stsp 	SHA1Update(ctx, buf, len);
1425 13b2bc37 2022-10-23 stsp 	return got_poll_write_full(fd, buf, len);
1426 2d9e6abf 2022-05-04 stsp }
1427 2d9e6abf 2022-05-04 stsp 
1428 336075a4 2022-06-25 op static const struct got_error *
1429 894e4711 2022-10-15 stsp hcopy(FILE *fsrc, int fd_dst, off_t len, SHA1_CTX *ctx)
1430 2d9e6abf 2022-05-04 stsp {
1431 13b2bc37 2022-10-23 stsp 	const struct got_error *err;
1432 2d9e6abf 2022-05-04 stsp 	unsigned char buf[65536];
1433 2d9e6abf 2022-05-04 stsp 	off_t remain = len;
1434 2d9e6abf 2022-05-04 stsp 	size_t n;
1435 2d9e6abf 2022-05-04 stsp 
1436 2d9e6abf 2022-05-04 stsp 	while (remain > 0) {
1437 2d9e6abf 2022-05-04 stsp 		size_t copylen = MIN(sizeof(buf), remain);
1438 2d9e6abf 2022-05-04 stsp 		n = fread(buf, 1, copylen, fsrc);
1439 2d9e6abf 2022-05-04 stsp 		if (n != copylen)
1440 2d9e6abf 2022-05-04 stsp 			return got_ferror(fsrc, GOT_ERR_IO);
1441 2d9e6abf 2022-05-04 stsp 		SHA1Update(ctx, buf, copylen);
1442 13b2bc37 2022-10-23 stsp 		err = got_poll_write_full(fd_dst, buf, copylen);
1443 13b2bc37 2022-10-23 stsp 		if (err)
1444 13b2bc37 2022-10-23 stsp 			return err;
1445 2d9e6abf 2022-05-04 stsp 		remain -= copylen;
1446 2d9e6abf 2022-05-04 stsp 	}
1447 2d9e6abf 2022-05-04 stsp 
1448 e6bcace5 2021-06-22 stsp 	return NULL;
1449 e6bcace5 2021-06-22 stsp }
1450 e93fb944 2022-05-10 stsp 
1451 336075a4 2022-06-25 op static const struct got_error *
1452 e93fb944 2022-05-10 stsp hcopy_mmap(uint8_t *src, off_t src_offset, size_t src_size,
1453 894e4711 2022-10-15 stsp     int fd, off_t len, SHA1_CTX *ctx)
1454 e93fb944 2022-05-10 stsp {
1455 e93fb944 2022-05-10 stsp 	if (src_offset + len > src_size)
1456 e93fb944 2022-05-10 stsp 		return got_error(GOT_ERR_RANGE);
1457 e93fb944 2022-05-10 stsp 
1458 e93fb944 2022-05-10 stsp 	SHA1Update(ctx, src + src_offset, len);
1459 13b2bc37 2022-10-23 stsp 	return got_poll_write_full(fd, src + src_offset, len);
1460 e93fb944 2022-05-10 stsp }
1461 e93fb944 2022-05-10 stsp 
1462 e6bcace5 2021-06-22 stsp static void
1463 e6bcace5 2021-06-22 stsp putbe32(char *b, uint32_t n)
1464 e6bcace5 2021-06-22 stsp {
1465 e6bcace5 2021-06-22 stsp 	b[0] = n >> 24;
1466 e6bcace5 2021-06-22 stsp 	b[1] = n >> 16;
1467 e6bcace5 2021-06-22 stsp 	b[2] = n >> 8;
1468 e6bcace5 2021-06-22 stsp 	b[3] = n >> 0;
1469 e6bcace5 2021-06-22 stsp }
1470 e6bcace5 2021-06-22 stsp 
1471 e6bcace5 2021-06-22 stsp static int
1472 e6bcace5 2021-06-22 stsp write_order_cmp(const void *pa, const void *pb)
1473 e6bcace5 2021-06-22 stsp {
1474 e6bcace5 2021-06-22 stsp 	struct got_pack_meta *a, *b, *ahd, *bhd;
1475 e6bcace5 2021-06-22 stsp 
1476 e6bcace5 2021-06-22 stsp 	a = *(struct got_pack_meta **)pa;
1477 e6bcace5 2021-06-22 stsp 	b = *(struct got_pack_meta **)pb;
1478 e6bcace5 2021-06-22 stsp 	ahd = (a->head == NULL) ? a : a->head;
1479 e6bcace5 2021-06-22 stsp 	bhd = (b->head == NULL) ? b : b->head;
1480 611e8e31 2022-05-01 stsp 	if (bhd->mtime < ahd->mtime)
1481 611e8e31 2022-05-01 stsp 		return -1;
1482 611e8e31 2022-05-01 stsp 	if (bhd->mtime > ahd->mtime)
1483 611e8e31 2022-05-01 stsp 		return 1;
1484 611e8e31 2022-05-01 stsp 	if (bhd < ahd)
1485 611e8e31 2022-05-01 stsp 		return -1;
1486 611e8e31 2022-05-01 stsp 	if (bhd > ahd)
1487 611e8e31 2022-05-01 stsp 		return 1;
1488 e6bcace5 2021-06-22 stsp 	if (a->nchain != b->nchain)
1489 e6bcace5 2021-06-22 stsp 		return a->nchain - b->nchain;
1490 611e8e31 2022-05-01 stsp 	if (a->mtime < b->mtime)
1491 611e8e31 2022-05-01 stsp 		return -1;
1492 611e8e31 2022-05-01 stsp 	if (a->mtime > b->mtime)
1493 611e8e31 2022-05-01 stsp 		return 1;
1494 611e8e31 2022-05-01 stsp 	return got_object_id_cmp(&a->id, &b->id);
1495 e6bcace5 2021-06-22 stsp }
1496 e6bcace5 2021-06-22 stsp 
1497 67fd6849 2022-02-13 stsp static int
1498 67fd6849 2022-02-13 stsp reuse_write_order_cmp(const void *pa, const void *pb)
1499 67fd6849 2022-02-13 stsp {
1500 67fd6849 2022-02-13 stsp 	struct got_pack_meta *a, *b;
1501 67fd6849 2022-02-13 stsp 
1502 67fd6849 2022-02-13 stsp 	a = *(struct got_pack_meta **)pa;
1503 67fd6849 2022-02-13 stsp 	b = *(struct got_pack_meta **)pb;
1504 67fd6849 2022-02-13 stsp 
1505 67fd6849 2022-02-13 stsp 	if (a->reused_delta_offset < b->reused_delta_offset)
1506 67fd6849 2022-02-13 stsp 		return -1;
1507 67fd6849 2022-02-13 stsp 	if (a->reused_delta_offset > b->reused_delta_offset)
1508 67fd6849 2022-02-13 stsp 		return 1;
1509 67fd6849 2022-02-13 stsp 	return 0;
1510 67fd6849 2022-02-13 stsp }
1511 67fd6849 2022-02-13 stsp 
1512 e6bcace5 2021-06-22 stsp static const struct got_error *
1513 e6bcace5 2021-06-22 stsp packhdr(int *hdrlen, char *hdr, size_t bufsize, int obj_type, size_t len)
1514 e6bcace5 2021-06-22 stsp {
1515 e6bcace5 2021-06-22 stsp 	size_t i;
1516 e6bcace5 2021-06-22 stsp 
1517 e6bcace5 2021-06-22 stsp 	*hdrlen = 0;
1518 e6bcace5 2021-06-22 stsp 
1519 e6bcace5 2021-06-22 stsp 	hdr[0] = obj_type << 4;
1520 e6bcace5 2021-06-22 stsp 	hdr[0] |= len & 0xf;
1521 e6bcace5 2021-06-22 stsp 	len >>= 4;
1522 e6bcace5 2021-06-22 stsp 	for (i = 1; len != 0; i++){
1523 e6bcace5 2021-06-22 stsp 		if (i >= bufsize)
1524 e6bcace5 2021-06-22 stsp 			return got_error(GOT_ERR_NO_SPACE);
1525 e6bcace5 2021-06-22 stsp 		hdr[i - 1] |= GOT_DELTA_SIZE_MORE;
1526 e6bcace5 2021-06-22 stsp 		hdr[i] = len & GOT_DELTA_SIZE_VAL_MASK;
1527 e6bcace5 2021-06-22 stsp 		len >>= GOT_DELTA_SIZE_SHIFT;
1528 e6bcace5 2021-06-22 stsp 	}
1529 e6bcace5 2021-06-22 stsp 
1530 e6bcace5 2021-06-22 stsp 	*hdrlen = i;
1531 e6bcace5 2021-06-22 stsp 	return NULL;
1532 e6bcace5 2021-06-22 stsp }
1533 e6bcace5 2021-06-22 stsp 
1534 e6bcace5 2021-06-22 stsp static int
1535 e6bcace5 2021-06-22 stsp packoff(char *hdr, off_t off)
1536 e6bcace5 2021-06-22 stsp {
1537 e6bcace5 2021-06-22 stsp 	int i, j;
1538 e6bcace5 2021-06-22 stsp 	char rbuf[8];
1539 e6bcace5 2021-06-22 stsp 
1540 e6bcace5 2021-06-22 stsp 	rbuf[0] = off & GOT_DELTA_SIZE_VAL_MASK;
1541 e6bcace5 2021-06-22 stsp 	for (i = 1; (off >>= GOT_DELTA_SIZE_SHIFT) != 0; i++) {
1542 e6bcace5 2021-06-22 stsp 		rbuf[i] = (--off & GOT_DELTA_SIZE_VAL_MASK) |
1543 e6bcace5 2021-06-22 stsp 		    GOT_DELTA_SIZE_MORE;
1544 e6bcace5 2021-06-22 stsp 	}
1545 e6bcace5 2021-06-22 stsp 
1546 e6bcace5 2021-06-22 stsp 	j = 0;
1547 e6bcace5 2021-06-22 stsp 	while (i > 0)
1548 e6bcace5 2021-06-22 stsp 		hdr[j++] = rbuf[--i];
1549 e6bcace5 2021-06-22 stsp 	return j;
1550 e6bcace5 2021-06-22 stsp }
1551 e6bcace5 2021-06-22 stsp 
1552 e6bcace5 2021-06-22 stsp static const struct got_error *
1553 894e4711 2022-10-15 stsp deltahdr(off_t *packfile_size, SHA1_CTX *ctx, int packfd,
1554 67fd6849 2022-02-13 stsp     struct got_pack_meta *m)
1555 5060d5a1 2022-01-10 stsp {
1556 5060d5a1 2022-01-10 stsp 	const struct got_error *err;
1557 5060d5a1 2022-01-10 stsp 	char buf[32];
1558 5060d5a1 2022-01-10 stsp 	int nh;
1559 5060d5a1 2022-01-10 stsp 
1560 67fd6849 2022-02-13 stsp 	if (m->prev->off != 0) {
1561 5060d5a1 2022-01-10 stsp 		err = packhdr(&nh, buf, sizeof(buf),
1562 5060d5a1 2022-01-10 stsp 		    GOT_OBJ_TYPE_OFFSET_DELTA, m->delta_len);
1563 5060d5a1 2022-01-10 stsp 		if (err)
1564 5060d5a1 2022-01-10 stsp 			return err;
1565 5060d5a1 2022-01-10 stsp 		nh += packoff(buf + nh, m->off - m->prev->off);
1566 894e4711 2022-10-15 stsp 		err = hwrite(packfd, buf, nh, ctx);
1567 5060d5a1 2022-01-10 stsp 		if (err)
1568 5060d5a1 2022-01-10 stsp 			return err;
1569 5060d5a1 2022-01-10 stsp 		*packfile_size += nh;
1570 5060d5a1 2022-01-10 stsp 	} else {
1571 5060d5a1 2022-01-10 stsp 		err = packhdr(&nh, buf, sizeof(buf),
1572 5060d5a1 2022-01-10 stsp 		    GOT_OBJ_TYPE_REF_DELTA, m->delta_len);
1573 5060d5a1 2022-01-10 stsp 		if (err)
1574 5060d5a1 2022-01-10 stsp 			return err;
1575 894e4711 2022-10-15 stsp 		err = hwrite(packfd, buf, nh, ctx);
1576 5060d5a1 2022-01-10 stsp 		if (err)
1577 5060d5a1 2022-01-10 stsp 			return err;
1578 5060d5a1 2022-01-10 stsp 		*packfile_size += nh;
1579 894e4711 2022-10-15 stsp 		err = hwrite(packfd, m->prev->id.sha1,
1580 5060d5a1 2022-01-10 stsp 		    sizeof(m->prev->id.sha1), ctx);
1581 5060d5a1 2022-01-10 stsp 		if (err)
1582 5060d5a1 2022-01-10 stsp 			return err;
1583 5060d5a1 2022-01-10 stsp 		*packfile_size += sizeof(m->prev->id.sha1);
1584 5060d5a1 2022-01-10 stsp 	}
1585 5060d5a1 2022-01-10 stsp 
1586 5060d5a1 2022-01-10 stsp 	return NULL;
1587 5060d5a1 2022-01-10 stsp }
1588 5060d5a1 2022-01-10 stsp 
1589 5060d5a1 2022-01-10 stsp static const struct got_error *
1590 894e4711 2022-10-15 stsp write_packed_object(off_t *packfile_size, int packfd,
1591 e93fb944 2022-05-10 stsp     FILE *delta_cache, uint8_t *delta_cache_map, size_t delta_cache_size,
1592 e93fb944 2022-05-10 stsp     struct got_pack_meta *m, int *outfd, SHA1_CTX *ctx,
1593 e93fb944 2022-05-10 stsp     struct got_repository *repo)
1594 67fd6849 2022-02-13 stsp {
1595 67fd6849 2022-02-13 stsp 	const struct got_error *err = NULL;
1596 67fd6849 2022-02-13 stsp 	struct got_deflate_checksum csum;
1597 67fd6849 2022-02-13 stsp 	char buf[32];
1598 67fd6849 2022-02-13 stsp 	int nh;
1599 67fd6849 2022-02-13 stsp 	struct got_raw_object *raw = NULL;
1600 67fd6849 2022-02-13 stsp 	off_t outlen;
1601 67fd6849 2022-02-13 stsp 
1602 67fd6849 2022-02-13 stsp 	csum.output_sha1 = ctx;
1603 67fd6849 2022-02-13 stsp 	csum.output_crc = NULL;
1604 67fd6849 2022-02-13 stsp 
1605 894e4711 2022-10-15 stsp 	m->off = *packfile_size;
1606 67fd6849 2022-02-13 stsp 	if (m->delta_len == 0) {
1607 67fd6849 2022-02-13 stsp 		err = got_object_raw_open(&raw, outfd, repo, &m->id);
1608 67fd6849 2022-02-13 stsp 		if (err)
1609 67fd6849 2022-02-13 stsp 			goto done;
1610 67fd6849 2022-02-13 stsp 		err = packhdr(&nh, buf, sizeof(buf),
1611 67fd6849 2022-02-13 stsp 		    m->obj_type, raw->size);
1612 67fd6849 2022-02-13 stsp 		if (err)
1613 67fd6849 2022-02-13 stsp 			goto done;
1614 894e4711 2022-10-15 stsp 		err = hwrite(packfd, buf, nh, ctx);
1615 67fd6849 2022-02-13 stsp 		if (err)
1616 67fd6849 2022-02-13 stsp 			goto done;
1617 67fd6849 2022-02-13 stsp 		*packfile_size += nh;
1618 67fd6849 2022-02-13 stsp 		if (raw->f == NULL) {
1619 894e4711 2022-10-15 stsp 			err = got_deflate_to_fd_mmap(&outlen,
1620 67fd6849 2022-02-13 stsp 			    raw->data + raw->hdrlen, 0, raw->size,
1621 894e4711 2022-10-15 stsp 			    packfd, &csum);
1622 67fd6849 2022-02-13 stsp 			if (err)
1623 67fd6849 2022-02-13 stsp 				goto done;
1624 67fd6849 2022-02-13 stsp 		} else {
1625 67fd6849 2022-02-13 stsp 			if (fseeko(raw->f, raw->hdrlen, SEEK_SET)
1626 67fd6849 2022-02-13 stsp 			    == -1) {
1627 67fd6849 2022-02-13 stsp 				err = got_error_from_errno("fseeko");
1628 67fd6849 2022-02-13 stsp 				goto done;
1629 67fd6849 2022-02-13 stsp 			}
1630 894e4711 2022-10-15 stsp 			err = got_deflate_to_fd(&outlen, raw->f,
1631 894e4711 2022-10-15 stsp 			    raw->size, packfd, &csum);
1632 67fd6849 2022-02-13 stsp 			if (err)
1633 67fd6849 2022-02-13 stsp 				goto done;
1634 67fd6849 2022-02-13 stsp 		}
1635 67fd6849 2022-02-13 stsp 		*packfile_size += outlen;
1636 67fd6849 2022-02-13 stsp 		got_object_raw_close(raw);
1637 67fd6849 2022-02-13 stsp 		raw = NULL;
1638 67fd6849 2022-02-13 stsp 	} else if (m->delta_buf) {
1639 894e4711 2022-10-15 stsp 		err = deltahdr(packfile_size, ctx, packfd, m);
1640 67fd6849 2022-02-13 stsp 		if (err)
1641 67fd6849 2022-02-13 stsp 			goto done;
1642 894e4711 2022-10-15 stsp 		err = hwrite(packfd, m->delta_buf,
1643 2d9e6abf 2022-05-04 stsp 		    m->delta_compressed_len, ctx);
1644 67fd6849 2022-02-13 stsp 		if (err)
1645 67fd6849 2022-02-13 stsp 			goto done;
1646 2d9e6abf 2022-05-04 stsp 		*packfile_size += m->delta_compressed_len;
1647 67fd6849 2022-02-13 stsp 		free(m->delta_buf);
1648 67fd6849 2022-02-13 stsp 		m->delta_buf = NULL;
1649 e93fb944 2022-05-10 stsp 	} else if (delta_cache_map) {
1650 894e4711 2022-10-15 stsp 		err = deltahdr(packfile_size, ctx, packfd, m);
1651 e93fb944 2022-05-10 stsp 		if (err)
1652 e93fb944 2022-05-10 stsp 			goto done;
1653 e93fb944 2022-05-10 stsp 		err = hcopy_mmap(delta_cache_map, m->delta_offset,
1654 894e4711 2022-10-15 stsp 		    delta_cache_size, packfd, m->delta_compressed_len,
1655 e93fb944 2022-05-10 stsp 		    ctx);
1656 e93fb944 2022-05-10 stsp 		if (err)
1657 e93fb944 2022-05-10 stsp 			goto done;
1658 e93fb944 2022-05-10 stsp 		*packfile_size += m->delta_compressed_len;
1659 67fd6849 2022-02-13 stsp 	} else {
1660 67fd6849 2022-02-13 stsp 		if (fseeko(delta_cache, m->delta_offset, SEEK_SET)
1661 67fd6849 2022-02-13 stsp 		    == -1) {
1662 67fd6849 2022-02-13 stsp 			err = got_error_from_errno("fseeko");
1663 67fd6849 2022-02-13 stsp 			goto done;
1664 67fd6849 2022-02-13 stsp 		}
1665 894e4711 2022-10-15 stsp 		err = deltahdr(packfile_size, ctx, packfd, m);
1666 67fd6849 2022-02-13 stsp 		if (err)
1667 67fd6849 2022-02-13 stsp 			goto done;
1668 894e4711 2022-10-15 stsp 		err = hcopy(delta_cache, packfd,
1669 2d9e6abf 2022-05-04 stsp 		    m->delta_compressed_len, ctx);
1670 67fd6849 2022-02-13 stsp 		if (err)
1671 67fd6849 2022-02-13 stsp 			goto done;
1672 2d9e6abf 2022-05-04 stsp 		*packfile_size += m->delta_compressed_len;
1673 67fd6849 2022-02-13 stsp 	}
1674 67fd6849 2022-02-13 stsp done:
1675 67fd6849 2022-02-13 stsp 	if (raw)
1676 67fd6849 2022-02-13 stsp 		got_object_raw_close(raw);
1677 67fd6849 2022-02-13 stsp 	return err;
1678 67fd6849 2022-02-13 stsp }
1679 67fd6849 2022-02-13 stsp 
1680 67fd6849 2022-02-13 stsp static const struct got_error *
1681 894e4711 2022-10-15 stsp genpack(uint8_t *pack_sha1, int packfd, FILE *delta_cache,
1682 67fd6849 2022-02-13 stsp     struct got_pack_meta **deltify, int ndeltify,
1683 67fd6849 2022-02-13 stsp     struct got_pack_meta **reuse, int nreuse,
1684 b8af7c06 2022-03-15 stsp     int ncolored, int nfound, int ntrees, int nours,
1685 b8af7c06 2022-03-15 stsp     struct got_repository *repo,
1686 05118f5a 2021-06-22 stsp     got_pack_progress_cb progress_cb, void *progress_arg,
1687 211cfef0 2022-01-05 stsp     struct got_ratelimit *rl,
1688 05118f5a 2021-06-22 stsp     got_cancel_cb cancel_cb, void *cancel_arg)
1689 e6bcace5 2021-06-22 stsp {
1690 e6bcace5 2021-06-22 stsp 	const struct got_error *err = NULL;
1691 67fd6849 2022-02-13 stsp 	int i;
1692 e6bcace5 2021-06-22 stsp 	SHA1_CTX ctx;
1693 e6bcace5 2021-06-22 stsp 	struct got_pack_meta *m;
1694 08347b73 2021-10-14 stsp 	char buf[32];
1695 67fd6849 2022-02-13 stsp 	off_t packfile_size = 0;
1696 cc7a354a 2021-10-15 stsp 	int outfd = -1;
1697 e93fb944 2022-05-10 stsp 	int delta_cache_fd = -1;
1698 e93fb944 2022-05-10 stsp 	uint8_t *delta_cache_map = NULL;
1699 e93fb944 2022-05-10 stsp 	size_t delta_cache_size = 0;
1700 e6bcace5 2021-06-22 stsp 
1701 e6bcace5 2021-06-22 stsp 	SHA1Init(&ctx);
1702 e6bcace5 2021-06-22 stsp 
1703 e93fb944 2022-05-10 stsp #ifndef GOT_PACK_NO_MMAP
1704 e93fb944 2022-05-10 stsp 	delta_cache_fd = dup(fileno(delta_cache));
1705 e93fb944 2022-05-10 stsp 	if (delta_cache_fd != -1) {
1706 e93fb944 2022-05-10 stsp 		struct stat sb;
1707 e93fb944 2022-05-10 stsp 		if (fstat(delta_cache_fd, &sb) == -1) {
1708 e93fb944 2022-05-10 stsp 			err = got_error_from_errno("fstat");
1709 e93fb944 2022-05-10 stsp 			goto done;
1710 e93fb944 2022-05-10 stsp 		}
1711 e93fb944 2022-05-10 stsp 		if (sb.st_size > 0 && sb.st_size <= SIZE_MAX) {
1712 e93fb944 2022-05-10 stsp 			delta_cache_map = mmap(NULL, sb.st_size,
1713 e93fb944 2022-05-10 stsp 			    PROT_READ, MAP_PRIVATE, delta_cache_fd, 0);
1714 e93fb944 2022-05-10 stsp 			if (delta_cache_map == MAP_FAILED) {
1715 e93fb944 2022-05-10 stsp 				if (errno != ENOMEM) {
1716 e93fb944 2022-05-10 stsp 					err = got_error_from_errno("mmap");
1717 e93fb944 2022-05-10 stsp 					goto done;
1718 e93fb944 2022-05-10 stsp 				}
1719 e93fb944 2022-05-10 stsp 				delta_cache_map = NULL; /* fallback on stdio */
1720 e93fb944 2022-05-10 stsp 			} else
1721 e93fb944 2022-05-10 stsp 				delta_cache_size = (size_t)sb.st_size;
1722 e93fb944 2022-05-10 stsp 		}
1723 e93fb944 2022-05-10 stsp 	}
1724 e93fb944 2022-05-10 stsp #endif
1725 894e4711 2022-10-15 stsp 	err = hwrite(packfd, "PACK", 4, &ctx);
1726 e6bcace5 2021-06-22 stsp 	if (err)
1727 e93fb944 2022-05-10 stsp 		goto done;
1728 e6bcace5 2021-06-22 stsp 	putbe32(buf, GOT_PACKFILE_VERSION);
1729 894e4711 2022-10-15 stsp 	err = hwrite(packfd, buf, 4, &ctx);
1730 e6bcace5 2021-06-22 stsp 	if (err)
1731 e6bcace5 2021-06-22 stsp 		goto done;
1732 67fd6849 2022-02-13 stsp 	putbe32(buf, ndeltify + nreuse);
1733 894e4711 2022-10-15 stsp 	err = hwrite(packfd, buf, 4, &ctx);
1734 e6bcace5 2021-06-22 stsp 	if (err)
1735 e6bcace5 2021-06-22 stsp 		goto done;
1736 67fd6849 2022-02-13 stsp 
1737 67fd6849 2022-02-13 stsp 	qsort(deltify, ndeltify, sizeof(struct got_pack_meta *),
1738 67fd6849 2022-02-13 stsp 	    write_order_cmp);
1739 67fd6849 2022-02-13 stsp 	for (i = 0; i < ndeltify; i++) {
1740 301e83b3 2022-10-16 stsp 		err = got_pack_report_progress(progress_cb, progress_arg, rl,
1741 b8af7c06 2022-03-15 stsp 		    ncolored, nfound, ntrees, packfile_size, nours,
1742 b8af7c06 2022-03-15 stsp 		    ndeltify + nreuse, ndeltify + nreuse, i);
1743 211cfef0 2022-01-05 stsp 		if (err)
1744 211cfef0 2022-01-05 stsp 			goto done;
1745 67fd6849 2022-02-13 stsp 		m = deltify[i];
1746 894e4711 2022-10-15 stsp 		err = write_packed_object(&packfile_size, packfd,
1747 e93fb944 2022-05-10 stsp 		    delta_cache, delta_cache_map, delta_cache_size,
1748 e93fb944 2022-05-10 stsp 		    m, &outfd, &ctx, repo);
1749 67fd6849 2022-02-13 stsp 		if (err)
1750 67fd6849 2022-02-13 stsp 			goto done;
1751 e6bcace5 2021-06-22 stsp 	}
1752 67fd6849 2022-02-13 stsp 
1753 67fd6849 2022-02-13 stsp 	qsort(reuse, nreuse, sizeof(struct got_pack_meta *),
1754 67fd6849 2022-02-13 stsp 	    reuse_write_order_cmp);
1755 67fd6849 2022-02-13 stsp 	for (i = 0; i < nreuse; i++) {
1756 301e83b3 2022-10-16 stsp 		err = got_pack_report_progress(progress_cb, progress_arg, rl,
1757 b8af7c06 2022-03-15 stsp 		    ncolored, nfound, ntrees, packfile_size, nours,
1758 b8af7c06 2022-03-15 stsp 		    ndeltify + nreuse, ndeltify + nreuse, ndeltify + i);
1759 67fd6849 2022-02-13 stsp 		if (err)
1760 67fd6849 2022-02-13 stsp 			goto done;
1761 67fd6849 2022-02-13 stsp 		m = reuse[i];
1762 894e4711 2022-10-15 stsp 		err = write_packed_object(&packfile_size, packfd,
1763 e93fb944 2022-05-10 stsp 		    delta_cache, delta_cache_map, delta_cache_size,
1764 e93fb944 2022-05-10 stsp 		    m, &outfd, &ctx, repo);
1765 67fd6849 2022-02-13 stsp 		if (err)
1766 67fd6849 2022-02-13 stsp 			goto done;
1767 67fd6849 2022-02-13 stsp 	}
1768 67fd6849 2022-02-13 stsp 
1769 e6bcace5 2021-06-22 stsp 	SHA1Final(pack_sha1, &ctx);
1770 13b2bc37 2022-10-23 stsp 	err = got_poll_write_full(packfd, pack_sha1, SHA1_DIGEST_LENGTH);
1771 894e4711 2022-10-15 stsp 	if (err)
1772 894e4711 2022-10-15 stsp 		goto done;
1773 05118f5a 2021-06-22 stsp 	packfile_size += SHA1_DIGEST_LENGTH;
1774 dc7edd42 2021-08-22 stsp 	packfile_size += sizeof(struct got_packfile_hdr);
1775 211cfef0 2022-01-05 stsp 	if (progress_cb) {
1776 b8af7c06 2022-03-15 stsp 		err = progress_cb(progress_arg, ncolored, nfound, ntrees,
1777 b8af7c06 2022-03-15 stsp 		    packfile_size, nours, ndeltify + nreuse,
1778 b8af7c06 2022-03-15 stsp 		    ndeltify + nreuse, ndeltify + nreuse);
1779 211cfef0 2022-01-05 stsp 		if (err)
1780 211cfef0 2022-01-05 stsp 			goto done;
1781 211cfef0 2022-01-05 stsp 	}
1782 e6bcace5 2021-06-22 stsp done:
1783 cc7a354a 2021-10-15 stsp 	if (outfd != -1 && close(outfd) == -1 && err == NULL)
1784 cc7a354a 2021-10-15 stsp 		err = got_error_from_errno("close");
1785 e93fb944 2022-05-10 stsp 	if (delta_cache_map && munmap(delta_cache_map, delta_cache_size) == -1)
1786 e93fb944 2022-05-10 stsp 		err = got_error_from_errno("munmap");
1787 e93fb944 2022-05-10 stsp 	if (delta_cache_fd != -1 && close(delta_cache_fd) == -1 && err == NULL)
1788 e93fb944 2022-05-10 stsp 		err = got_error_from_errno("close");
1789 e6bcace5 2021-06-22 stsp 	return err;
1790 67fd6849 2022-02-13 stsp }
1791 67fd6849 2022-02-13 stsp 
1792 67fd6849 2022-02-13 stsp static const struct got_error *
1793 67fd6849 2022-02-13 stsp add_meta_idset_cb(struct got_object_id *id, void *data, void *arg)
1794 67fd6849 2022-02-13 stsp {
1795 67fd6849 2022-02-13 stsp 	struct got_pack_meta *m = data;
1796 67fd6849 2022-02-13 stsp 	struct got_pack_metavec *v = arg;
1797 f5e78e05 2022-05-04 stsp 
1798 411cbec1 2022-05-20 stsp 	if (m->reused_delta_offset != 0)
1799 f5e78e05 2022-05-04 stsp 		return NULL;
1800 67fd6849 2022-02-13 stsp 
1801 301e83b3 2022-10-16 stsp 	return got_pack_add_meta(m, v);
1802 67fd6849 2022-02-13 stsp }
1803 67fd6849 2022-02-13 stsp 
1804 e6bcace5 2021-06-22 stsp const struct got_error *
1805 a32780aa 2022-10-15 stsp got_pack_create(uint8_t *packsha1, int packfd, FILE *delta_cache,
1806 e6bcace5 2021-06-22 stsp     struct got_object_id **theirs, int ntheirs,
1807 e6bcace5 2021-06-22 stsp     struct got_object_id **ours, int nours,
1808 f8a36e22 2021-08-26 stsp     struct got_repository *repo, int loose_obj_only, int allow_empty,
1809 05118f5a 2021-06-22 stsp     got_pack_progress_cb progress_cb, void *progress_arg,
1810 cae60ab8 2022-10-18 stsp     struct got_ratelimit *rl, got_cancel_cb cancel_cb, void *cancel_arg)
1811 e6bcace5 2021-06-22 stsp {
1812 e6bcace5 2021-06-22 stsp 	const struct got_error *err;
1813 67fd6849 2022-02-13 stsp 	int delta_cache_fd = -1;
1814 67fd6849 2022-02-13 stsp 	struct got_object_idset *idset;
1815 67fd6849 2022-02-13 stsp 	struct got_pack_metavec deltify, reuse;
1816 b8af7c06 2022-03-15 stsp 	int ncolored = 0, nfound = 0, ntrees = 0;
1817 f5e78e05 2022-05-04 stsp 	size_t ndeltify;
1818 d6a28ffe 2022-05-20 op 	uint32_t seed;
1819 e6bcace5 2021-06-22 stsp 
1820 d6a28ffe 2022-05-20 op 	seed = arc4random();
1821 d6a28ffe 2022-05-20 op 
1822 67fd6849 2022-02-13 stsp 	memset(&deltify, 0, sizeof(deltify));
1823 67fd6849 2022-02-13 stsp 	memset(&reuse, 0, sizeof(reuse));
1824 67fd6849 2022-02-13 stsp 
1825 67fd6849 2022-02-13 stsp 	idset = got_object_idset_alloc();
1826 67fd6849 2022-02-13 stsp 	if (idset == NULL)
1827 67fd6849 2022-02-13 stsp 		return got_error_from_errno("got_object_idset_alloc");
1828 67fd6849 2022-02-13 stsp 
1829 b8af7c06 2022-03-15 stsp 	err = load_object_ids(&ncolored, &nfound, &ntrees, idset, theirs,
1830 d6a28ffe 2022-05-20 op 	    ntheirs, ours, nours, repo, seed, loose_obj_only,
1831 cae60ab8 2022-10-18 stsp 	    progress_cb, progress_arg, rl, cancel_cb, cancel_arg);
1832 e6bcace5 2021-06-22 stsp 	if (err)
1833 17259bfa 2022-05-19 stsp 		goto done;
1834 e6bcace5 2021-06-22 stsp 
1835 28526235 2022-02-13 stsp 	if (progress_cb) {
1836 b8af7c06 2022-03-15 stsp 		err = progress_cb(progress_arg, ncolored, nfound, ntrees,
1837 b8af7c06 2022-03-15 stsp 		    0L, nours, got_object_idset_num_elements(idset), 0, 0);
1838 28526235 2022-02-13 stsp 		if (err)
1839 28526235 2022-02-13 stsp 			goto done;
1840 28526235 2022-02-13 stsp 	}
1841 28526235 2022-02-13 stsp 
1842 67fd6849 2022-02-13 stsp 	if (got_object_idset_num_elements(idset) == 0 && !allow_empty) {
1843 05118f5a 2021-06-22 stsp 		err = got_error(GOT_ERR_CANNOT_PACK);
1844 05118f5a 2021-06-22 stsp 		goto done;
1845 05118f5a 2021-06-22 stsp 	}
1846 74881701 2021-10-15 stsp 
1847 a32780aa 2022-10-15 stsp 	delta_cache_fd = dup(fileno(delta_cache));
1848 67fd6849 2022-02-13 stsp 	if (delta_cache_fd == -1) {
1849 a32780aa 2022-10-15 stsp 		err = got_error_from_errno("dup");
1850 74881701 2021-10-15 stsp 		goto done;
1851 74881701 2021-10-15 stsp 	}
1852 74881701 2021-10-15 stsp 
1853 67fd6849 2022-02-13 stsp 	reuse.metasz = 64;
1854 67fd6849 2022-02-13 stsp 	reuse.meta = calloc(reuse.metasz,
1855 67fd6849 2022-02-13 stsp 	    sizeof(struct got_pack_meta *));
1856 67fd6849 2022-02-13 stsp 	if (reuse.meta == NULL) {
1857 67fd6849 2022-02-13 stsp 		err = got_error_from_errno("calloc");
1858 67fd6849 2022-02-13 stsp 		goto done;
1859 67fd6849 2022-02-13 stsp 	}
1860 67fd6849 2022-02-13 stsp 
1861 301e83b3 2022-10-16 stsp 	err = got_pack_search_deltas(&reuse, idset, delta_cache_fd,
1862 301e83b3 2022-10-16 stsp 	    ncolored, nfound, ntrees, nours,
1863 cae60ab8 2022-10-18 stsp 	    repo, progress_cb, progress_arg, rl, cancel_cb, cancel_arg);
1864 67fd6849 2022-02-13 stsp 	if (err)
1865 67fd6849 2022-02-13 stsp 		goto done;
1866 67fd6849 2022-02-13 stsp 
1867 67fd6849 2022-02-13 stsp 	if (fseeko(delta_cache, 0L, SEEK_END) == -1) {
1868 67fd6849 2022-02-13 stsp 		err = got_error_from_errno("fseeko");
1869 67fd6849 2022-02-13 stsp 		goto done;
1870 67fd6849 2022-02-13 stsp 	}
1871 67fd6849 2022-02-13 stsp 
1872 f5e78e05 2022-05-04 stsp 	ndeltify = got_object_idset_num_elements(idset) - reuse.nmeta;
1873 f5e78e05 2022-05-04 stsp 	if (ndeltify > 0) {
1874 f5e78e05 2022-05-04 stsp 		deltify.meta = calloc(ndeltify, sizeof(struct got_pack_meta *));
1875 f5e78e05 2022-05-04 stsp 		if (deltify.meta == NULL) {
1876 f5e78e05 2022-05-04 stsp 			err = got_error_from_errno("calloc");
1877 f5e78e05 2022-05-04 stsp 			goto done;
1878 f5e78e05 2022-05-04 stsp 		}
1879 f5e78e05 2022-05-04 stsp 		deltify.metasz = ndeltify;
1880 67fd6849 2022-02-13 stsp 
1881 f5e78e05 2022-05-04 stsp 		err = got_object_idset_for_each(idset, add_meta_idset_cb,
1882 f5e78e05 2022-05-04 stsp 		    &deltify);
1883 f8a36e22 2021-08-26 stsp 		if (err)
1884 f8a36e22 2021-08-26 stsp 			goto done;
1885 f5e78e05 2022-05-04 stsp 		if (deltify.nmeta > 0) {
1886 f5e78e05 2022-05-04 stsp 			err = pick_deltas(deltify.meta, deltify.nmeta,
1887 f5e78e05 2022-05-04 stsp 			    ncolored, nfound, ntrees, nours, reuse.nmeta,
1888 cae60ab8 2022-10-18 stsp 			    delta_cache, repo, progress_cb, progress_arg, rl,
1889 f5e78e05 2022-05-04 stsp 			    cancel_cb, cancel_arg);
1890 f5e78e05 2022-05-04 stsp 			if (err)
1891 f5e78e05 2022-05-04 stsp 				goto done;
1892 f5e78e05 2022-05-04 stsp 		}
1893 f8a36e22 2021-08-26 stsp 	}
1894 e6bcace5 2021-06-22 stsp 
1895 2d9e6abf 2022-05-04 stsp 	if (fflush(delta_cache) == EOF) {
1896 2d9e6abf 2022-05-04 stsp 		err = got_error_from_errno("fflush");
1897 2d9e6abf 2022-05-04 stsp 		goto done;
1898 2d9e6abf 2022-05-04 stsp 	}
1899 13b2bc37 2022-10-23 stsp 
1900 13b2bc37 2022-10-23 stsp 	if (progress_cb) {
1901 13b2bc37 2022-10-23 stsp 		/*
1902 13b2bc37 2022-10-23 stsp 		 * Report a 1-byte packfile write to indicate we are about
1903 13b2bc37 2022-10-23 stsp 		 * to start sending packfile data. gotd(8) needs this.
1904 13b2bc37 2022-10-23 stsp 		 */
1905 13b2bc37 2022-10-23 stsp 		err = progress_cb(progress_arg, ncolored, nfound, ntrees,
1906 13b2bc37 2022-10-23 stsp 		    1 /* packfile_size */, nours,
1907 13b2bc37 2022-10-23 stsp 		    got_object_idset_num_elements(idset),
1908 13b2bc37 2022-10-23 stsp 		    deltify.nmeta + reuse.nmeta, 0);
1909 13b2bc37 2022-10-23 stsp 		if (err)
1910 13b2bc37 2022-10-23 stsp 			goto done;
1911 13b2bc37 2022-10-23 stsp 	}
1912 13b2bc37 2022-10-23 stsp 
1913 894e4711 2022-10-15 stsp 	err = genpack(packsha1, packfd, delta_cache, deltify.meta,
1914 b8af7c06 2022-03-15 stsp 	    deltify.nmeta, reuse.meta, reuse.nmeta, ncolored, nfound, ntrees,
1915 cae60ab8 2022-10-18 stsp 	    nours, repo, progress_cb, progress_arg, rl,
1916 b8af7c06 2022-03-15 stsp 	    cancel_cb, cancel_arg);
1917 e6bcace5 2021-06-22 stsp 	if (err)
1918 e6bcace5 2021-06-22 stsp 		goto done;
1919 e6bcace5 2021-06-22 stsp done:
1920 67fd6849 2022-02-13 stsp 	free_nmeta(deltify.meta, deltify.nmeta);
1921 67fd6849 2022-02-13 stsp 	free_nmeta(reuse.meta, reuse.nmeta);
1922 67fd6849 2022-02-13 stsp 	got_object_idset_free(idset);
1923 67fd6849 2022-02-13 stsp 	if (delta_cache_fd != -1 && close(delta_cache_fd) == -1 && err == NULL)
1924 67fd6849 2022-02-13 stsp 		err = got_error_from_errno("close");
1925 e6bcace5 2021-06-22 stsp 	return err;
1926 e6bcace5 2021-06-22 stsp }