commit 13b2bc374c1870ec27b2eeb40efe68fd465f64bb from: Stefan Sperling date: Sun Oct 23 00:37:19 2022 UTC introduce gotd(8), a Git repository server reachable via ssh(1) This is an initial barebones implementation which provides the absolute minimum of functionality required to serve got(1) and git(1) clients. Basic fetch/send functionality has been tested and seems to work here, but this server is not yet expected to be stable. More testing is welcome. See the man pages for setup instructions. The current design uses one reader and one writer process per repository, which will have to be extended to N readers and N writers in the future. At startup, each process will chroot(2) into its assigned repository. This works because gotd(8) can only be started as root, and will then fork+exec, chroot, and privdrop. At present the parent process runs with the following pledge(2) promises: "stdio rpath wpath cpath proc getpw sendfd recvfd fattr flock unix unveil" The parent is the only process able to modify the repository in a way that becomes visible to Git clients. The parent uses unveil(2) to restrict its view of the filesystem to /tmp and the repositories listed in the configuration file gotd.conf(5). Per-repository chroot(2) processes use "stdio rpath sendfd recvfd". The writer defers to the parent for modifying references in the repository to point at newly uploaded commits. The reader is fine without such help, because Git repositories can be read without having to create any lock-files. gotd(8) requires a dedicated user ID, which should own repositories on the filesystem, and a separate secondary group, which should not have filesystem-level repository access, and must be allowed access to the gotd(8) socket. To obtain Git repository access, users must be members of this secondary group, and must have their login shell set to gotsh(1). gotsh(1) connects to the gotd(8) socket and speaks Git-protocol towards the client on the other end of the SSH connection. gotsh(1) is not an interactive command shell. At present, authenticated clients are granted read/write access to all repositories and all references (except for the "refs/got/" and the "refs/remotes/" namespaces, which are already being protected from modification). While complicated access control mechanism are not a design goal, making it possible to safely offer anonymous Git repository access over ssh(1) is on the road map. commit - 1af8800025bf22cf87cde038bbcfda0d2564eefc commit + 13b2bc374c1870ec27b2eeb40efe68fd465f64bb blob - 8b6c6977676250da107793984432e332c86e1251 blob + dfc19a3497e88dd48a01d757c267998ea8be1241 --- Makefile +++ Makefile @@ -7,7 +7,7 @@ SUBDIR += regress .endif .if make(clean) || make(obj) || make(release) -SUBDIR += gotweb gotwebd +SUBDIR += gotweb gotwebd gotd gotsh .endif .if make(tags) || make(cleandir) @@ -45,4 +45,12 @@ webd: webd-install: ${MAKE} -C gotwebd install +server: + ${MAKE} -C gotd + ${MAKE} -C gotsh + +server-install: + ${MAKE} -C gotd install + ${MAKE} -C gotsh install + .include blob - 46cff8ad9392ec8efcf5bb4848df93cee0dcca4d blob + 89c442c3a1faa744027e0511d4fd421f9474ee20 --- README +++ README @@ -11,7 +11,7 @@ for any functionality which has not yet been implement It will always remain possible to work with both Got and Git on the same repository. -To compile the Got tool suite on OpenBSD, run: +To compile the Got client tool suite on OpenBSD, run: $ make obj $ make @@ -57,6 +57,24 @@ Man page files in the Got source tree can be viewed wi EXAMPLES in got.1 contains a quick-start guide for OpenBSD developers. +To compile the Got server tool suite on OpenBSD, run: + + $ make obj + $ make server + $ make server-install + +This will install the following commands: + + gotd, the repository server program + gotsh, the login shell for users accessing the server via the network + +See the following manual page files for information about server setup: + + $ man -l gotd/gotd.8 + $ man -l gotd/gotd.conf.5 + $ man -l gotsh/gotsh.1 + + Game of Trees Web (Gotweb) is a CGI program which displays repository data and is designed to work with httpd(8) and slowcgi(8). It requires the Kristaps Dzonsons kcgi library, version 0.12.0 or greater. blob - 652bddb959b1bd1a528daf84da62d9a2f52d5d51 blob + 5d8f4c8b71e72109e36cd63e72e3f77d6e211bdd --- got/Makefile +++ got/Makefile @@ -15,7 +15,8 @@ SRCS= got.c blame.c commit_graph.c delta.c diff.c \ diff_patience.c send.c deltify.c pack_create.c dial.c \ bloom.c murmurhash2.c ratelimit.c patch.c sigs.c date.c \ object_open_privsep.c read_gitconfig_privsep.c \ - read_gotconfig_privsep.c pack_create_privsep.c + read_gotconfig_privsep.c pack_create_privsep.c pollfd.c \ + reference_parse.c MAN = ${PROG}.1 got-worktree.5 git-repository.5 got.conf.5 blob - 00e5e77e780621968aa2a18606882840fb9f4d2c blob + ef6f8e38e4cc6e4b1ea84fc72c54665a093b32ab --- gotadmin/Makefile +++ gotadmin/Makefile @@ -11,7 +11,7 @@ SRCS= gotadmin.c \ worktree_open.c sha1.c bloom.c murmurhash2.c ratelimit.c \ sigs.c buf.c date.c object_open_privsep.c \ read_gitconfig_privsep.c read_gotconfig_privsep.c \ - pack_create_privsep.c + pack_create_privsep.c pollfd.c reference_parse.c MAN = ${PROG}.1 CPPFLAGS = -I${.CURDIR}/../include -I${.CURDIR}/../lib blob - ae74a7f106da33fa511bef27e2ec204cdb107072 blob + 3f9b3180520b398deed0f58abea911812958de70 --- gotweb/Makefile +++ gotweb/Makefile @@ -16,7 +16,8 @@ SRCS = gotweb.c parse.y blame.c commit_graph.c delta. diff_output_plain.c diff_output_unidiff.c \ diff_output_edscript.c diff_patience.c \ bloom.c murmurhash2.c sigs.c date.c object_open_privsep.c \ - read_gitconfig_privsep.c read_gotconfig_privsep.c + read_gitconfig_privsep.c read_gotconfig_privsep.c \ + pollfd.c reference_parse.c MAN = ${PROG}.conf.5 ${PROG}.8 CPPFLAGS += -I${.CURDIR}/../include -I${.CURDIR}/../lib -I${.CURDIR} \ blob - 8a3f38ce4c45ce1386bdc180e342b043f8abe809 blob + 40a8a89af35bb2ad537815f6c18fd5fd38f1d013 --- gotweb/libexec/got-read-blob/Makefile +++ gotweb/libexec/got-read-blob/Makefile @@ -4,7 +4,7 @@ PROG= got-read-blob SRCS= got-read-blob.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c + path.c privsep.c sha1.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../../include -I${.CURDIR}/../../../lib LDADD = -lutil -lz blob - 0996068f0f3d0bb71779d03c1c1a7ec355644166 blob + cf8660bd4f6b01319d7acb1039c734dfa629da91 --- gotweb/libexec/got-read-commit/Makefile +++ gotweb/libexec/got-read-commit/Makefile @@ -4,7 +4,7 @@ PROG= got-read-commit SRCS= got-read-commit.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c + path.c privsep.c sha1.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../../include -I${.CURDIR}/../../../lib LDADD = -lutil -lz blob - 77cc7852cae9199c991f2908f2d9e8a27da650ee blob + 8879f48df6de273719bba71dc1c09ad0f12400e0 --- gotweb/libexec/got-read-gitconfig/Makefile +++ gotweb/libexec/got-read-gitconfig/Makefile @@ -4,8 +4,8 @@ PROG= got-read-gitconfig SRCS= got-read-gitconfig.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c gitconfig.c - + path.c privsep.c sha1.c gitconfig.c pollfd.c + CPPFLAGS = -I${.CURDIR}/../../../include -I${.CURDIR}/../../../lib LDADD = -lutil -lz DPADD = ${LIBZ} ${LIBUTIL} blob - 29605918a07b9e1969e3a2722605c3424e77d13f blob + 4b42a2accc0b3f70acfa45cbe4996b1eb9298e2f --- gotweb/libexec/got-read-gotconfig/Makefile +++ gotweb/libexec/got-read-gotconfig/Makefile @@ -4,7 +4,7 @@ PROG= got-read-gotconfig SRCS= got-read-gotconfig.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c parse.y + path.c privsep.c sha1.c parse.y pollfd.c CPPFLAGS = -I${.CURDIR}/../../../include -I${.CURDIR}/../../../lib \ -I${.CURDIR}/../../../libexec/got-read-gotconfig blob - 4889fe0bab46bbcdb20610ea9255916cd11d0e0d blob + 5de9e23dff74ce4b6002d8f839c2f1daaf5db9bf --- gotweb/libexec/got-read-object/Makefile +++ gotweb/libexec/got-read-object/Makefile @@ -4,7 +4,7 @@ PROG= got-read-object SRCS= got-read-object.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c + path.c privsep.c sha1.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../../include -I${.CURDIR}/../../../lib LDADD = -lutil -lz blob - a28b9cfd2d1c5d17ffcbe771790c2183e406f924 blob + f2b80f8b19809b9120fa5cd3263b48fac0ba01b1 --- gotweb/libexec/got-read-pack/Makefile +++ gotweb/libexec/got-read-pack/Makefile @@ -5,7 +5,7 @@ PROG= got-read-pack SRCS= got-read-pack.c delta.c error.c inflate.c object_cache.c \ object_idset.c object_parse.c opentemp.c pack.c path.c \ - privsep.c sha1.c delta_cache.c + privsep.c sha1.c delta_cache.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../../include -I${.CURDIR}/../../../lib LDADD = -lutil -lz blob - 3a0b798c57ea849bde67efe66e3b721f7486e287 blob + 203f0dabf48adc72a071b923ee0ca33f8cb7cc14 --- gotweb/libexec/got-read-tag/Makefile +++ gotweb/libexec/got-read-tag/Makefile @@ -4,7 +4,7 @@ PROG= got-read-tag SRCS= got-read-tag.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c + path.c privsep.c sha1.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../../include -I${.CURDIR}/../../../lib LDADD = -lutil -lz blob - 19a4c9cfa3379ca5fec4fafdc691d38c80c996e8 blob + 5e2f4bda5f9910c7dd4473e99d088fe961a387ec --- gotweb/libexec/got-read-tree/Makefile +++ gotweb/libexec/got-read-tree/Makefile @@ -4,7 +4,7 @@ PROG= got-read-tree SRCS= got-read-tree.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c + path.c privsep.c sha1.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../../include -I${.CURDIR}/../../../lib LDADD = -lutil -lz blob - /dev/null blob + 21de99d339291999ae9953fa640799e945b50872 (mode 644) --- /dev/null +++ gotd/Makefile @@ -0,0 +1,34 @@ +.PATH:${.CURDIR}/../lib + +.include "../got-version.mk" + +PREFIX ?= /usr/local +BINDIR ?= ${PREFIX}/sbin + +PROG= gotd +SRCS= gotd.c repo_read.c repo_write.c log.c privsep_stub.c imsg.c \ + parse.y pack_create.c ratelimit.c deltify.c \ + bloom.c buf.c date.c deflate.c delta.c delta_cache.c error.c \ + gitconfig.c gotconfig.c inflate.c lockfile.c murmurhash2.c \ + object.c object_cache.c object_create.c object_idset.c \ + object_open_io.c object_parse.c opentemp.c pack.c path.c \ + read_gitconfig.c read_gotconfig.c reference.c repository.c \ + sha1.c sigs.c pack_create_io.c pollfd.c reference_parse.c \ + repo_imsg.c pack_index.c + +MAN = ${PROG}.conf.5 ${PROG}.8 + +CPPFLAGS = -I${.CURDIR}/../include -I${.CURDIR}/../lib -I${.CURDIR} + +.if defined(PROFILE) +LDADD = -lutil_p -lz_p -lm_p -lc_p -levent_p +.else +LDADD = -lutil -lz -lm -levent +.endif +DPADD = ${LIBZ} ${LIBUTIL} + +.if ${GOT_RELEASE} != "Yes" +NOMAN = Yes +.endif + +.include blob - /dev/null blob + e99e90c3c5df0c094d5e6c41d6b62ea59c49e87b (mode 644) --- /dev/null +++ gotd/gotd.8 @@ -0,0 +1,74 @@ +.\" +.\" Copyright (c) 2022 Stefan Sperling +.\" +.\" Permission to use, copy, modify, and distribute this software for any +.\" purpose with or without fee is hereby granted, provided that the above +.\" copyright notice and this permission notice appear in all copies. +.\" +.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +.\" +.Dd $Mdocdate$ +.Dt GOTD 8 +.Os +.Sh NAME +.Nm gotd +.Nd Game of Trees Daemon +.Sh SYNOPSIS +.Nm +.Op Fl dv +.Op Fl f Ar config-file +.Sh DESCRIPTION +.Nm +is a Git repository server which listens on a +.Xr unix 4 +socket and relies on its companion tool +.Xr gotsh 1 +to handle Git-protocol communication over the network, via +.Xr ssh 1 . +.Pp +The Git repository format is described in +.Xr git-repository 5 . +.Pp +.Nm +requires a configuration file in order to run. +The configuration file format is described in +.Xr gotd.conf 5 . +.Pp +The options for +.Nm +are as follows: +.Bl -tag -width Ds +.It Fl d +Do not daemonize and log to stderr. +.It Fl f Ar config-file +Set the path to the configuration file. +If not specified, the file +.Pa /etc/gotd.conf +will be used. +.It Fl v +Verbose mode. +Verbosity increases if this option is used multiple times. +.Sh FILES +.Bl -tag -width Ds -compact +.It Pa /etc/gotd.conf +Default location of the configuration file. +.It Pa /var/run/gotd.sock +Default location of the unix socket which +.Nm +is listening on. +This path can be configured in +.Xr gotd.conf 5 . +.El +.Sh SEE ALSO +.Xr got 1 , +.Xr gotsh 1 , +.Xr git-repository 5 , +.Xr gotd.conf 5 +.Sh AUTHORS +.An Stefan Sperling Aq Mt stsp@openbsd.org blob - /dev/null blob + 06e0b7d3a39e2d8c01b4f512282f193eda50c43f (mode 644) --- /dev/null +++ gotd/gotd.c @@ -0,0 +1,2083 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "got_error.h" +#include "got_opentemp.h" +#include "got_path.h" +#include "got_repository.h" +#include "got_object.h" +#include "got_reference.h" + +#include "got_lib_delta.h" +#include "got_lib_object.h" +#include "got_lib_object_cache.h" +#include "got_lib_sha1.h" +#include "got_lib_gitproto.h" +#include "got_lib_pack.h" +#include "got_lib_repository.h" + +#include "gotd.h" +#include "log.h" +#include "repo_read.h" +#include "repo_write.h" + +#ifndef nitems +#define nitems(_a) (sizeof((_a)) / sizeof((_a)[0])) +#endif + +struct gotd_client { + STAILQ_ENTRY(gotd_client) entry; + enum gotd_client_state state; + struct gotd_client_capability *capabilities; + size_t ncapa_alloc; + size_t ncapabilities; + uint32_t id; + int fd; + int delta_cache_fd; + struct gotd_imsgev iev; + struct event tmo; + uid_t euid; + gid_t egid; + struct gotd_child_proc *repo_read; + struct gotd_child_proc *repo_write; + char *packfile_path; + char *packidx_path; + int nref_updates; +}; +STAILQ_HEAD(gotd_clients, gotd_client); + +static struct gotd_clients gotd_clients[GOTD_CLIENT_TABLE_SIZE]; +static SIPHASH_KEY clients_hash_key; +volatile int client_cnt; +static struct timeval timeout = { 3600, 0 }; +static int inflight; +static struct gotd gotd; + +void gotd_sighdlr(int sig, short event, void *arg); + +__dead static void +usage() +{ + fprintf(stderr, "%s: [-dv] [-f config-file]\n", getprogname()); + exit (1); +} + +static int +unix_socket_listen(const char *unix_socket_path, uid_t uid, gid_t gid) +{ + struct sockaddr_un sun; + int fd = -1; + mode_t old_umask, mode; + + fd = socket(AF_UNIX, SOCK_STREAM | SOCK_NONBLOCK| SOCK_CLOEXEC, 0); + if (fd == -1) { + log_warn("socket"); + return -1; + } + + sun.sun_family = AF_UNIX; + if (strlcpy(sun.sun_path, unix_socket_path, + sizeof(sun.sun_path)) >= sizeof(sun.sun_path)) { + log_warnx("%s: name too long", unix_socket_path); + close(fd); + return -1; + } + + if (unlink(unix_socket_path) == -1) { + if (errno != ENOENT) { + log_warn("unlink %s", unix_socket_path); + close(fd); + return -1; + } + } + + old_umask = umask(S_IXUSR|S_IXGRP|S_IWOTH|S_IROTH|S_IXOTH); + mode = S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP; + + if (bind(fd, (struct sockaddr *)&sun, sizeof(sun)) == -1) { + log_warn("bind: %s", unix_socket_path); + close(fd); + umask(old_umask); + return -1; + } + + umask(old_umask); + + if (chmod(unix_socket_path, mode) == -1) { + log_warn("chmod %o %s", mode, unix_socket_path); + close(fd); + unlink(unix_socket_path); + return -1; + } + + if (chown(unix_socket_path, uid, gid) == -1) { + log_warn("chown %s uid=%d gid=%d", unix_socket_path, uid, gid); + close(fd); + unlink(unix_socket_path); + return -1; + } + + if (listen(fd, GOTD_UNIX_SOCKET_BACKLOG) == -1) { + log_warn("listen"); + close(fd); + unlink(unix_socket_path); + return -1; + } + + return fd; +} + +static struct group * +match_group(gid_t *groups, int ngroups, const char *unix_group_name) +{ + struct group *gr; + int i; + + for (i = 0; i < ngroups; i++) { + gr = getgrgid(groups[i]); + if (gr == NULL) { + log_warn("getgrgid %d", groups[i]); + continue; + } + if (strcmp(gr->gr_name, unix_group_name) == 0) + return gr; + } + + return NULL; +} + +static int +accept_reserve(int fd, struct sockaddr *addr, socklen_t *addrlen, + int reserve, volatile int *counter) +{ + int ret; + + if (getdtablecount() + reserve + + ((*counter + 1) * GOTD_FD_NEEDED) >= getdtablesize()) { + log_debug("inflight fds exceeded"); + errno = EMFILE; + return -1; + } + + if ((ret = accept4(fd, addr, addrlen, + SOCK_NONBLOCK | SOCK_CLOEXEC)) > -1) { + (*counter)++; + } + + return ret; +} + +static uint64_t +client_hash(uint32_t client_id) +{ + return SipHash24(&clients_hash_key, &client_id, sizeof(client_id)); +} + +static void +add_client(struct gotd_client *client) +{ + uint64_t slot = client_hash(client->id) % nitems(gotd_clients); + STAILQ_INSERT_HEAD(&gotd_clients[slot], client, entry); + client_cnt++; +} + +static struct gotd_client * +find_client(uint32_t client_id) +{ + uint64_t slot; + struct gotd_client *c; + + slot = client_hash(client_id) % nitems(gotd_clients); + STAILQ_FOREACH(c, &gotd_clients[slot], entry) { + if (c->id == client_id) + return c; + } + + return NULL; +} + +static uint32_t +get_client_id(void) +{ + int duplicate = 0; + uint32_t id; + + do { + id = arc4random(); + duplicate = (find_client(id) != NULL); + } while (duplicate || id == 0); + + return id; +} + +static struct gotd_child_proc * +get_client_proc(struct gotd_client *client) +{ + if (client->repo_read && client->repo_write) { + fatalx("uid %d is reading and writing in the same session", + client->euid); + /* NOTREACHED */ + } + + if (client->repo_read) + return client->repo_read; + else if (client->repo_write) + return client->repo_write; + else { + fatal("uid %d is neither reading nor writing", client->euid); + /* NOTREACHED */ + } + return NULL; +} + +static int +client_is_reading(struct gotd_client *client) +{ + return client->repo_read != NULL; +} + +static int +client_is_writing(struct gotd_client *client) +{ + return client->repo_write != NULL; +} + +static const struct got_error * +ensure_client_is_reading(struct gotd_client *client) +{ + if (!client_is_reading(client)) { + return got_error_fmt(GOT_ERR_BAD_PACKET, + "uid %d made a read-request but is not reading from " + "a repository", client->euid); + } + + return NULL; +} + +static const struct got_error * +ensure_client_is_writing(struct gotd_client *client) +{ + if (!client_is_writing(client)) { + return got_error_fmt(GOT_ERR_BAD_PACKET, + "uid %d made a write-request but is not writing to " + "a repository", client->euid); + } + + return NULL; +} + +static const struct got_error * +ensure_client_is_not_writing(struct gotd_client *client) +{ + if (client_is_writing(client)) { + return got_error_fmt(GOT_ERR_BAD_PACKET, + "uid %d made a read-request but is writing to " + "a repository", client->euid); + } + + return NULL; +} + +static const struct got_error * +ensure_client_is_not_reading(struct gotd_client *client) +{ + if (client_is_reading(client)) { + return got_error_fmt(GOT_ERR_BAD_PACKET, + "uid %d made a write-request but is reading from " + "a repository", client->euid); + } + + return NULL; +} + +static void +disconnect(struct gotd_client *client) +{ + struct gotd_imsg_disconnect idisconnect; + struct gotd_child_proc *proc = get_client_proc(client); + uint64_t slot; + + log_debug("uid %d: disconnecting", client->euid); + + idisconnect.client_id = client->id; + if (gotd_imsg_compose_event(&proc->iev, + GOTD_IMSG_DISCONNECT, PROC_GOTD, -1, + &idisconnect, sizeof(idisconnect)) == -1) + log_warn("imsg compose DISCONNECT"); + + slot = client_hash(client->id) % nitems(gotd_clients); + STAILQ_REMOVE(&gotd_clients[slot], client, gotd_client, entry); + imsg_clear(&client->iev.ibuf); + event_del(&client->iev.ev); + evtimer_del(&client->tmo); + close(client->fd); + if (client->delta_cache_fd != -1) + close(client->delta_cache_fd); + if (client->packfile_path) { + if (unlink(client->packfile_path) == -1 && errno != ENOENT) + log_warn("unlink %s: ", client->packfile_path); + free(client->packfile_path); + } + if (client->packidx_path) { + if (unlink(client->packidx_path) == -1 && errno != ENOENT) + log_warn("unlink %s: ", client->packidx_path); + free(client->packidx_path); + } + free(client->capabilities); + free(client); + inflight--; + client_cnt--; +} + +static void +disconnect_on_error(struct gotd_client *client, const struct got_error *err) +{ + struct imsgbuf ibuf; + + log_warnx("uid %d: %s", client->euid, err->msg); + if (err->code != GOT_ERR_EOF) { + imsg_init(&ibuf, client->fd); + gotd_imsg_send_error(&ibuf, 0, PROC_GOTD, err); + imsg_clear(&ibuf); + } + disconnect(client); +} + +static struct gotd_child_proc * +find_proc_by_repo_name(enum gotd_procid proc_id, const char *repo_name) +{ + struct gotd_child_proc *proc; + int i; + size_t namelen; + + for (i = 0; i < gotd.nprocs; i++) { + proc = &gotd.procs[i]; + if (proc->type != proc_id) + continue; + namelen = strlen(proc->repo_name); + if (strncmp(proc->repo_name, repo_name, namelen) != 0) + continue; + if (repo_name[namelen] == '\0' || + strcmp(&repo_name[namelen], ".git") == 0) + return proc; + } + + return NULL; +} + +static struct gotd_child_proc * +find_proc_by_fd(int fd) +{ + struct gotd_child_proc *proc; + int i; + + for (i = 0; i < gotd.nprocs; i++) { + proc = &gotd.procs[i]; + if (proc->iev.ibuf.fd == fd) + return proc; + } + + return NULL; +} + +static const struct got_error * +forward_list_refs_request(struct gotd_client *client, struct imsg *imsg) +{ + const struct got_error *err; + struct gotd_imsg_list_refs ireq; + struct gotd_imsg_list_refs_internal ilref; + struct gotd_child_proc *proc = NULL; + size_t datalen; + int fd = -1; + + log_debug("list-refs request from uid %d", client->euid); + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(ireq)) + return got_error(GOT_ERR_PRIVSEP_LEN); + + memcpy(&ireq, imsg->data, datalen); + + memset(&ilref, 0, sizeof(ilref)); + ilref.client_id = client->id; + + if (ireq.client_is_reading) { + err = ensure_client_is_not_writing(client); + if (err) + return err; + client->repo_read = find_proc_by_repo_name(PROC_REPO_READ, + ireq.repo_name); + if (client->repo_read == NULL) + return got_error(GOT_ERR_NOT_GIT_REPO); + } else { + err = ensure_client_is_not_reading(client); + if (err) + return err; + client->repo_write = find_proc_by_repo_name(PROC_REPO_WRITE, + ireq.repo_name); + if (client->repo_write == NULL) + return got_error(GOT_ERR_NOT_GIT_REPO); + } + + fd = dup(client->fd); + if (fd == -1) + return got_error_from_errno("dup"); + + proc = get_client_proc(client); + if (gotd_imsg_compose_event(&proc->iev, + GOTD_IMSG_LIST_REFS_INTERNAL, PROC_GOTD, fd, + &ilref, sizeof(ilref)) == -1) { + err = got_error_from_errno("imsg compose WANT"); + close(fd); + return err; + } + + return NULL; +} + +static const struct got_error * +forward_want(struct gotd_client *client, struct imsg *imsg) +{ + struct gotd_imsg_want ireq; + struct gotd_imsg_want iwant; + size_t datalen; + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(ireq)) + return got_error(GOT_ERR_PRIVSEP_LEN); + + memcpy(&ireq, imsg->data, datalen); + + memset(&iwant, 0, sizeof(iwant)); + memcpy(iwant.object_id, ireq.object_id, SHA1_DIGEST_LENGTH); + iwant.client_id = client->id; + + if (gotd_imsg_compose_event(&client->repo_read->iev, GOTD_IMSG_WANT, + PROC_GOTD, -1, &iwant, sizeof(iwant)) == -1) + return got_error_from_errno("imsg compose WANT"); + + return NULL; +} + +static const struct got_error * +forward_ref_update(struct gotd_client *client, struct imsg *imsg) +{ + const struct got_error *err = NULL; + struct gotd_imsg_ref_update ireq; + struct gotd_imsg_ref_update *iref = NULL; + size_t datalen; + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen < sizeof(ireq)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&ireq, imsg->data, sizeof(ireq)); + if (datalen != sizeof(ireq) + ireq.name_len) + return got_error(GOT_ERR_PRIVSEP_LEN); + + iref = malloc(datalen); + if (iref == NULL) + return got_error_from_errno("malloc"); + memcpy(iref, imsg->data, datalen); + + iref->client_id = client->id; + if (gotd_imsg_compose_event(&client->repo_write->iev, + GOTD_IMSG_REF_UPDATE, PROC_GOTD, -1, iref, datalen) == -1) + err = got_error_from_errno("imsg compose REF_UPDATE"); + free(iref); + return err; +} + +static const struct got_error * +forward_have(struct gotd_client *client, struct imsg *imsg) +{ + struct gotd_imsg_have ireq; + struct gotd_imsg_have ihave; + size_t datalen; + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(ireq)) + return got_error(GOT_ERR_PRIVSEP_LEN); + + memcpy(&ireq, imsg->data, datalen); + + memset(&ihave, 0, sizeof(ihave)); + memcpy(ihave.object_id, ireq.object_id, SHA1_DIGEST_LENGTH); + ihave.client_id = client->id; + + if (gotd_imsg_compose_event(&client->repo_read->iev, GOTD_IMSG_HAVE, + PROC_GOTD, -1, &ihave, sizeof(ihave)) == -1) + return got_error_from_errno("imsg compose HAVE"); + + return NULL; +} + +static int +client_has_capability(struct gotd_client *client, const char *capastr) +{ + struct gotd_client_capability *capa; + size_t i; + + if (client->ncapabilities == 0) + return 0; + + for (i = 0; i < client->ncapabilities; i++) { + capa = &client->capabilities[i]; + if (strcmp(capa->key, capastr) == 0) + return 1; + } + + return 0; +} + +static const struct got_error * +recv_capabilities(struct gotd_client *client, struct imsg *imsg) +{ + struct gotd_imsg_capabilities icapas; + size_t datalen; + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(icapas)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&icapas, imsg->data, sizeof(icapas)); + + client->ncapa_alloc = icapas.ncapabilities; + client->capabilities = calloc(client->ncapa_alloc, + sizeof(*client->capabilities)); + if (client->capabilities == NULL) { + client->ncapa_alloc = 0; + return got_error_from_errno("calloc"); + } + + log_debug("expecting %zu capabilities from uid %d", + client->ncapa_alloc, client->euid); + return NULL; +} + +static const struct got_error * +recv_capability(struct gotd_client *client, struct imsg *imsg) +{ + struct gotd_imsg_capability icapa; + struct gotd_client_capability *capa; + size_t datalen; + char *key, *value = NULL; + + if (client->capabilities == NULL || + client->ncapabilities >= client->ncapa_alloc) { + return got_error_msg(GOT_ERR_BAD_REQUEST, + "unexpected capability received"); + } + + memset(&icapa, 0, sizeof(icapa)); + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen < sizeof(icapa)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&icapa, imsg->data, sizeof(icapa)); + + if (datalen != sizeof(icapa) + icapa.key_len + icapa.value_len) + return got_error(GOT_ERR_PRIVSEP_LEN); + + key = malloc(icapa.key_len + 1); + if (key == NULL) + return got_error_from_errno("malloc"); + if (icapa.value_len > 0) { + value = malloc(icapa.value_len + 1); + if (value == NULL) { + free(key); + return got_error_from_errno("malloc"); + } + } + + memcpy(key, imsg->data + sizeof(icapa), icapa.key_len); + key[icapa.key_len] = '\0'; + if (value) { + memcpy(value, imsg->data + sizeof(icapa) + icapa.key_len, + icapa.value_len); + value[icapa.value_len] = '\0'; + } + + capa = &client->capabilities[client->ncapabilities++]; + capa->key = key; + capa->value = value; + + if (value) + log_debug("uid %d: capability %s=%s", client->euid, key, value); + else + log_debug("uid %d: capability %s", client->euid, key); + + return NULL; +} + +static const struct got_error * +send_packfile(struct gotd_client *client) +{ + const struct got_error *err = NULL; + struct gotd_imsg_send_packfile ipack; + struct gotd_imsg_packfile_pipe ipipe; + int pipe[2]; + + if (socketpair(AF_UNIX, SOCK_STREAM, PF_UNSPEC, pipe) == -1) + return got_error_from_errno("socketpair"); + + memset(&ipack, 0, sizeof(ipack)); + memset(&ipipe, 0, sizeof(ipipe)); + + ipack.client_id = client->id; + if (client_has_capability(client, GOT_CAPA_SIDE_BAND_64K)) + ipack.report_progress = 1; + + client->delta_cache_fd = got_opentempfd(); + if (client->delta_cache_fd == -1) + return got_error_from_errno("got_opentempfd"); + + if (gotd_imsg_compose_event(&client->repo_read->iev, + GOTD_IMSG_SEND_PACKFILE, PROC_GOTD, client->delta_cache_fd, + &ipack, sizeof(ipack)) == -1) { + err = got_error_from_errno("imsg compose SEND_PACKFILE"); + close(pipe[0]); + close(pipe[1]); + return err; + } + + ipipe.client_id = client->id; + + if (gotd_imsg_compose_event(&client->repo_read->iev, + GOTD_IMSG_PACKFILE_PIPE, PROC_GOTD, pipe[0], + &ipipe, sizeof(ipipe)) == -1) { + err = got_error_from_errno("imsg compose PACKFILE_PIPE"); + close(pipe[1]); + return err; + } + + if (gotd_imsg_compose_event(&client->repo_read->iev, + GOTD_IMSG_PACKFILE_PIPE, PROC_GOTD, pipe[1], + &ipipe, sizeof(ipipe)) == -1) + err = got_error_from_errno("imsg compose PACKFILE_PIPE"); + + return err; +} + +static const struct got_error * +recv_packfile(struct gotd_client *client) +{ + const struct got_error *err = NULL; + struct gotd_imsg_recv_packfile ipack; + struct gotd_imsg_packfile_pipe ipipe; + struct gotd_imsg_packidx_file ifile; + char *basepath = NULL, *pack_path = NULL, *idx_path = NULL; + int packfd = -1, idxfd = -1; + int pipe[2] = { -1, -1 }; + + if (client->packfile_path) { + return got_error_fmt(GOT_ERR_PRIVSEP_MSG, + "uid %d already has a pack file", client->euid); + } + + if (socketpair(AF_UNIX, SOCK_STREAM, PF_UNSPEC, pipe) == -1) + return got_error_from_errno("socketpair"); + + memset(&ipipe, 0, sizeof(ipipe)); + ipipe.client_id = client->id; + + if (gotd_imsg_compose_event(&client->repo_write->iev, + GOTD_IMSG_PACKFILE_PIPE, PROC_GOTD, pipe[0], + &ipipe, sizeof(ipipe)) == -1) { + err = got_error_from_errno("imsg compose PACKFILE_PIPE"); + pipe[0] = -1; + goto done; + } + pipe[0] = -1; + + if (gotd_imsg_compose_event(&client->repo_write->iev, + GOTD_IMSG_PACKFILE_PIPE, PROC_GOTD, pipe[1], + &ipipe, sizeof(ipipe)) == -1) + err = got_error_from_errno("imsg compose PACKFILE_PIPE"); + pipe[1] = -1; + + if (asprintf(&basepath, "%s/%s/receiving-from-uid-%d.pack", + client->repo_write->chroot_path, GOT_OBJECTS_PACK_DIR, + client->euid) == -1) { + err = got_error_from_errno("asprintf"); + goto done; + } + + err = got_opentemp_named_fd(&pack_path, &packfd, basepath); + if (err) + goto done; + + free(basepath); + if (asprintf(&basepath, "%s/%s/receiving-from-uid-%d.idx", + client->repo_write->chroot_path, GOT_OBJECTS_PACK_DIR, + client->euid) == -1) { + err = got_error_from_errno("asprintf"); + basepath = NULL; + goto done; + } + err = got_opentemp_named_fd(&idx_path, &idxfd, basepath); + if (err) + goto done; + + memset(&ifile, 0, sizeof(ifile)); + ifile.client_id = client->id; + if (gotd_imsg_compose_event(&client->repo_write->iev, + GOTD_IMSG_PACKIDX_FILE, PROC_GOTD, idxfd, + &ifile, sizeof(ifile)) == -1) { + err = got_error_from_errno("imsg compose PACKIDX_FILE"); + idxfd = -1; + goto done; + } + idxfd = -1; + + memset(&ipack, 0, sizeof(ipack)); + ipack.client_id = client->id; + if (client_has_capability(client, GOT_CAPA_REPORT_STATUS)) + ipack.report_status = 1; + + if (gotd_imsg_compose_event(&client->repo_write->iev, + GOTD_IMSG_RECV_PACKFILE, PROC_GOTD, packfd, + &ipack, sizeof(ipack)) == -1) { + err = got_error_from_errno("imsg compose RECV_PACKFILE"); + packfd = -1; + goto done; + } + packfd = -1; + +done: + free(basepath); + if (pipe[0] != -1 && close(pipe[0]) == -1 && err == NULL) + err = got_error_from_errno("close"); + if (pipe[1] != -1 && close(pipe[1]) == -1 && err == NULL) + err = got_error_from_errno("close"); + if (packfd != -1 && close(packfd) == -1 && err == NULL) + err = got_error_from_errno("close"); + if (idxfd != -1 && close(idxfd) == -1 && err == NULL) + err = got_error_from_errno("close"); + if (err) { + free(pack_path); + free(idx_path); + } else { + client->packfile_path = pack_path; + client->packidx_path = idx_path; + } + return err; +} + +static void +gotd_request(int fd, short events, void *arg) +{ + struct gotd_imsgev *iev = arg; + struct imsgbuf *ibuf = &iev->ibuf; + struct gotd_client *client = iev->handler_arg; + const struct got_error *err = NULL; + struct imsg imsg; + ssize_t n; + + if (events & EV_WRITE) { + while (ibuf->w.queued) { + n = msgbuf_write(&ibuf->w); + if (n == -1 && errno == EPIPE) { + /* + * The client has closed its socket. + * This can happen when Git clients are + * done sending pack file data. + */ + msgbuf_clear(&ibuf->w); + continue; + } else if (n == -1 && errno != EAGAIN) { + err = got_error_from_errno("imsg_flush"); + disconnect_on_error(client, err); + return; + } + if (n == 0) { + /* Connection closed. */ + err = got_error(GOT_ERR_EOF); + disconnect_on_error(client, err); + return; + } + } + } + + if ((events & EV_READ) == 0) + return; + + memset(&imsg, 0, sizeof(imsg)); + + while (err == NULL) { + err = gotd_imsg_recv(&imsg, ibuf, 0); + if (err) { + if (err->code == GOT_ERR_PRIVSEP_READ) + err = NULL; + break; + } + + evtimer_del(&client->tmo); + + switch (imsg.hdr.type) { + case GOTD_IMSG_LIST_REFS: + if (client->state != GOTD_STATE_EXPECT_LIST_REFS) { + err = got_error_msg(GOT_ERR_BAD_REQUEST, + "unexpected list-refs request received"); + break; + } + err = forward_list_refs_request(client, &imsg); + if (err) + break; + client->state = GOTD_STATE_EXPECT_CAPABILITIES; + log_debug("uid %d: expecting capabilities", + client->euid); + break; + case GOTD_IMSG_CAPABILITIES: + if (client->state != GOTD_STATE_EXPECT_CAPABILITIES) { + err = got_error_msg(GOT_ERR_BAD_REQUEST, + "unexpected capabilities received"); + break; + } + log_debug("receiving capabilities from uid %d", + client->euid); + err = recv_capabilities(client, &imsg); + break; + case GOTD_IMSG_CAPABILITY: + if (client->state != GOTD_STATE_EXPECT_CAPABILITIES) { + err = got_error_msg(GOT_ERR_BAD_REQUEST, + "unexpected capability received"); + break; + } + err = recv_capability(client, &imsg); + if (err || client->ncapabilities < client->ncapa_alloc) + break; + if (client_is_reading(client)) { + client->state = GOTD_STATE_EXPECT_WANT; + log_debug("uid %d: expecting want-lines", + client->euid); + } else if (client_is_writing(client)) { + client->state = GOTD_STATE_EXPECT_REF_UPDATE; + log_debug("uid %d: expecting ref-update-lines", + client->euid); + } else + fatalx("client %d is both reading and writing", + client->euid); + break; + case GOTD_IMSG_WANT: + if (client->state != GOTD_STATE_EXPECT_WANT) { + err = got_error_msg(GOT_ERR_BAD_REQUEST, + "unexpected want-line received"); + break; + } + log_debug("received want-line from uid %d", + client->euid); + err = ensure_client_is_reading(client); + if (err) + break; + err = forward_want(client, &imsg); + break; + case GOTD_IMSG_REF_UPDATE: + if (client->state != GOTD_STATE_EXPECT_REF_UPDATE) { + err = got_error_msg(GOT_ERR_BAD_REQUEST, + "unexpected ref-update-line received"); + break; + } + log_debug("received ref-update-line from uid %d", + client->euid); + err = ensure_client_is_writing(client); + if (err) + break; + err = forward_ref_update(client, &imsg); + if (err) + break; + client->state = GOTD_STATE_EXPECT_MORE_REF_UPDATES; + break; + case GOTD_IMSG_HAVE: + if (client->state != GOTD_STATE_EXPECT_HAVE) { + err = got_error_msg(GOT_ERR_BAD_REQUEST, + "unexpected have-line received"); + break; + } + log_debug("received have-line from uid %d", + client->euid); + err = ensure_client_is_reading(client); + if (err) + break; + err = forward_have(client, &imsg); + if (err) + break; + break; + case GOTD_IMSG_FLUSH: + if (client->state == GOTD_STATE_EXPECT_WANT || + client->state == GOTD_STATE_EXPECT_HAVE) { + err = ensure_client_is_reading(client); + if (err) + break; + } else if (client->state == + GOTD_STATE_EXPECT_MORE_REF_UPDATES) { + err = ensure_client_is_writing(client); + if (err) + break; + } else { + err = got_error_msg(GOT_ERR_BAD_REQUEST, + "unexpected flush-pkt received"); + break; + } + log_debug("received flush-pkt from uid %d", + client->euid); + if (client->state == GOTD_STATE_EXPECT_WANT) { + client->state = GOTD_STATE_EXPECT_HAVE; + log_debug("uid %d: expecting have-lines", + client->euid); + } else if (client->state == GOTD_STATE_EXPECT_HAVE) { + client->state = GOTD_STATE_EXPECT_DONE; + log_debug("uid %d: expecting 'done'", + client->euid); + } else if (client->state == + GOTD_STATE_EXPECT_MORE_REF_UPDATES) { + client->state = GOTD_STATE_EXPECT_PACKFILE; + log_debug("uid %d: expecting packfile", + client->euid); + err = recv_packfile(client); + } else { + /* should not happen, see above */ + err = got_error_msg(GOT_ERR_BAD_REQUEST, + "unexpected client state"); + break; + } + break; + case GOTD_IMSG_DONE: + if (client->state != GOTD_STATE_EXPECT_HAVE && + client->state != GOTD_STATE_EXPECT_DONE) { + err = got_error_msg(GOT_ERR_BAD_REQUEST, + "unexpected flush-pkt received"); + break; + } + log_debug("received 'done' from uid %d", client->euid); + err = ensure_client_is_reading(client); + if (err) + break; + client->state = GOTD_STATE_DONE; + err = send_packfile(client); + break; + default: + err = got_error(GOT_ERR_PRIVSEP_MSG); + break; + } + + imsg_free(&imsg); + } + + if (err) { + if (err->code != GOT_ERR_EOF || + client->state != GOTD_STATE_EXPECT_PACKFILE) + disconnect_on_error(client, err); + } else { + gotd_imsg_event_add(&client->iev); + evtimer_add(&client->tmo, &timeout); + } +} + +static void +gotd_request_timeout(int fd, short events, void *arg) +{ + struct gotd_client *client = arg; + + log_debug("disconnecting uid %d due to timeout", client->euid); + disconnect(client); +} + +static void +gotd_accept(int fd, short event, void *arg) +{ + struct sockaddr_storage ss; + struct timeval backoff; + socklen_t len; + int s = -1; + struct gotd_client *client = NULL; + uid_t euid; + gid_t egid; + + backoff.tv_sec = 1; + backoff.tv_usec = 0; + + if (event_add(&gotd.ev, NULL) == -1) { + log_warn("event_add"); + return; + } + if (event & EV_TIMEOUT) + return; + + len = sizeof(ss); + + s = accept_reserve(fd, (struct sockaddr *)&ss, &len, GOTD_FD_RESERVE, + &inflight); + + if (s == -1) { + switch (errno) { + case EINTR: + case EWOULDBLOCK: + case ECONNABORTED: + return; + case EMFILE: + case ENFILE: + event_del(&gotd.ev); + evtimer_add(&gotd.pause, &backoff); + return; + default: + log_warn("%s: accept", __func__); + return; + } + } + + if (client_cnt >= GOTD_MAXCLIENTS) + goto err; + + if (getpeereid(s, &euid, &egid) == -1) { + log_warn("%s: getpeereid", __func__); + goto err; + } + + client = calloc(1, sizeof(*client)); + if (client == NULL) { + log_warn("%s: calloc", __func__); + goto err; + } + + client->state = GOTD_STATE_EXPECT_LIST_REFS; + client->id = get_client_id(); + client->fd = s; + s = -1; + client->delta_cache_fd = -1; + client->euid = euid; + client->egid = egid; + client->nref_updates = -1; + + imsg_init(&client->iev.ibuf, client->fd); + client->iev.handler = gotd_request; + client->iev.events = EV_READ; + client->iev.handler_arg = client; + + event_set(&client->iev.ev, client->fd, EV_READ, gotd_request, + &client->iev); + gotd_imsg_event_add(&client->iev); + + evtimer_set(&client->tmo, gotd_request_timeout, client); + + add_client(client); + log_debug("%s: new client uid %d connected on fd %d", __func__, + client->euid, client->fd); + return; +err: + inflight--; + if (s != -1) + close(s); + free(client); +} + +static void +gotd_accept_paused(int fd, short event, void *arg) +{ + event_add(&gotd.ev, NULL); +} + +static const char *gotd_proc_names[PROC_MAX] = { + "parent", + "repo_read", + "repo_write" +}; + +static struct gotd_child_proc * +get_proc_for_pid(pid_t pid) +{ + struct gotd_child_proc *proc; + int i; + + for (i = 0; i < gotd.nprocs; i++) { + proc = &gotd.procs[i]; + if (proc->pid == pid) + return proc; + } + + return NULL; +} + +static void +kill_proc(struct gotd_child_proc *proc, int fatal) +{ + if (fatal) { + log_warnx("sending SIGKILL to PID %d", proc->pid); + kill(proc->pid, SIGKILL); + } else + kill(proc->pid, SIGTERM); +} + +static void +gotd_shutdown(void) +{ + pid_t pid; + int status, i; + struct gotd_child_proc *proc; + + for (i = 0; i < gotd.nprocs; i++) { + proc = &gotd.procs[i]; + msgbuf_clear(&proc->iev.ibuf.w); + close(proc->iev.ibuf.fd); + kill_proc(proc, 0); + } + + log_debug("waiting for children to terminate"); + do { + pid = wait(&status); + if (pid == -1) { + if (errno != EINTR && errno != ECHILD) + fatal("wait"); + } else if (WIFSIGNALED(status)) { + proc = get_proc_for_pid(pid); + log_warnx("%s %s child process terminated; signal %d", + proc ? gotd_proc_names[proc->type] : "", + proc ? proc->chroot_path : "", WTERMSIG(status)); + } + } while (pid != -1 || (pid == -1 && errno == EINTR)); + + log_info("terminating"); + exit(0); +} + +void +gotd_sighdlr(int sig, short event, void *arg) +{ + /* + * Normal signal handler rules don't apply because libevent + * decouples for us. + */ + + switch (sig) { + case SIGHUP: + log_info("%s: ignoring SIGHUP", __func__); + break; + case SIGUSR1: + log_info("%s: ignoring SIGUSR1", __func__); + break; + case SIGTERM: + case SIGINT: + gotd_shutdown(); + log_warnx("gotd terminating"); + exit(0); + break; + default: + fatalx("unexpected signal"); + } +} + +static const struct got_error * +ensure_proc_is_reading(struct gotd_client *client, + struct gotd_child_proc *proc) +{ + if (!client_is_reading(client)) { + kill_proc(proc, 1); + return got_error_fmt(GOT_ERR_BAD_PACKET, + "PID %d handled a read-request for uid %d but this " + "user is not reading from a repository", proc->pid, + client->euid); + } + + return NULL; +} + +static const struct got_error * +ensure_proc_is_writing(struct gotd_client *client, + struct gotd_child_proc *proc) +{ + if (!client_is_writing(client)) { + kill_proc(proc, 1); + return got_error_fmt(GOT_ERR_BAD_PACKET, + "PID %d handled a write-request for uid %d but this " + "user is not writing to a repository", proc->pid, + client->euid); + } + + return NULL; +} + +static int +verify_imsg_src(struct gotd_client *client, struct gotd_child_proc *proc, + struct imsg *imsg) +{ + const struct got_error *err; + struct gotd_child_proc *client_proc; + int ret = 0; + + client_proc = get_client_proc(client); + if (proc->pid != client_proc->pid) { + kill_proc(proc, 1); + log_warnx("received message from PID %d for uid %d, while " + "PID %d is the process serving this user", + proc->pid, client->euid, client_proc->pid); + return 0; + } + + switch (imsg->hdr.type) { + case GOTD_IMSG_ERROR: + ret = 1; + break; + case GOTD_IMSG_PACKFILE_DONE: + err = ensure_proc_is_reading(client, proc); + if (err) + log_warnx("uid %d: %s", client->euid, err->msg); + else + ret = 1; + break; + case GOTD_IMSG_PACKFILE_INSTALL: + case GOTD_IMSG_REF_UPDATES_START: + case GOTD_IMSG_REF_UPDATE: + err = ensure_proc_is_writing(client, proc); + if (err) + log_warnx("uid %d: %s", client->euid, err->msg); + else + ret = 1; + break; + default: + log_debug("%s: unexpected imsg %d", __func__, imsg->hdr.type); + break; + } + + return ret; +} + +static const struct got_error * +recv_packfile_done(uint32_t *client_id, struct imsg *imsg) +{ + struct gotd_imsg_packfile_done idone; + size_t datalen; + + log_debug("packfile-done received"); + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(idone)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&idone, imsg->data, sizeof(idone)); + + *client_id = idone.client_id; + return NULL; +} + +static const struct got_error * +recv_packfile_install(uint32_t *client_id, struct imsg *imsg) +{ + struct gotd_imsg_packfile_install inst; + size_t datalen; + + log_debug("packfile-install received"); + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(inst)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&inst, imsg->data, sizeof(inst)); + + *client_id = inst.client_id; + return NULL; +} + +static const struct got_error * +recv_ref_updates_start(uint32_t *client_id, struct imsg *imsg) +{ + struct gotd_imsg_ref_updates_start istart; + size_t datalen; + + log_debug("ref-updates-start received"); + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(istart)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&istart, imsg->data, sizeof(istart)); + + *client_id = istart.client_id; + return NULL; +} + +static const struct got_error * +recv_ref_update(uint32_t *client_id, struct imsg *imsg) +{ + struct gotd_imsg_ref_update iref; + size_t datalen; + + log_debug("ref-update received"); + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen < sizeof(iref)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&iref, imsg->data, sizeof(iref)); + + *client_id = iref.client_id; + return NULL; +} + +static const struct got_error * +send_ref_update_ok(struct gotd_client *client, + struct gotd_imsg_ref_update *iref, const char *refname) +{ + struct gotd_imsg_ref_update_ok iok; + struct ibuf *wbuf; + size_t len; + + memset(&iok, 0, sizeof(iok)); + iok.client_id = client->id; + memcpy(iok.old_id, iref->old_id, SHA1_DIGEST_LENGTH); + memcpy(iok.new_id, iref->new_id, SHA1_DIGEST_LENGTH); + iok.name_len = strlen(refname); + + len = sizeof(iok) + iok.name_len; + wbuf = imsg_create(&client->iev.ibuf, GOTD_IMSG_REF_UPDATE_OK, + PROC_GOTD, gotd.pid, len); + if (wbuf == NULL) + return got_error_from_errno("imsg_create REF_UPDATE_OK"); + + if (imsg_add(wbuf, &iok, sizeof(iok)) == -1) + return got_error_from_errno("imsg_add REF_UPDATE_OK"); + if (imsg_add(wbuf, refname, iok.name_len) == -1) + return got_error_from_errno("imsg_add REF_UPDATE_OK"); + + wbuf->fd = -1; + imsg_close(&client->iev.ibuf, wbuf); + gotd_imsg_event_add(&client->iev); + return NULL; +} + +static void +send_refs_updated(struct gotd_client *client) +{ + if (gotd_imsg_compose_event(&client->iev, + GOTD_IMSG_REFS_UPDATED, PROC_GOTD, -1, NULL, 0) == -1) + log_warn("imsg compose REFS_UPDATED"); +} + +static const struct got_error * +send_ref_update_ng(struct gotd_client *client, + struct gotd_imsg_ref_update *iref, const char *refname, + const char *reason) +{ + const struct got_error *ng_err; + struct gotd_imsg_ref_update_ng ing; + struct ibuf *wbuf; + size_t len; + + memset(&ing, 0, sizeof(ing)); + ing.client_id = client->id; + memcpy(ing.old_id, iref->old_id, SHA1_DIGEST_LENGTH); + memcpy(ing.new_id, iref->new_id, SHA1_DIGEST_LENGTH); + ing.name_len = strlen(refname); + + ng_err = got_error_fmt(GOT_ERR_REF_BUSY, "%s", reason); + ing.reason_len = strlen(ng_err->msg); + + len = sizeof(ing) + ing.name_len + ing.reason_len; + wbuf = imsg_create(&client->iev.ibuf, GOTD_IMSG_REF_UPDATE_NG, + PROC_GOTD, gotd.pid, len); + if (wbuf == NULL) + return got_error_from_errno("imsg_create REF_UPDATE_NG"); + + if (imsg_add(wbuf, &ing, sizeof(ing)) == -1) + return got_error_from_errno("imsg_add REF_UPDATE_NG"); + if (imsg_add(wbuf, refname, ing.name_len) == -1) + return got_error_from_errno("imsg_add REF_UPDATE_NG"); + if (imsg_add(wbuf, ng_err->msg, ing.reason_len) == -1) + return got_error_from_errno("imsg_add REF_UPDATE_NG"); + + wbuf->fd = -1; + imsg_close(&client->iev.ibuf, wbuf); + gotd_imsg_event_add(&client->iev); + return NULL; +} + +static const struct got_error * +install_pack(struct gotd_client *client, const char *repo_path, + struct imsg *imsg) +{ + const struct got_error *err = NULL; + struct gotd_imsg_packfile_install inst; + char hex[SHA1_DIGEST_STRING_LENGTH]; + size_t datalen; + char *packfile_path = NULL, *packidx_path = NULL; + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(inst)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&inst, imsg->data, sizeof(inst)); + + if (client->packfile_path == NULL) + return got_error_msg(GOT_ERR_BAD_REQUEST, + "client has no pack file"); + if (client->packidx_path == NULL) + return got_error_msg(GOT_ERR_BAD_REQUEST, + "client has no pack file index"); + + if (got_sha1_digest_to_str(inst.pack_sha1, hex, sizeof(hex)) == NULL) + return got_error_msg(GOT_ERR_NO_SPACE, + "could not convert pack file SHA1 to hex"); + + if (asprintf(&packfile_path, "/%s/%s/pack-%s.pack", + repo_path, GOT_OBJECTS_PACK_DIR, hex) == -1) { + err = got_error_from_errno("asprintf"); + goto done; + } + + if (asprintf(&packidx_path, "/%s/%s/pack-%s.idx", + repo_path, GOT_OBJECTS_PACK_DIR, hex) == -1) { + err = got_error_from_errno("asprintf"); + goto done; + } + + if (rename(client->packfile_path, packfile_path) == -1) { + err = got_error_from_errno3("rename", client->packfile_path, + packfile_path); + goto done; + } + + free(client->packfile_path); + client->packfile_path = NULL; + + if (rename(client->packidx_path, packidx_path) == -1) { + err = got_error_from_errno3("rename", client->packidx_path, + packidx_path); + goto done; + } + + free(client->packidx_path); + client->packidx_path = NULL; +done: + free(packfile_path); + free(packidx_path); + return err; +} + +static const struct got_error * +begin_ref_updates(struct gotd_client *client, struct imsg *imsg) +{ + struct gotd_imsg_ref_updates_start istart; + size_t datalen; + + if (client->nref_updates != -1) + return got_error(GOT_ERR_PRIVSEP_MSG); + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(istart)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&istart, imsg->data, sizeof(istart)); + + if (istart.nref_updates <= 0) + return got_error(GOT_ERR_PRIVSEP_MSG); + + client->nref_updates = istart.nref_updates; + return NULL; +} + +static const struct got_error * +update_ref(struct gotd_client *client, const char *repo_path, + struct imsg *imsg) +{ + const struct got_error *err = NULL; + struct got_repository *repo = NULL; + struct got_reference *ref = NULL; + struct gotd_imsg_ref_update iref; + struct got_object_id old_id, new_id; + struct got_object_id *id = NULL; + struct got_object *obj = NULL; + char *refname = NULL; + size_t datalen; + int locked = 0; + + log_debug("update-ref from uid %d", client->euid); + + if (client->nref_updates <= 0) + return got_error(GOT_ERR_PRIVSEP_MSG); + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen < sizeof(iref)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&iref, imsg->data, sizeof(iref)); + if (datalen != sizeof(iref) + iref.name_len) + return got_error(GOT_ERR_PRIVSEP_LEN); + refname = malloc(iref.name_len + 1); + if (refname == NULL) + return got_error_from_errno("malloc"); + memcpy(refname, imsg->data + sizeof(iref), iref.name_len); + refname[iref.name_len] = '\0'; + + log_debug("updating ref %s for uid %d", refname, client->euid); + + err = got_repo_open(&repo, repo_path, NULL, NULL); + if (err) + goto done; + + memcpy(old_id.sha1, iref.old_id, SHA1_DIGEST_LENGTH); + memcpy(new_id.sha1, iref.new_id, SHA1_DIGEST_LENGTH); + err = got_object_open(&obj, repo, &new_id); + if (err) + goto done; + + if (iref.ref_is_new) { + err = got_ref_open(&ref, repo, refname, 0); + if (err) { + if (err->code != GOT_ERR_NOT_REF) + goto done; + err = got_ref_alloc(&ref, refname, &new_id); + if (err) + goto done; + err = got_ref_write(ref, repo); /* will lock/unlock */ + if (err) + goto done; + } else { + err = got_error_fmt(GOT_ERR_REF_BUSY, + "%s has been created by someone else " + "while transaction was in progress", + got_ref_get_name(ref)); + goto done; + } + } else { + err = got_ref_open(&ref, repo, refname, 1 /* lock */); + if (err) + goto done; + locked = 1; + + err = got_ref_resolve(&id, repo, ref); + if (err) + goto done; + + if (got_object_id_cmp(id, &old_id) != 0) { + err = got_error_fmt(GOT_ERR_REF_BUSY, + "%s has been modified by someone else " + "while transaction was in progress", + got_ref_get_name(ref)); + goto done; + } + + err = got_ref_change_ref(ref, &new_id); + if (err) + goto done; + + err = got_ref_write(ref, repo); + if (err) + goto done; + + free(id); + id = NULL; + } +done: + if (err) { + if (err->code == GOT_ERR_LOCKFILE_TIMEOUT) { + err = got_error_fmt(GOT_ERR_LOCKFILE_TIMEOUT, + "could not acquire exclusive file lock for %s", + refname); + } + send_ref_update_ng(client, &iref, refname, err->msg); + } else + send_ref_update_ok(client, &iref, refname); + + if (client->nref_updates > 0) { + client->nref_updates--; + if (client->nref_updates == 0) + send_refs_updated(client); + + } + if (locked) { + const struct got_error *unlock_err; + unlock_err = got_ref_unlock(ref); + if (unlock_err && err == NULL) + err = unlock_err; + } + if (ref) + got_ref_close(ref); + if (obj) + got_object_close(obj); + if (repo) + got_repo_close(repo); + free(refname); + free(id); + return err; +} + +static void +gotd_dispatch(int fd, short event, void *arg) +{ + struct gotd_imsgev *iev = arg; + struct imsgbuf *ibuf = &iev->ibuf; + struct gotd_child_proc *proc = NULL; + ssize_t n; + int shut = 0; + struct imsg imsg; + + if (event & EV_READ) { + if ((n = imsg_read(ibuf)) == -1 && errno != EAGAIN) + fatal("imsg_read error"); + if (n == 0) { + /* Connection closed. */ + shut = 1; + goto done; + } + } + + if (event & EV_WRITE) { + n = msgbuf_write(&ibuf->w); + if (n == -1 && errno != EAGAIN) + fatal("msgbuf_write"); + if (n == 0) { + /* Connection closed. */ + shut = 1; + goto done; + } + } + + proc = find_proc_by_fd(fd); + if (proc == NULL) + fatalx("cannot find child process for fd %d", fd); + + for (;;) { + const struct got_error *err = NULL; + struct gotd_client *client = NULL; + uint32_t client_id = 0; + int do_disconnect = 0; + int do_ref_updates = 0, do_ref_update = 0; + int do_packfile_install = 0; + + if ((n = imsg_get(ibuf, &imsg)) == -1) + fatal("%s: imsg_get error", __func__); + if (n == 0) /* No more messages. */ + break; + + switch (imsg.hdr.type) { + case GOTD_IMSG_ERROR: + do_disconnect = 1; + err = gotd_imsg_recv_error(&client_id, &imsg); + break; + case GOTD_IMSG_PACKFILE_DONE: + do_disconnect = 1; + err = recv_packfile_done(&client_id, &imsg); + break; + case GOTD_IMSG_PACKFILE_INSTALL: + err = recv_packfile_install(&client_id, &imsg); + if (err == NULL) + do_packfile_install = 1; + break; + case GOTD_IMSG_REF_UPDATES_START: + err = recv_ref_updates_start(&client_id, &imsg); + if (err == NULL) + do_ref_updates = 1; + break; + case GOTD_IMSG_REF_UPDATE: + err = recv_ref_update(&client_id, &imsg); + if (err == NULL) + do_ref_update = 1; + break; + default: + log_debug("unexpected imsg %d", imsg.hdr.type); + break; + } + + client = find_client(client_id); + if (client == NULL) { + log_warnx("%s: client not found", __func__); + imsg_free(&imsg); + continue; + } + + if (!verify_imsg_src(client, proc, &imsg)) { + log_debug("dropping imsg type %d from PID %d", + imsg.hdr.type, proc->pid); + imsg_free(&imsg); + continue; + } + if (err) + log_warnx("uid %d: %s", client->euid, err->msg); + + if (do_disconnect) { + if (err) + disconnect_on_error(client, err); + else + disconnect(client); + } else if (do_packfile_install) + err = install_pack(client, proc->chroot_path, &imsg); + else if (do_ref_updates) + err = begin_ref_updates(client, &imsg); + else if (do_ref_update) + err = update_ref(client, proc->chroot_path, &imsg); + + if (err) + log_warnx("uid %d: %s", client->euid, err->msg); + imsg_free(&imsg); + } +done: + if (!shut) { + gotd_imsg_event_add(iev); + } else { + /* This pipe is dead. Remove its event handler */ + event_del(&iev->ev); + event_loopexit(NULL); + } +} + +static pid_t +start_child(enum gotd_procid proc_id, const char *chroot_path, + char *argv0, int fd, int daemonize, int verbosity) +{ + char *argv[9]; + int argc = 0; + pid_t pid; + + switch (pid = fork()) { + case -1: + fatal("cannot fork"); + case 0: + break; + default: + close(fd); + return pid; + } + + if (fd != GOTD_SOCK_FILENO) { + if (dup2(fd, GOTD_SOCK_FILENO) == -1) + fatal("cannot setup imsg fd"); + } else if (fcntl(fd, F_SETFD, 0) == -1) + fatal("cannot setup imsg fd"); + + argv[argc++] = argv0; + switch (proc_id) { + case PROC_REPO_READ: + argv[argc++] = (char *)"-R"; + break; + case PROC_REPO_WRITE: + argv[argc++] = (char *)"-W"; + break; + default: + fatalx("invalid process id %d", proc_id); + } + + argv[argc++] = (char *)"-P"; + argv[argc++] = (char *)chroot_path; + + if (!daemonize) + argv[argc++] = (char *)"-d"; + if (verbosity > 0) + argv[argc++] = (char *)"-v"; + if (verbosity > 1) + argv[argc++] = (char *)"-v"; + argv[argc++] = NULL; + + execvp(argv0, argv); + fatal("execvp"); +} + +static void +start_repo_children(struct gotd *gotd, char *argv0, int daemonize, + int verbosity) +{ + struct gotd_repo *repo = NULL; + struct gotd_child_proc *proc; + int i; + + /* + * XXX For now, use one reader and one writer per repository. + * This should be changed to N readers + M writers. + */ + gotd->nprocs = gotd->nrepos * 2; + gotd->procs = calloc(gotd->nprocs, sizeof(*gotd->procs)); + if (gotd->procs == NULL) + fatal("calloc"); + for (i = 0; i < gotd->nprocs; i++) { + if (repo == NULL) + repo = TAILQ_FIRST(&gotd->repos); + proc = &gotd->procs[i]; + if (i < gotd->nrepos) + proc->type = PROC_REPO_READ; + else + proc->type = PROC_REPO_WRITE; + if (strlcpy(proc->repo_name, repo->name, + sizeof(proc->repo_name)) >= sizeof(proc->repo_name)) + fatalx("repository name too long: %s", repo->name); + log_debug("adding repository %s", repo->name); + if (realpath(repo->path, proc->chroot_path) == NULL) + fatal("%s", repo->path); + if (socketpair(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, + PF_UNSPEC, proc->pipe) == -1) + fatal("socketpair"); + proc->pid = start_child(proc->type, proc->chroot_path, argv0, + proc->pipe[1], daemonize, verbosity); + imsg_init(&proc->iev.ibuf, proc->pipe[0]); + log_debug("proc %s %s is on fd %d", + gotd_proc_names[proc->type], proc->chroot_path, + proc->pipe[0]); + proc->iev.handler = gotd_dispatch; + proc->iev.events = EV_READ; + proc->iev.handler_arg = NULL; + event_set(&proc->iev.ev, proc->iev.ibuf.fd, EV_READ, + gotd_dispatch, &proc->iev); + + repo = TAILQ_NEXT(repo, entry); + } +} + +static void +apply_unveil(void) +{ + struct gotd_repo *repo; + + TAILQ_FOREACH(repo, &gotd.repos, entry) { + if (unveil(repo->path, "rwc") == -1) + fatal("unveil %s", repo->path); + } + + if (unveil(GOT_TMPDIR_STR, "rwc") == -1) + fatal("unveil %s", GOT_TMPDIR_STR); + + if (unveil(NULL, NULL) == -1) + fatal("unveil"); +} + +int +main(int argc, char **argv) +{ + const struct got_error *error = NULL; + int ch, fd = -1, daemonize = 1, verbosity = 0, noaction = 0; + const char *confpath = GOTD_CONF_PATH; + char *argv0 = argv[0]; + char title[2048]; + gid_t groups[NGROUPS_MAX + 1]; + int ngroups = NGROUPS_MAX + 1; + struct passwd *pw = NULL; + struct group *gr = NULL; + char *repo_path = NULL; + enum gotd_procid proc_id = PROC_GOTD; + struct event evsigint, evsigterm, evsighup, evsigusr1; + int *pack_fds = NULL, *temp_fds = NULL; + + log_init(1, LOG_DAEMON); /* Log to stderr until daemonized. */ + + while ((ch = getopt(argc, argv, "df:nvRWP:")) != -1) { + switch (ch) { + case 'd': + daemonize = 0; + break; + case 'f': + confpath = optarg; + break; + case 'n': + noaction = 1; + break; + case 'v': + if (verbosity < 3) + verbosity++; + break; + case 'R': + proc_id = PROC_REPO_READ; + break; + case 'W': + proc_id = PROC_REPO_WRITE; + break; + case 'P': + repo_path = realpath(optarg, NULL); + if (repo_path == NULL) + fatal("realpath '%s'", optarg); + break; + default: + usage(); + } + } + + argc -= optind; + argv += optind; + + if (argc != 0) + usage(); + + if (geteuid()) + fatalx("need root privileges"); + + log_init(daemonize ? 0 : 1, LOG_DAEMON); + log_setverbose(verbosity); + + if (parse_config(confpath, proc_id, &gotd) != 0) + return 1; + + if (proc_id == PROC_GOTD && + (gotd.nrepos == 0 || TAILQ_EMPTY(&gotd.repos))) + fatalx("no repository defined in configuration file"); + + pw = getpwnam(gotd.user_name); + if (pw == NULL) + fatal("getpwuid: user %s not found", gotd.user_name); + + if (pw->pw_uid == 0) { + fatalx("cannot run %s as %s: the user running %s " + "must not be the superuser", + getprogname(), pw->pw_name, getprogname()); + } + + if (getgrouplist(pw->pw_name, pw->pw_gid, groups, &ngroups) == -1) + log_warnx("group membership list truncated"); + + gr = match_group(groups, ngroups, gotd.unix_group_name); + if (gr == NULL) { + fatalx("cannot start %s: the user running %s " + "must be a secondary member of group %s", + getprogname(), getprogname(), gotd.unix_group_name); + } + if (gr->gr_gid == pw->pw_gid) { + fatalx("cannot start %s: the user running %s " + "must be a secondary member of group %s, but " + "%s is the user's primary group", + getprogname(), getprogname(), gotd.unix_group_name, + gotd.unix_group_name); + } + + if (proc_id == PROC_GOTD && + !got_path_is_absolute(gotd.unix_socket_path)) + fatalx("bad unix socket path \"%s\": must be an absolute path", + gotd.unix_socket_path); + + if (noaction) + return 0; + + if (proc_id == PROC_GOTD && verbosity) { + log_info("socket: %s", gotd.unix_socket_path); + log_info("user: %s", pw->pw_name); + log_info("secondary group: %s", gr->gr_name); + + fd = unix_socket_listen(gotd.unix_socket_path, pw->pw_uid, + gr->gr_gid); + if (fd == -1) { + fatal("cannot listen on unix socket %s", + gotd.unix_socket_path); + } + } + + if (proc_id == PROC_GOTD) { + gotd.pid = getpid(); + snprintf(title, sizeof(title), "%s", gotd_proc_names[proc_id]); + start_repo_children(&gotd, argv0, daemonize, verbosity); + arc4random_buf(&clients_hash_key, sizeof(clients_hash_key)); + if (daemonize && daemon(0, 0) == -1) + fatal("daemon"); + } else if (proc_id == PROC_REPO_READ || proc_id == PROC_REPO_WRITE) { + error = got_repo_pack_fds_open(&pack_fds); + if (error != NULL) + fatalx("cannot open pack tempfiles: %s", error->msg); + error = got_repo_temp_fds_open(&temp_fds); + if (error != NULL) + fatalx("cannot open pack tempfiles: %s", error->msg); + if (repo_path == NULL) + fatalx("repository path not specified"); + snprintf(title, sizeof(title), "%s %s", + gotd_proc_names[proc_id], repo_path); + if (chroot(repo_path) == -1) + fatal("chroot"); + if (chdir("/") == -1) + fatal("chdir(\"/\")"); + if (daemonize && daemon(1, 0) == -1) + fatal("daemon"); + } else + fatal("invalid process id %d", proc_id); + + setproctitle("%s", title); + log_procinit(title); + + /* Drop root privileges. */ + if (setgid(pw->pw_gid) == -1) + fatal("setgid %d failed", pw->pw_gid); + if (setuid(pw->pw_uid) == -1) + fatal("setuid %d failed", pw->pw_uid); + + event_init(); + + switch (proc_id) { + case PROC_GOTD: +#ifndef PROFILE + if (pledge("stdio rpath wpath cpath proc getpw sendfd recvfd " + "fattr flock unix unveil", NULL) == -1) + err(1, "pledge"); +#endif + break; + case PROC_REPO_READ: +#ifndef PROFILE + if (pledge("stdio rpath sendfd recvfd", NULL) == -1) + err(1, "pledge"); +#endif + repo_read_main(title, pack_fds, temp_fds); + /* NOTREACHED */ + exit(0); + case PROC_REPO_WRITE: +#ifndef PROFILE + if (pledge("stdio rpath sendfd recvfd", NULL) == -1) + err(1, "pledge"); +#endif + repo_write_main(title, pack_fds, temp_fds); + /* NOTREACHED */ + exit(0); + default: + fatal("invalid process id %d", proc_id); + } + + if (proc_id != PROC_GOTD) + fatal("invalid process id %d", proc_id); + + apply_unveil(); + + signal_set(&evsigint, SIGINT, gotd_sighdlr, NULL); + signal_set(&evsigterm, SIGTERM, gotd_sighdlr, NULL); + signal_set(&evsighup, SIGHUP, gotd_sighdlr, NULL); + signal_set(&evsigusr1, SIGUSR1, gotd_sighdlr, NULL); + signal(SIGPIPE, SIG_IGN); + + signal_add(&evsigint, NULL); + signal_add(&evsigterm, NULL); + signal_add(&evsighup, NULL); + signal_add(&evsigusr1, NULL); + + event_set(&gotd.ev, fd, EV_READ | EV_PERSIST, gotd_accept, NULL); + if (event_add(&gotd.ev, NULL)) + fatalx("event add"); + evtimer_set(&gotd.pause, gotd_accept_paused, NULL); + + event_dispatch(); + + if (fd != -1) + close(fd); + if (pack_fds) + got_repo_pack_fds_close(pack_fds); + free(repo_path); + return 0; +} blob - /dev/null blob + a0518c44a5f26becf1f162b99350cdecae58cb06 (mode 644) --- /dev/null +++ gotd/gotd.conf.5 @@ -0,0 +1,133 @@ +.\" +.\" Copyright (c) 2022 Stefan Sperling +.\" +.\" Permission to use, copy, modify, and distribute this software for any +.\" purpose with or without fee is hereby granted, provided that the above +.\" copyright notice and this permission notice appear in all copies. +.\" +.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +.\" +.Dd $Mdocdate$ +.Dt GOTD.CONF 5 +.Os +.Sh NAME +.Nm gotd.conf +.Nd gotd configuration file +.Sh DESCRIPTION +.Nm +is the run-time configuration file for +.Xr gotd 8 . +.Pp +The file format is line-based, with one configuration directive per line. +Any lines beginning with a +.Sq # +are treated as comments and ignored. +.Sh GLOBAL CONFIGURATION +The available global configuration directives are as follows: +.Bl -tag -width Ds +.It Ic unix_socket Ar path +Set the path to the unix socket which +.Xr gotd 8 +should listen on. +If not specified, the path +.Pa /var/run/gotd.sock +will be used. +.It Ic unix_group Ar group +Set the +.Ar group , +defined in the +.Xr group 5 +file, which is allowed to access +.Xr gotd 8 +via +.Xr gotsh 1 . +The +.Xr gotd 8 +user must be a secondary member of this group. +If not specified, the group _gotsh will be used. +.It Ic user Ar user +Set the +.Ar user +which will run +.Xr gotd 8 . +Initially, +.Xr gotd 8 +requires root privileges in order to create its unix socket and start +child processes in a +.Xr chroot 2 +environment. +Afterwards, +.Xr gotd 8 +drops privileges to the specified +.Ar user . +If not specified, the user _gotd will be used. +.El +.Sh REPOSITORY CONFIGURATION +At least one repository context must exist for +.Xr gotd 8 +to function. +.Pp +A repository context is declared with a unique +.Ar name , +followed by repository-specific configuration directives inside curly braces: +.Pp +.Ic repository Ar name Brq ... +.Pp +.Xr got 1 +and +.Xr git 1 +clients can connect to a repository by including the repository's unique +.Ar name +in the request URL. +Clients appending the string +.Dq .git +to the +.Ar name +will also be accepted. +.Pp +If desired, the +.Ar name +may contain path-separators, +.Dq / , +to expose repositories as part of a virtual client-visible directory hierarchy. +.Pp +The available repository configuration directives are as follows: +.Bl -tag -width Ds +.It Ic path Ar path +Set the path to the Git repository. +.EL +.Sh FILES +.Bl -tag -width Ds -compact +.It Pa /etc/gotd.conf +Location of the +.Nm +configuration file. +.El +.Sh EXAMPLES +.Bd -literal -offset indent +# Default unix_group and user values: +unix_group _gotsh +user _gotd + +# This repository can be accessed via ssh://user@example.com/src +repository "src" { + path "/var/git/src.git" +} + +# This repository can be accessed via +# ssh://user@example.com/openbsd/ports +repository "openbsd/ports" { + path "/var/git/ports.git" +} +.Ed +.Sh SEE ALSO +.Xr got 1 , +.Xr gotsh 1 , +.Xr group 5 , +.Xr gotd 8 blob - /dev/null blob + 181834ce44e984caf7169755b86ad50d182325fd (mode 644) --- /dev/null +++ gotd/gotd.h @@ -0,0 +1,368 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + + +#define GOTD_UNIX_SOCKET "/var/run/gotd.sock" +#define GOTD_UNIX_SOCKET_BACKLOG 10 +#define GOTD_UNIX_GROUP "_gotsh" +#define GOTD_USER "_gotd" +#define GOTD_CONF_PATH "/etc/gotd.conf" + +#define GOTD_MAXCLIENTS 1024 +#define GOTD_FD_RESERVE 5 +#define GOTD_FD_NEEDED 6 +#define GOTD_SOCK_FILENO 3 + +/* Client hash tables need some extra room. */ +#define GOTD_CLIENT_TABLE_SIZE (GOTD_MAXCLIENTS * 4) + +enum gotd_procid { + PROC_GOTD = 0, + PROC_REPO_READ, + PROC_REPO_WRITE, + PROC_MAX, +}; + +struct gotd_imsgev { + struct imsgbuf ibuf; + void (*handler)(int, short, void *); + void *handler_arg; + struct event ev; + short events; +}; + +struct gotd_child_proc { + pid_t pid; + enum gotd_procid type; + char repo_name[NAME_MAX]; + char chroot_path[PATH_MAX]; + int pipe[2]; + struct gotd_imsgev iev; + size_t nhelpers; +}; + +struct gotd_repo { + TAILQ_ENTRY(gotd_repo) entry; + + char name[NAME_MAX]; + char path[PATH_MAX]; +}; +TAILQ_HEAD(gotd_repolist, gotd_repo); + +enum gotd_client_state { + GOTD_STATE_EXPECT_LIST_REFS, + GOTD_STATE_EXPECT_CAPABILITIES, + GOTD_STATE_EXPECT_WANT, + GOTD_STATE_EXPECT_REF_UPDATE, + GOTD_STATE_EXPECT_MORE_REF_UPDATES, + GOTD_STATE_EXPECT_HAVE, + GOTD_STATE_EXPECT_PACKFILE, + GOTD_STATE_EXPECT_DONE, + GOTD_STATE_DONE, +}; + +struct gotd_client_capability { + char *key; + char *value; +}; + +struct gotd_object_id_array { + struct got_object_id **ids; + size_t nalloc; + size_t nids; +}; + +struct gotd { + pid_t pid; + char unix_socket_path[PATH_MAX]; + char unix_group_name[32]; + char user_name[32]; + struct gotd_repolist repos; + int nrepos; + int verbosity; + struct event ev; + struct event pause; + struct gotd_child_proc *procs; + int nprocs; +}; + +enum gotd_imsg_type { + /* An error occured while processing a request. */ + GOTD_IMSG_ERROR, + + /* Request a list of references. */ + GOTD_IMSG_LIST_REFS, + GOTD_IMSG_LIST_REFS_INTERNAL, + + /* References. */ + GOTD_IMSG_REFLIST, + GOTD_IMSG_REF, + GOTD_IMSG_SYMREF, + + /* Git protocol capabilities. */ + GOTD_IMSG_CAPABILITIES, + GOTD_IMSG_CAPABILITY, + + /* Git protocol chatter. */ + GOTD_IMSG_WANT, /* The client wants an object. */ + GOTD_IMSG_HAVE, /* The client has an object. */ + GOTD_IMSG_ACK, /* The server has an object or a reference. */ + GOTD_IMSG_NAK, /* The server does not have an object/ref. */ + GOTD_IMSG_REF_UPDATE, /* The client wants to update a reference. */ + GOTD_IMSG_REF_DELETE, /* The client wants to delete a reference. */ + GOTD_IMSG_FLUSH, /* The client sent a flush packet. */ + GOTD_IMSG_DONE, /* The client is done chatting. */ + + /* Sending or receiving a pack file. */ + GOTD_IMSG_SEND_PACKFILE, /* The server is sending a pack file. */ + GOTD_IMSG_RECV_PACKFILE, /* The server is receiving a pack file. */ + GOTD_IMSG_PACKIDX_FILE, /* Temporary file handle for new pack index. */ + GOTD_IMSG_PACKFILE_PIPE, /* Pipe to send/receive a pack file stream. */ + GOTD_IMSG_PACKFILE_PROGRESS, /* Progress reporting. */ + GOTD_IMSG_PACKFILE_READY, /* Pack file is ready to be sent. */ + GOTD_IMSG_PACKFILE_STATUS, /* Received pack success/failure status. */ + GOTD_IMSG_PACKFILE_INSTALL, /* Received pack file can be installed. */ + GOTD_IMSG_PACKFILE_DONE, /* Pack file has been sent/received. */ + + /* Reference updates. */ + GOTD_IMSG_REF_UPDATES_START, /* Ref updates starting. */ + GOTD_IMSG_REF_UPDATE_OK, /* Update went OK. */ + GOTD_IMSG_REF_UPDATE_NG, /* Update was not good. */ + GOTD_IMSG_REFS_UPDATED, /* The server proccessed all ref updates. */ + + /* Client is disconnecting. */ + GOTD_IMSG_DISCONNECT, +}; + +/* Structure for GOTD_IMSG_ERROR. */ +struct gotd_imsg_error { + int code; /* an error code from got_error.h */ + int errno_code; /* in case code equals GOT_ERR_ERRNO */ + uint32_t client_id; + char msg[GOT_ERR_MAX_MSG_SIZE]; +} __attribute__((__packed__)); + +/* Structure for GOTD_IMSG_LIST_REFS. */ +struct gotd_imsg_list_refs { + char repo_name[NAME_MAX]; + int client_is_reading; /* 1 if reading, 0 if writing */ +}; + +/* Structure for GOTD_IMSG_LIST_REFS_INTERNAL. */ +struct gotd_imsg_list_refs_internal { + uint32_t client_id; +}; + +/* Structure for GOTD_IMSG_REFLIST. */ +struct gotd_imsg_reflist { + size_t nrefs; + + /* Followed by nrefs times of gotd_imsg_ref/gotd_imsg_symref data. */ +} __attribute__((__packed__)); + +/* Structure for GOTD_IMSG_REF data. */ +struct gotd_imsg_ref { + uint8_t id[SHA1_DIGEST_LENGTH]; + size_t name_len; + /* Followed by name_len data bytes. */ +} __attribute__((__packed__)); + +/* Structure for GOTD_IMSG_SYMREF data. */ +struct gotd_imsg_symref { + size_t name_len; + size_t target_len; + uint8_t target_id[SHA1_DIGEST_LENGTH]; + + /* + * Followed by name_len + target_len data bytes. + */ +} __attribute__((__packed__)); + +/* Structure for GOTD_IMSG_CAPABILITIES data. */ +struct gotd_imsg_capabilities { + size_t ncapabilities; + + /* + * Followed by ncapabilities * GOTD_IMSG_CAPABILITY. + */ +} __attribute__((__packed__)); + +/* Structure for GOTD_IMSG_CAPABILITY data. */ +struct gotd_imsg_capability { + size_t key_len; + size_t value_len; + + /* + * Followed by key_len + value_len data bytes. + */ +} __attribute__((__packed__)); + +/* Structure for GOTD_IMSG_WANT data. */ +struct gotd_imsg_want { + uint8_t object_id[SHA1_DIGEST_LENGTH]; + uint32_t client_id; +} __attribute__((__packed__)); + +/* Structure for GOTD_IMSG_HAVE data. */ +struct gotd_imsg_have { + uint8_t object_id[SHA1_DIGEST_LENGTH]; + uint32_t client_id; +} __attribute__((__packed__)); + +/* Structure for GOTD_IMSG_ACK data. */ +struct gotd_imsg_ack { + uint8_t object_id[SHA1_DIGEST_LENGTH]; + uint32_t client_id; +} __attribute__((__packed__)); + +/* Structure for GOTD_IMSG_NAK data. */ +struct gotd_imsg_nak { + uint8_t object_id[SHA1_DIGEST_LENGTH]; + uint32_t client_id; +} __attribute__((__packed__)); + +/* Structure for GOTD_IMSG_PACKFILE_STATUS data. */ +struct gotd_imsg_packfile_status { + size_t reason_len; + + /* Followed by reason_len data bytes. */ +} __attribute__((__packed__)); + + +/* Structure for GOTD_IMSG_REF_UPDATE data. */ +struct gotd_imsg_ref_update { + uint8_t old_id[SHA1_DIGEST_LENGTH]; + uint8_t new_id[SHA1_DIGEST_LENGTH]; + int ref_is_new; + uint32_t client_id; + size_t name_len; + + /* Followed by name_len data bytes. */ +} __attribute__((__packed__)); + +/* Structure for GOTD_IMSG_REF_UPDATES_START data. */ +struct gotd_imsg_ref_updates_start { + int nref_updates; + uint32_t client_id; + + /* Followed by nref_updates GOT_IMSG_REF_UPDATE_OK/NG messages. */ +}; + +/* Structure for GOTD_IMSG_REF_UPDATE_OK data. */ +struct gotd_imsg_ref_update_ok { + uint8_t old_id[SHA1_DIGEST_LENGTH]; + uint8_t new_id[SHA1_DIGEST_LENGTH]; + int ref_is_new; + uint32_t client_id; + size_t name_len; + + /* Followed by name_len data bytes. */ +} __attribute__((__packed__)); + +/* Structure for GOTD_IMSG_REF_UPDATE_NG data. */ +struct gotd_imsg_ref_update_ng { + uint8_t old_id[SHA1_DIGEST_LENGTH]; + uint8_t new_id[SHA1_DIGEST_LENGTH]; + uint32_t client_id; + size_t name_len; + size_t reason_len; + + /* Followed by name_len + reason_len data bytes. */ +} __attribute__((__packed__)); + +/* Structure for GOTD_IMSG_SEND_PACKFILE data. */ +struct gotd_imsg_send_packfile { + uint32_t client_id; + int report_progress; + + /* delta cache file is sent as a file descriptor */ + + /* followed by two GOTD_IMSG_PACKFILE_PIPE messages */ +}; + +/* Structure for GOTD_IMSG_RECV_PACKFILE data. */ +struct gotd_imsg_recv_packfile { + uint32_t client_id; + int report_status; + + /* pack destination temp file is sent as a file descriptor */ +}; + +/* Structure for GOTD_IMSG_PACKFILE_PIPE data. */ +struct gotd_imsg_packfile_pipe { + uint32_t client_id; +}; + +/* Structure for GOTD_IMSG_PACKIDX_FILE data. */ +struct gotd_imsg_packidx_file { + uint32_t client_id; +}; + + +/* + * Structure for GOTD_IMSG_PACKFILE_PROGRESS and + * GOTD_IMSG_PACKFILE_READY data. + */ +struct gotd_imsg_packfile_progress { + uint32_t client_id; + int ncolored; + int nfound; + int ntrees; + off_t packfile_size; + int ncommits; + int nobj_total; + int nobj_deltify; + int nobj_written; +}; + +/* Structure for GOTD_IMSG_PACKFILE_INSTALL. */ +struct gotd_imsg_packfile_install { + uint32_t client_id; + uint8_t pack_sha1[SHA1_DIGEST_LENGTH]; +}; + +/* Structure for GOTD_IMSG_PACKFILE_DONE data. */ +struct gotd_imsg_packfile_done { + uint32_t client_id; +}; + +/* Structure for GOTD_IMSG_DISCONNECT data. */ +struct gotd_imsg_disconnect { + uint32_t client_id; +}; + +int parse_config(const char *, enum gotd_procid, struct gotd *); + +/* imsg.c */ +const struct got_error *gotd_imsg_flush(struct imsgbuf *); +const struct got_error *gotd_imsg_recv(struct imsg *, struct imsgbuf *, size_t); +const struct got_error *gotd_imsg_poll_recv(struct imsg *, struct imsgbuf *, + size_t); +const struct got_error *gotd_imsg_recv_error(uint32_t *client_id, + struct imsg *imsg); +int gotd_imsg_send_error(struct imsgbuf *ibuf, uint32_t, uint32_t, + const struct got_error *); +int gotd_imsg_send_error_event(struct gotd_imsgev *, uint32_t, uint32_t, + const struct got_error *); +void gotd_imsg_event_add(struct gotd_imsgev *); +int gotd_imsg_compose_event(struct gotd_imsgev *, uint16_t, uint32_t, int, + void *, uint16_t); +int gotd_imsg_forward(struct gotd_imsgev *, struct imsg *, int); + +void gotd_imsg_send_ack(struct got_object_id *, struct imsgbuf *, + uint32_t, pid_t); +void gotd_imsg_send_nak(struct got_object_id *, struct imsgbuf *, + uint32_t, pid_t); blob - /dev/null blob + 3c1a799bc318b9a6294c4a902145a0983b95815b (mode 644) --- /dev/null +++ gotd/imsg.c @@ -0,0 +1,203 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "got_error.h" + +#include "got_lib_poll.h" + +#include "gotd.h" + +const struct got_error * +gotd_imsg_recv_error(uint32_t *client_id, struct imsg *imsg) +{ + struct gotd_imsg_error ierr; + size_t datalen; + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(ierr)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&ierr, imsg->data, sizeof(ierr)); + + if (client_id) + *client_id = ierr.client_id; + + if (ierr.code == GOT_ERR_ERRNO) + errno = ierr.errno_code; + + return got_error_msg(ierr.code, ierr.msg); +} + +const struct got_error * +gotd_imsg_flush(struct imsgbuf *ibuf) +{ + const struct got_error *err; + + err = got_poll_fd(ibuf->fd, POLLOUT, INFTIM); + if (err) + return err; + + if (imsg_flush(ibuf) == -1) { + imsg_clear(ibuf); + return got_error_from_errno("imsg_flush"); + } + + return NULL; +} + +const struct got_error * +gotd_imsg_recv(struct imsg *imsg, struct imsgbuf *ibuf, size_t min_datalen) +{ + ssize_t n; + + n = imsg_get(ibuf, imsg); + if (n == -1) + return got_error_from_errno("imsg_get"); + + if (n == 0) { + n = imsg_read(ibuf); + if (n == -1) { + if (errno == EAGAIN) + return got_error(GOT_ERR_PRIVSEP_READ); + return got_error_from_errno("imsg_read"); + } + if (n == 0) + return got_error(GOT_ERR_EOF); + n = imsg_get(ibuf, imsg); + if (n == -1) + return got_error_from_errno("imsg_get"); + } + + if (imsg->hdr.len < IMSG_HEADER_SIZE + min_datalen) + return got_error(GOT_ERR_PRIVSEP_LEN); + + return NULL; +} + +const struct got_error * +gotd_imsg_poll_recv(struct imsg *imsg, struct imsgbuf *ibuf, size_t min_datalen) +{ + const struct got_error *err = NULL; + + for (;;) { + err = gotd_imsg_recv(imsg, ibuf, min_datalen); + if (err == NULL || err->code != GOT_ERR_PRIVSEP_READ) + return err; + + err = got_poll_fd(ibuf->fd, POLLIN, INFTIM); + if (err) + break; + } + + return err; +} + +int +gotd_imsg_send_error(struct imsgbuf *ibuf, uint32_t peerid, + uint32_t client_id, const struct got_error *err) +{ + const struct got_error *flush_err; + struct gotd_imsg_error ierr; + int ret; + + ierr.code = err->code; + if (err->code == GOT_ERR_ERRNO) + ierr.errno_code = errno; + else + ierr.errno_code = 0; + ierr.client_id = client_id; + strlcpy(ierr.msg, err->msg, sizeof(ierr.msg)); + + ret = imsg_compose(ibuf, GOTD_IMSG_ERROR, peerid, getpid(), -1, + &ierr, sizeof(ierr)); + if (ret == -1) + return -1; + + flush_err = gotd_imsg_flush(ibuf); + if (flush_err) + return -1; + + return 0; +} + +int +gotd_imsg_send_error_event(struct gotd_imsgev *iev, uint32_t peerid, + uint32_t client_id, const struct got_error *err) +{ + struct gotd_imsg_error ierr; + int ret; + + ierr.code = err->code; + if (err->code == GOT_ERR_ERRNO) + ierr.errno_code = errno; + else + ierr.errno_code = 0; + ierr.client_id = client_id; + strlcpy(ierr.msg, err->msg, sizeof(ierr.msg)); + + ret = gotd_imsg_compose_event(iev, GOTD_IMSG_ERROR, peerid, -1, + &ierr, sizeof(ierr)); + if (ret == -1) + return -1; + + return 0; +} + +void +gotd_imsg_event_add(struct gotd_imsgev *iev) +{ + iev->events = EV_READ; + if (iev->ibuf.w.queued) + iev->events |= EV_WRITE; + + event_del(&iev->ev); + event_set(&iev->ev, iev->ibuf.fd, iev->events, iev->handler, iev); + event_add(&iev->ev, NULL); +} + +int +gotd_imsg_compose_event(struct gotd_imsgev *iev, uint16_t type, uint32_t peerid, + int fd, void *data, uint16_t datalen) +{ + int ret; + + ret = imsg_compose(&iev->ibuf, type, peerid, getpid(), fd, + data, datalen); + if (ret != -1) + gotd_imsg_event_add(iev); + + return ret; +} + +int +gotd_imsg_forward(struct gotd_imsgev *iev, struct imsg *imsg, int fd) +{ + return gotd_imsg_compose_event(iev, imsg->hdr.type, imsg->hdr.peerid, + fd, imsg->data, imsg->hdr.len - IMSG_HEADER_SIZE); +} blob - /dev/null blob + ee4438ac0458a156488cc7febe21debb181f25a7 (mode 644) --- /dev/null +++ gotd/log.c @@ -0,0 +1,195 @@ +/* + * Copyright (c) 2003, 2004 Henning Brauer + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "log.h" + +static int debug; +static int verbose; +const char *log_procname; + +void +log_init(int n_debug, int facility) +{ + debug = n_debug; + verbose = n_debug; + log_procinit(getprogname()); + + if (!debug) + openlog(getprogname(), LOG_PID | LOG_NDELAY, facility); + + tzset(); +} + +void +log_procinit(const char *procname) +{ + if (procname != NULL) + log_procname = procname; +} + +void +log_setverbose(int v) +{ + verbose = v; +} + +int +log_getverbose(void) +{ + return (verbose); +} + +void +logit(int pri, const char *fmt, ...) +{ + va_list ap; + + va_start(ap, fmt); + vlog(pri, fmt, ap); + va_end(ap); +} + +void +vlog(int pri, const char *fmt, va_list ap) +{ + char *nfmt; + int saved_errno = errno; + + if (debug) { + /* best effort in out of mem situations */ + if (asprintf(&nfmt, "%s\n", fmt) == -1) { + vfprintf(stderr, fmt, ap); + fprintf(stderr, "\n"); + } else { + vfprintf(stderr, nfmt, ap); + free(nfmt); + } + fflush(stderr); + } else + vsyslog(pri, fmt, ap); + + errno = saved_errno; +} + +void +log_warn(const char *emsg, ...) +{ + char *nfmt; + va_list ap; + int saved_errno = errno; + + /* best effort to even work in out of memory situations */ + if (emsg == NULL) + logit(LOG_CRIT, "%s", strerror(saved_errno)); + else { + va_start(ap, emsg); + + if (asprintf(&nfmt, "%s: %s", emsg, + strerror(saved_errno)) == -1) { + /* we tried it... */ + vlog(LOG_CRIT, emsg, ap); + logit(LOG_CRIT, "%s", strerror(saved_errno)); + } else { + vlog(LOG_CRIT, nfmt, ap); + free(nfmt); + } + va_end(ap); + } + + errno = saved_errno; +} + +void +log_warnx(const char *emsg, ...) +{ + va_list ap; + + va_start(ap, emsg); + vlog(LOG_CRIT, emsg, ap); + va_end(ap); +} + +void +log_info(const char *emsg, ...) +{ + va_list ap; + + va_start(ap, emsg); + vlog(LOG_INFO, emsg, ap); + va_end(ap); +} + +void +log_debug(const char *emsg, ...) +{ + va_list ap; + + if (verbose) { + va_start(ap, emsg); + vlog(LOG_DEBUG, emsg, ap); + va_end(ap); + } +} + +static void +vfatalc(int code, const char *emsg, va_list ap) +{ + static char s[BUFSIZ]; + const char *sep; + + if (emsg != NULL) { + (void)vsnprintf(s, sizeof(s), emsg, ap); + sep = ": "; + } else { + s[0] = '\0'; + sep = ""; + } + if (code) + logit(LOG_CRIT, "%s: %s%s%s", + log_procname, s, sep, strerror(code)); + else + logit(LOG_CRIT, "%s%s%s", log_procname, sep, s); +} + +void +fatal(const char *emsg, ...) +{ + va_list ap; + + va_start(ap, emsg); + vfatalc(errno, emsg, ap); + va_end(ap); + exit(1); +} + +void +fatalx(const char *emsg, ...) +{ + va_list ap; + + va_start(ap, emsg); + vfatalc(0, emsg, ap); + va_end(ap); + exit(1); +} blob - /dev/null blob + fff6f87ffd0edd1a69a1db346fad051aadc39c21 (mode 644) --- /dev/null +++ gotd/log.h @@ -0,0 +1,36 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +void log_init(int, int); +void log_procinit(const char *); +void log_setverbose(int); +int log_getverbose(void); +void log_warn(const char *, ...) + __attribute__((__format__ (printf, 1, 2))); +void log_warnx(const char *, ...) + __attribute__((__format__ (printf, 1, 2))); +void log_info(const char *, ...) + __attribute__((__format__ (printf, 1, 2))); +void log_debug(const char *, ...) + __attribute__((__format__ (printf, 1, 2))); +void logit(int, const char *, ...) + __attribute__((__format__ (printf, 2, 3))); +void vlog(int, const char *, va_list) + __attribute__((__format__ (printf, 2, 0))); +__dead void fatal(const char *, ...) + __attribute__((__format__ (printf, 1, 2))); +__dead void fatalx(const char *, ...) + __attribute__((__format__ (printf, 1, 2))); blob - /dev/null blob + 5ccdb6be0bc1138dacf2aa6282eca1b79c9bba8f (mode 644) --- /dev/null +++ gotd/parse.y @@ -0,0 +1,733 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * Copyright (c) 2016-2019, 2020-2021 Tracey Emery + * Copyright (c) 2004, 2005 Esben Norby + * Copyright (c) 2004 Ryan McBride + * Copyright (c) 2002, 2003, 2004 Henning Brauer + * Copyright (c) 2001 Markus Friedl. All rights reserved. + * Copyright (c) 2001 Daniel Hartmeier. All rights reserved. + * Copyright (c) 2001 Theo de Raadt. All rights reserved. + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +%{ +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "got_error.h" +#include "got_path.h" + +#include "log.h" +#include "gotd.h" + +TAILQ_HEAD(files, file) files = TAILQ_HEAD_INITIALIZER(files); +static struct file { + TAILQ_ENTRY(file) entry; + FILE *stream; + char *name; + int lineno; + int errors; +} *file; +struct file *newfile(const char *, int); +static void closefile(struct file *); +int check_file_secrecy(int, const char *); +int yyparse(void); +int yylex(void); +int yyerror(const char *, ...) + __attribute__((__format__ (printf, 1, 2))) + __attribute__((__nonnull__ (1))); +int kw_cmp(const void *, const void *); +int lookup(char *); +int lgetc(int); +int lungetc(int); +int findeol(void); + +TAILQ_HEAD(symhead, sym) symhead = TAILQ_HEAD_INITIALIZER(symhead); +struct sym { + TAILQ_ENTRY(sym) entry; + int used; + int persist; + char *nam; + char *val; +}; + +int symset(const char *, const char *, int); +char *symget(const char *); + +static int errors; + +static struct gotd *gotd; +static struct gotd_repo *new_repo; +static struct gotd_repo *conf_new_repo(const char *); +static enum gotd_procid gotd_proc_id; + +typedef struct { + union { + long long number; + char *string; + } v; + int lineno; +} YYSTYPE; + +%} + +%token PATH ERROR ON UNIX_SOCKET UNIX_GROUP USER REPOSITORY + +%token STRING +%token NUMBER +%type boolean + +%% + +grammar : + | grammar '\n' + | grammar main '\n' + | grammar repository '\n' + ; + +boolean : STRING { + if (strcasecmp($1, "1") == 0 || + strcasecmp($1, "yes") == 0 || + strcasecmp($1, "on") == 0) + $$ = 1; + else if (strcasecmp($1, "0") == 0 || + strcasecmp($1, "off") == 0 || + strcasecmp($1, "no") == 0) + $$ = 0; + else { + yyerror("invalid boolean value '%s'", $1); + free($1); + YYERROR; + } + free($1); + } + | ON { $$ = 1; } + | NUMBER { $$ = $1; } + ; + +main : UNIX_SOCKET STRING { + if (gotd_proc_id == PROC_GOTD) { + if (strlcpy(gotd->unix_socket_path, $2, + sizeof(gotd->unix_socket_path)) >= + sizeof(gotd->unix_socket_path)) { + yyerror("%s: unix socket path too long", + __func__); + free($2); + YYERROR; + } + } + free($2); + } + | UNIX_GROUP STRING { + if (strlcpy(gotd->unix_group_name, $2, + sizeof(gotd->unix_group_name)) >= + sizeof(gotd->unix_group_name)) { + yyerror("%s: unix group name too long", + __func__); + free($2); + YYERROR; + } + free($2); + } + | USER STRING { + if (strlcpy(gotd->user_name, $2, + sizeof(gotd->user_name)) >= + sizeof(gotd->user_name)) { + yyerror("%s: user name too long", __func__); + free($2); + YYERROR; + } + free($2); + } + ; + +repository : REPOSITORY STRING { + struct gotd_repo *repo; + + TAILQ_FOREACH(repo, &gotd->repos, entry) { + if (strcmp(repo->name, $2) == 0) { + yyerror("duplicate repository '%s'", $2); + free($2); + YYERROR; + } + } + + if (gotd_proc_id == PROC_GOTD) { + new_repo = conf_new_repo($2); + } + free($2); + } + | REPOSITORY STRING { + struct gotd_repo *repo; + + TAILQ_FOREACH(repo, &gotd->repos, entry) { + if (strcmp(repo->name, $2) == 0) { + yyerror("duplicate repository '%s'", $2); + free($2); + YYERROR; + } + } + + if (gotd_proc_id == PROC_GOTD) { + new_repo = conf_new_repo($2); + } + free($2); + } '{' optnl repoopts2 '}' { + } + ; + +repoopts1 : PATH STRING { + if (gotd_proc_id == PROC_GOTD) { + if (!got_path_is_absolute($2)) { + yyerror("%s: path %s is not absolute", + __func__, $2); + free($2); + YYERROR; + } + if (strlcpy(new_repo->path, $2, + sizeof(new_repo->path)) >= + sizeof(new_repo->path)) { + yyerror("%s: path truncated", __func__); + free($2); + YYERROR; + } + } + free($2); + } + ; + +repoopts2 : repoopts2 repoopts1 nl + | repoopts1 optnl + ; + +nl : '\n' optnl + ; + +optnl : '\n' optnl /* zero or more newlines */ + | /* empty */ + ; + +%% + +struct keywords { + const char *k_name; + int k_val; +}; + +int +yyerror(const char *fmt, ...) +{ + va_list ap; + char *msg; + + file->errors++; + va_start(ap, fmt); + if (vasprintf(&msg, fmt, ap) == -1) + fatalx("yyerror vasprintf"); + va_end(ap); + logit(LOG_CRIT, "%s:%d: %s", file->name, yylval.lineno, msg); + free(msg); + return (0); +} + +int +kw_cmp(const void *k, const void *e) +{ + return (strcmp(k, ((const struct keywords *)e)->k_name)); +} + +int +lookup(char *s) +{ + /* This has to be sorted always. */ + static const struct keywords keywords[] = { + { "on", ON }, + { "path", PATH }, + { "repository", REPOSITORY }, + { "unix_group", UNIX_GROUP }, + { "unix_socket", UNIX_SOCKET }, + { "user", USER }, + }; + const struct keywords *p; + + p = bsearch(s, keywords, sizeof(keywords)/sizeof(keywords[0]), + sizeof(keywords[0]), kw_cmp); + + if (p) + return (p->k_val); + else + return (STRING); +} + +#define MAXPUSHBACK 128 + +unsigned char *parsebuf; +int parseindex; +unsigned char pushback_buffer[MAXPUSHBACK]; +int pushback_index = 0; + +int +lgetc(int quotec) +{ + int c, next; + + if (parsebuf) { + /* Read character from the parsebuffer instead of input. */ + if (parseindex >= 0) { + c = parsebuf[parseindex++]; + if (c != '\0') + return (c); + parsebuf = NULL; + } else + parseindex++; + } + + if (pushback_index) + return (pushback_buffer[--pushback_index]); + + if (quotec) { + c = getc(file->stream); + if (c == EOF) + yyerror("reached end of file while parsing " + "quoted string"); + return (c); + } + + c = getc(file->stream); + while (c == '\\') { + next = getc(file->stream); + if (next != '\n') { + c = next; + break; + } + yylval.lineno = file->lineno; + file->lineno++; + c = getc(file->stream); + } + + return (c); +} + +int +lungetc(int c) +{ + if (c == EOF) + return (EOF); + if (parsebuf) { + parseindex--; + if (parseindex >= 0) + return (c); + } + if (pushback_index < MAXPUSHBACK-1) + return (pushback_buffer[pushback_index++] = c); + else + return (EOF); +} + +int +findeol(void) +{ + int c; + + parsebuf = NULL; + + /* Skip to either EOF or the first real EOL. */ + while (1) { + if (pushback_index) + c = pushback_buffer[--pushback_index]; + else + c = lgetc(0); + if (c == '\n') { + file->lineno++; + break; + } + if (c == EOF) + break; + } + return (ERROR); +} + +int +yylex(void) +{ + unsigned char buf[8096]; + unsigned char *p, *val; + int quotec, next, c; + int token; + +top: + p = buf; + c = lgetc(0); + while (c == ' ' || c == '\t') + c = lgetc(0); /* nothing */ + + yylval.lineno = file->lineno; + if (c == '#') { + c = lgetc(0); + while (c != '\n' && c != EOF) + c = lgetc(0); /* nothing */ + } + if (c == '$' && parsebuf == NULL) { + while (1) { + c = lgetc(0); + if (c == EOF) + return (0); + + if (p + 1 >= buf + sizeof(buf) - 1) { + yyerror("string too long"); + return (findeol()); + } + if (isalnum(c) || c == '_') { + *p++ = c; + continue; + } + *p = '\0'; + lungetc(c); + break; + } + val = symget(buf); + if (val == NULL) { + yyerror("macro '%s' not defined", buf); + return (findeol()); + } + parsebuf = val; + parseindex = 0; + goto top; + } + + switch (c) { + case '\'': + case '"': + quotec = c; + while (1) { + c = lgetc(quotec); + if (c == EOF) + return (0); + if (c == '\n') { + file->lineno++; + continue; + } else if (c == '\\') { + next = lgetc(quotec); + if (next == EOF) + return (0); + if (next == quotec || c == ' ' || c == '\t') + c = next; + else if (next == '\n') { + file->lineno++; + continue; + } else + lungetc(next); + } else if (c == quotec) { + *p = '\0'; + break; + } else if (c == '\0') { + yyerror("syntax error"); + return (findeol()); + } + if (p + 1 >= buf + sizeof(buf) - 1) { + yyerror("string too long"); + return (findeol()); + } + *p++ = c; + } + yylval.v.string = strdup(buf); + if (yylval.v.string == NULL) + err(1, "yylex: strdup"); + return (STRING); + } + +#define allowed_to_end_number(x) \ + (isspace(x) || x == ')' || x ==',' || x == '/' || x == '}' || x == '=') + + if (c == '-' || isdigit(c)) { + do { + *p++ = c; + if ((unsigned)(p-buf) >= sizeof(buf)) { + yyerror("string too long"); + return (findeol()); + } + c = lgetc(0); + } while (c != EOF && isdigit(c)); + lungetc(c); + if (p == buf + 1 && buf[0] == '-') + goto nodigits; + if (c == EOF || allowed_to_end_number(c)) { + const char *errstr = NULL; + + *p = '\0'; + yylval.v.number = strtonum(buf, LLONG_MIN, + LLONG_MAX, &errstr); + if (errstr) { + yyerror("\"%s\" invalid number: %s", + buf, errstr); + return (findeol()); + } + return (NUMBER); + } else { +nodigits: + while (p > buf + 1) + lungetc(*--p); + c = *--p; + if (c == '-') + return (c); + } + } + +#define allowed_in_string(x) \ + (isalnum(x) || (ispunct(x) && x != '(' && x != ')' && \ + x != '{' && x != '}' && \ + x != '!' && x != '=' && x != '#' && \ + x != ',')) + + if (isalnum(c) || c == ':' || c == '_') { + do { + *p++ = c; + if ((unsigned)(p-buf) >= sizeof(buf)) { + yyerror("string too long"); + return (findeol()); + } + c = lgetc(0); + } while (c != EOF && (allowed_in_string(c))); + lungetc(c); + *p = '\0'; + token = lookup(buf); + if (token == STRING) { + yylval.v.string = strdup(buf); + if (yylval.v.string == NULL) + err(1, "yylex: strdup"); + } + return (token); + } + if (c == '\n') { + yylval.lineno = file->lineno; + file->lineno++; + } + if (c == EOF) + return (0); + return (c); +} + +int +check_file_secrecy(int fd, const char *fname) +{ + struct stat st; + + if (fstat(fd, &st)) { + log_warn("cannot stat %s", fname); + return (-1); + } + if (st.st_uid != 0 && st.st_uid != getuid()) { + log_warnx("%s: owner not root or current user", fname); + return (-1); + } + if (st.st_mode & (S_IWGRP | S_IXGRP | S_IRWXO)) { + log_warnx("%s: group writable or world read/writable", fname); + return (-1); + } + return (0); +} + +struct file * +newfile(const char *name, int secret) +{ + struct file *nfile; + + nfile = calloc(1, sizeof(struct file)); + if (nfile == NULL) { + log_warn("calloc"); + return (NULL); + } + nfile->name = strdup(name); + if (nfile->name == NULL) { + log_warn("strdup"); + free(nfile); + return (NULL); + } + nfile->stream = fopen(nfile->name, "r"); + if (nfile->stream == NULL) { + /* no warning, we don't require a conf file */ + free(nfile->name); + free(nfile); + return (NULL); + } else if (secret && + check_file_secrecy(fileno(nfile->stream), nfile->name)) { + fclose(nfile->stream); + free(nfile->name); + free(nfile); + return (NULL); + } + nfile->lineno = 1; + return (nfile); +} + +static void +closefile(struct file *xfile) +{ + fclose(xfile->stream); + free(xfile->name); + free(xfile); +} + +int +parse_config(const char *filename, enum gotd_procid proc_id, + struct gotd *env) +{ + struct sym *sym, *next; + + memset(env, 0, sizeof(*env)); + + gotd = env; + gotd_proc_id = proc_id; + TAILQ_INIT(&gotd->repos); + + /* Apply default values. */ + if (strlcpy(gotd->unix_socket_path, GOTD_UNIX_SOCKET, + sizeof(gotd->unix_socket_path)) >= sizeof(gotd->unix_socket_path)) { + fprintf(stderr, "%s: unix socket path too long", __func__); + return -1; + } + if (strlcpy(gotd->unix_group_name, GOTD_UNIX_GROUP, + sizeof(gotd->unix_group_name)) >= sizeof(gotd->unix_group_name)) { + fprintf(stderr, "%s: unix group name too long", __func__); + return -1; + } + if (strlcpy(gotd->user_name, GOTD_USER, + sizeof(gotd->user_name)) >= sizeof(gotd->user_name)) { + fprintf(stderr, "%s: user name too long", __func__); + return -1; + } + + file = newfile(filename, 0); + if (file == NULL) { + /* just return, as we don't require a conf file */ + return (0); + } + + yyparse(); + errors = file->errors; + closefile(file); + + /* Free macros and check which have not been used. */ + TAILQ_FOREACH_SAFE(sym, &symhead, entry, next) { + if ((gotd->verbosity > 1) && !sym->used) + fprintf(stderr, "warning: macro '%s' not used\n", + sym->nam); + if (!sym->persist) { + free(sym->nam); + free(sym->val); + TAILQ_REMOVE(&symhead, sym, entry); + free(sym); + } + } + + if (errors) + return (-1); + + return (0); +} + +static struct gotd_repo * +conf_new_repo(const char *name) +{ + struct gotd_repo *repo; + + if (strchr(name, '\n') != NULL) { + fatalx("%s: repository names must not contain linefeeds: %s", + getprogname(), name); + } + + repo = calloc(1, sizeof(*repo)); + if (repo == NULL) + fatalx("%s: calloc", __func__); + + if (strlcpy(repo->name, name, sizeof(repo->name)) >= + sizeof(repo->name)) + fatalx("%s: strlcpy", __func__); + + TAILQ_INSERT_TAIL(&gotd->repos, repo, entry); + gotd->nrepos++; + + return repo; +}; + +int +symset(const char *nam, const char *val, int persist) +{ + struct sym *sym; + + TAILQ_FOREACH(sym, &symhead, entry) { + if (strcmp(nam, sym->nam) == 0) + break; + } + + if (sym != NULL) { + if (sym->persist == 1) + return (0); + else { + free(sym->nam); + free(sym->val); + TAILQ_REMOVE(&symhead, sym, entry); + free(sym); + } + } + sym = calloc(1, sizeof(*sym)); + if (sym == NULL) + return (-1); + + sym->nam = strdup(nam); + if (sym->nam == NULL) { + free(sym); + return (-1); + } + sym->val = strdup(val); + if (sym->val == NULL) { + free(sym->nam); + free(sym); + return (-1); + } + sym->used = 0; + sym->persist = persist; + TAILQ_INSERT_TAIL(&symhead, sym, entry); + return (0); +} + +char * +symget(const char *nam) +{ + struct sym *sym; + + TAILQ_FOREACH(sym, &symhead, entry) { + if (strcmp(nam, sym->nam) == 0) { + sym->used = 1; + return (sym->val); + } + } + return (NULL); +} blob - /dev/null blob + cd9aa4370a65e5d58169777bb0760f75b984b0b1 (mode 644) --- /dev/null +++ gotd/privsep_stub.c @@ -0,0 +1,64 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include "got_error.h" +#include "got_object.h" +#include "got_path.h" + +#include "got_lib_delta.h" +#include "got_lib_object.h" +#include "got_lib_object_cache.h" +#include "got_lib_pack.h" +#include "got_lib_repository.h" +#include "got_lib_privsep.h" + +const struct got_error * +got_privsep_send_stop(int fd) +{ + return got_error(GOT_ERR_NOT_IMPL); +} + +const struct got_error * +got_privsep_wait_for_child(pid_t pid) +{ + return got_error(GOT_ERR_NOT_IMPL); +} + +void +got_privsep_exec_child(int imsg_fds[2], const char *path, const char *repo_path) +{ + fprintf(stderr, "%s: cannot run libexec helpers\n", getprogname()); + exit(1); +} + +const struct got_error * +got_privsep_init_pack_child(struct imsgbuf *ibuf, struct got_pack *pack, + struct got_packidx *packidx) +{ + return got_error(GOT_ERR_NOT_IMPL); +} blob - /dev/null blob + de57ba5027a29b8074cadbe8bb9367f848c92623 (mode 644) --- /dev/null +++ gotd/repo_imsg.c @@ -0,0 +1,90 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include + +#include "got_error.h" +#include "got_object.h" + +#include "got_lib_sha1.h" + +#include "gotd.h" +#include "log.h" + +void +gotd_imsg_send_ack(struct got_object_id *id, struct imsgbuf *ibuf, + uint32_t peerid, pid_t pid) +{ + const struct got_error *err = NULL; + struct gotd_imsg_ack iack; + char hex[SHA1_DIGEST_STRING_LENGTH]; + + if (log_getverbose() > 0 && + got_sha1_digest_to_str(id->sha1, hex, sizeof(hex))) + log_debug("sending ACK for %s", hex); + + memset(&iack, 0, sizeof(iack)); + memcpy(iack.object_id, id->sha1, SHA1_DIGEST_LENGTH); + + if (imsg_compose(ibuf, GOTD_IMSG_ACK, peerid, pid, -1, + &iack, sizeof(iack)) == -1) { + err = got_error_from_errno("imsg_compose ACK"); + goto done; + } + + err = gotd_imsg_flush(ibuf); +done: + if (err) + log_warnx("sending ACK: %s", err->msg); +} + +void +gotd_imsg_send_nak(struct got_object_id *id, struct imsgbuf *ibuf, + uint32_t peerid, pid_t pid) +{ + const struct got_error *err = NULL; + struct gotd_imsg_nak inak; + char hex[SHA1_DIGEST_STRING_LENGTH]; + + if (log_getverbose() > 0 && + got_sha1_digest_to_str(id->sha1, hex, sizeof(hex))) + log_debug("sending NAK for %s", hex); + + memset(&inak, 0, sizeof(inak)); + memcpy(inak.object_id, id->sha1, SHA1_DIGEST_LENGTH); + + if (imsg_compose(ibuf, GOTD_IMSG_NAK, peerid, pid, -1, + &inak, sizeof(inak)) == -1) { + err = got_error_from_errno("imsg_compose NAK"); + goto done; + } + + err = gotd_imsg_flush(ibuf); +done: + if (err) + log_warnx("sending NAK: %s", err->msg); +} blob - /dev/null blob + b1796ad22b340fe20d6d27d7e25500c103a0e37f (mode 644) --- /dev/null +++ gotd/repo_read.c @@ -0,0 +1,929 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "got_error.h" +#include "got_cancel.h" +#include "got_object.h" +#include "got_repository.h" +#include "got_reference.h" +#include "got_repository_admin.h" + +#include "got_lib_delta.h" +#include "got_lib_object.h" +#include "got_lib_object_idset.h" +#include "got_lib_sha1.h" +#include "got_lib_pack.h" +#include "got_lib_ratelimit.h" +#include "got_lib_pack_create.h" +#include "got_lib_poll.h" + +#include "log.h" +#include "gotd.h" +#include "repo_read.h" + +#ifndef nitems +#define nitems(_a) (sizeof((_a)) / sizeof((_a)[0])) +#endif + +static struct repo_read { + pid_t pid; + const char *title; + struct got_repository *repo; + int *pack_fds; + int *temp_fds; +} repo_read; + +struct repo_read_client { + STAILQ_ENTRY(repo_read_client) entry; + uint32_t id; + int fd; + int delta_cache_fd; + int report_progress; + int pack_pipe[2]; + struct gotd_object_id_array want_ids; + struct gotd_object_id_array have_ids; +}; +STAILQ_HEAD(repo_read_clients, repo_read_client); + +static struct repo_read_clients repo_read_clients[GOTD_CLIENT_TABLE_SIZE]; +static SIPHASH_KEY clients_hash_key; + +static uint64_t +client_hash(uint32_t client_id) +{ + return SipHash24(&clients_hash_key, &client_id, sizeof(client_id)); +} + +static void +add_client(struct repo_read_client *client, uint32_t client_id, int fd) +{ + uint64_t slot; + + client->id = client_id; + client->fd = fd; + client->delta_cache_fd = -1; + client->pack_pipe[0] = -1; + client->pack_pipe[1] = -1; + slot = client_hash(client->id) % nitems(repo_read_clients); + STAILQ_INSERT_HEAD(&repo_read_clients[slot], client, entry); +} + +static struct repo_read_client * +find_client(uint32_t client_id) +{ + uint64_t slot; + struct repo_read_client *c; + + slot = client_hash(client_id) % nitems(repo_read_clients); + STAILQ_FOREACH(c, &repo_read_clients[slot], entry) { + if (c->id == client_id) + return c; + } + + return NULL; +} + +static volatile sig_atomic_t sigint_received; +static volatile sig_atomic_t sigterm_received; + +static void +catch_sigint(int signo) +{ + sigint_received = 1; +} + +static void +catch_sigterm(int signo) +{ + sigterm_received = 1; +} + +static const struct got_error * +check_cancelled(void *arg) +{ + if (sigint_received || sigterm_received) + return got_error(GOT_ERR_CANCELLED); + + return NULL; +} + +static const struct got_error * +send_symref(struct got_reference *symref, struct imsgbuf *ibuf) +{ + const struct got_error *err = NULL; + struct gotd_imsg_symref isymref; + const char *refname = got_ref_get_name(symref); + const char *target = got_ref_get_symref_target(symref); + size_t len; + struct ibuf *wbuf; + struct got_object_id *target_id; + + err = got_ref_resolve(&target_id, repo_read.repo, symref); + if (err) + return err; + + memset(&isymref, 0, sizeof(isymref)); + isymref.name_len = strlen(refname); + isymref.target_len = strlen(target); + memcpy(isymref.target_id, target_id->sha1, sizeof(isymref.target_id)); + + len = sizeof(isymref) + isymref.name_len + isymref.target_len; + if (len > MAX_IMSGSIZE - IMSG_HEADER_SIZE) { + err = got_error(GOT_ERR_NO_SPACE); + goto done; + } + + wbuf = imsg_create(ibuf, GOTD_IMSG_SYMREF, 0, 0, len); + if (wbuf == NULL) { + err = got_error_from_errno("imsg_create SYMREF"); + goto done; + } + + if (imsg_add(wbuf, &isymref, sizeof(isymref)) == -1) { + err = got_error_from_errno("imsg_add SYMREF"); + goto done; + } + if (imsg_add(wbuf, refname, isymref.name_len) == -1) { + err = got_error_from_errno("imsg_add SYMREF"); + goto done; + } + if (imsg_add(wbuf, target, isymref.target_len) == -1) { + err = got_error_from_errno("imsg_add SYMREF"); + goto done; + } + + wbuf->fd = -1; + imsg_close(ibuf, wbuf); +done: + free(target_id); + return err; +} + +static const struct got_error * +send_peeled_tag_ref(struct got_reference *ref, struct got_object *obj, + struct imsgbuf *ibuf) +{ + const struct got_error *err = NULL; + struct got_tag_object *tag; + size_t namelen, len; + char *peeled_refname = NULL; + struct got_object_id *id; + struct ibuf *wbuf; + + err = got_object_tag_open(&tag, repo_read.repo, obj); + if (err) + return err; + + if (asprintf(&peeled_refname, "%s^{}", got_ref_get_name(ref)) == -1) { + err = got_error_from_errno("asprintf"); + goto done; + } + + id = got_object_tag_get_object_id(tag); + namelen = strlen(peeled_refname); + + len = sizeof(struct gotd_imsg_ref) + namelen; + if (len > MAX_IMSGSIZE - IMSG_HEADER_SIZE) { + err = got_error(GOT_ERR_NO_SPACE); + goto done; + } + + wbuf = imsg_create(ibuf, GOTD_IMSG_REF, PROC_REPO_READ, + repo_read.pid, len); + if (wbuf == NULL) { + err = got_error_from_errno("imsg_create MREF"); + goto done; + } + + /* Keep in sync with struct gotd_imsg_ref definition. */ + if (imsg_add(wbuf, id->sha1, SHA1_DIGEST_LENGTH) == -1) { + err = got_error_from_errno("imsg_add REF"); + goto done; + } + if (imsg_add(wbuf, &namelen, sizeof(namelen)) == -1) { + err = got_error_from_errno("imsg_add REF"); + goto done; + } + if (imsg_add(wbuf, peeled_refname, namelen) == -1) { + err = got_error_from_errno("imsg_add REF"); + goto done; + } + + wbuf->fd = -1; + imsg_close(ibuf, wbuf); +done: + got_object_tag_close(tag); + return err; +} + +static const struct got_error * +send_ref(struct got_reference *ref, struct imsgbuf *ibuf) +{ + const struct got_error *err; + const char *refname = got_ref_get_name(ref); + size_t namelen; + struct got_object_id *id = NULL; + struct got_object *obj = NULL; + size_t len; + struct ibuf *wbuf; + + namelen = strlen(refname); + + len = sizeof(struct gotd_imsg_ref) + namelen; + if (len > MAX_IMSGSIZE - IMSG_HEADER_SIZE) + return got_error(GOT_ERR_NO_SPACE); + + err = got_ref_resolve(&id, repo_read.repo, ref); + if (err) + return err; + + wbuf = imsg_create(ibuf, GOTD_IMSG_REF, PROC_REPO_READ, + repo_read.pid, len); + if (wbuf == NULL) { + err = got_error_from_errno("imsg_create REF"); + goto done; + } + + /* Keep in sync with struct gotd_imsg_ref definition. */ + if (imsg_add(wbuf, id->sha1, SHA1_DIGEST_LENGTH) == -1) + return got_error_from_errno("imsg_add REF"); + if (imsg_add(wbuf, &namelen, sizeof(namelen)) == -1) + return got_error_from_errno("imsg_add REF"); + if (imsg_add(wbuf, refname, namelen) == -1) + return got_error_from_errno("imsg_add REF"); + + wbuf->fd = -1; + imsg_close(ibuf, wbuf); + + err = got_object_open(&obj, repo_read.repo, id); + if (err) + goto done; + if (obj->type == GOT_OBJ_TYPE_TAG) + err = send_peeled_tag_ref(ref, obj, ibuf); +done: + if (obj) + got_object_close(obj); + free(id); + return err; +} + +static const struct got_error * +list_refs(struct repo_read_client **client, struct imsg *imsg) +{ + const struct got_error *err; + struct got_reflist_head refs; + struct got_reflist_entry *re; + struct gotd_imsg_list_refs_internal ireq; + size_t datalen; + struct gotd_imsg_reflist irefs; + struct imsgbuf ibuf; + int client_fd = imsg->fd; + + TAILQ_INIT(&refs); + + if (client_fd == -1) + return got_error(GOT_ERR_PRIVSEP_NO_FD); + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(ireq)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&ireq, imsg->data, sizeof(ireq)); + + *client = find_client(ireq.client_id); + if (*client) + return got_error_msg(GOT_ERR_CLIENT_ID, "duplicate client ID"); + + *client = calloc(1, sizeof(**client)); + if (*client == NULL) + return got_error_from_errno("calloc"); + add_client(*client, ireq.client_id, client_fd); + + imsg_init(&ibuf, client_fd); + + err = got_ref_list(&refs, repo_read.repo, "", + got_ref_cmp_by_name, NULL); + if (err) + return err; + + memset(&irefs, 0, sizeof(irefs)); + TAILQ_FOREACH(re, &refs, entry) { + struct got_object_id *id; + int obj_type; + + if (got_ref_is_symbolic(re->ref)) { + const char *refname = got_ref_get_name(re->ref); + if (strcmp(refname, GOT_REF_HEAD) == 0) + irefs.nrefs++; + continue; + } + + irefs.nrefs++; + + /* Account for a peeled tag refs. */ + err = got_ref_resolve(&id, repo_read.repo, re->ref); + if (err) + goto done; + err = got_object_get_type(&obj_type, repo_read.repo, id); + free(id); + if (err) + goto done; + if (obj_type == GOT_OBJ_TYPE_TAG) + irefs.nrefs++; + } + + if (imsg_compose(&ibuf, GOTD_IMSG_REFLIST, PROC_REPO_READ, + repo_read.pid, -1, &irefs, sizeof(irefs)) == -1) { + err = got_error_from_errno("imsg_compose REFLIST"); + goto done; + } + + /* + * Send the HEAD symref first. In Git-protocol versions < 2 + * the HEAD symref must be announced on the initial line of + * the server's ref advertisement. + * For now, we do not advertise symrefs other than HEAD. + */ + TAILQ_FOREACH(re, &refs, entry) { + if (!got_ref_is_symbolic(re->ref) || + strcmp(got_ref_get_name(re->ref), GOT_REF_HEAD) != 0) + continue; + err = send_symref(re->ref, &ibuf); + if (err) + goto done; + break; + } + TAILQ_FOREACH(re, &refs, entry) { + if (got_ref_is_symbolic(re->ref)) + continue; + err = send_ref(re->ref, &ibuf); + if (err) + goto done; + } + + err = gotd_imsg_flush(&ibuf); +done: + got_ref_list_free(&refs); + imsg_clear(&ibuf); + return err; +} + +static const struct got_error * +record_object_id(struct gotd_object_id_array *array, struct got_object_id *id) +{ + const size_t alloc_chunksz = 256; + + if (array->ids == NULL) { + array->ids = reallocarray(NULL, alloc_chunksz, + sizeof(*array->ids)); + if (array->ids == NULL) + return got_error_from_errno("reallocarray"); + array->nalloc = alloc_chunksz; + array->nids = 0; + } else if (array->nalloc <= array->nids) { + struct got_object_id **new; + new = recallocarray(array->ids, array->nalloc, + array->nalloc + alloc_chunksz, sizeof(*new)); + if (new == NULL) + return got_error_from_errno("recallocarray"); + array->ids = new; + array->nalloc += alloc_chunksz; + } + + array->ids[array->nids] = got_object_id_dup(id); + if (array->ids[array->nids] == NULL) + return got_error_from_errno("got_object_id_dup"); + array->nids++; + return NULL; +} + +static void +free_object_ids(struct gotd_object_id_array *array) +{ + size_t i; + + for (i = 0; i < array->nids; i++) + free(array->ids[i]); + free(array->ids); + + array->ids = NULL; + array->nalloc = 0; + array->nids = 0; +} + +static const struct got_error * +recv_want(struct repo_read_client **client, struct imsg *imsg) +{ + const struct got_error *err; + struct gotd_imsg_want iwant; + size_t datalen; + char hex[SHA1_DIGEST_STRING_LENGTH]; + struct got_object_id id; + int obj_type; + struct imsgbuf ibuf; + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(iwant)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&iwant, imsg->data, sizeof(iwant)); + + memset(&id, 0, sizeof(id)); + memcpy(id.sha1, iwant.object_id, SHA1_DIGEST_LENGTH); + + if (log_getverbose() > 0 && + got_sha1_digest_to_str(id.sha1, hex, sizeof(hex))) + log_debug("client wants %s", hex); + + *client = find_client(iwant.client_id); + if (*client == NULL) + return got_error(GOT_ERR_CLIENT_ID); + + imsg_init(&ibuf, (*client)->fd); + + err = got_object_get_type(&obj_type, repo_read.repo, &id); + if (err) + return err; + + if (obj_type != GOT_OBJ_TYPE_COMMIT && + obj_type != GOT_OBJ_TYPE_TAG) + return got_error(GOT_ERR_OBJ_TYPE); + + err = record_object_id(&(*client)->want_ids, &id); + if (err) + return err; + + gotd_imsg_send_ack(&id, &ibuf, PROC_REPO_READ, repo_read.pid); + imsg_clear(&ibuf); + return err; +} + +static const struct got_error * +recv_have(struct repo_read_client **client, struct imsg *imsg) +{ + const struct got_error *err; + struct gotd_imsg_have ihave; + size_t datalen; + char hex[SHA1_DIGEST_STRING_LENGTH]; + struct got_object_id id; + int obj_type; + struct imsgbuf ibuf; + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(ihave)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&ihave, imsg->data, sizeof(ihave)); + + memset(&id, 0, sizeof(id)); + memcpy(id.sha1, ihave.object_id, SHA1_DIGEST_LENGTH); + + if (log_getverbose() > 0 && + got_sha1_digest_to_str(id.sha1, hex, sizeof(hex))) + log_debug("client has %s", hex); + + *client = find_client(ihave.client_id); + if (*client == NULL) + return got_error(GOT_ERR_CLIENT_ID); + + imsg_init(&ibuf, (*client)->fd); + + err = got_object_get_type(&obj_type, repo_read.repo, &id); + if (err) { + if (err->code == GOT_ERR_NO_OBJ) { + gotd_imsg_send_nak(&id, &ibuf, + PROC_REPO_READ, repo_read.pid); + err = NULL; + } + goto done; + } + + if (obj_type != GOT_OBJ_TYPE_COMMIT && + obj_type != GOT_OBJ_TYPE_TAG) { + gotd_imsg_send_nak(&id, &ibuf, PROC_REPO_READ, repo_read.pid); + err = got_error(GOT_ERR_OBJ_TYPE); + goto done; + } + + err = record_object_id(&(*client)->have_ids, &id); + if (err) + return err; + + gotd_imsg_send_ack(&id, &ibuf, PROC_REPO_READ, repo_read.pid); +done: + imsg_clear(&ibuf); + return err; +} + +struct repo_read_pack_progress_arg { + int report_progress; + struct imsgbuf *ibuf; + int sent_ready; +}; + +static const struct got_error * +pack_progress(void *arg, int ncolored, int nfound, int ntrees, + off_t packfile_size, int ncommits, int nobj_total, int nobj_deltify, + int nobj_written) +{ + struct repo_read_pack_progress_arg *a = arg; + struct gotd_imsg_packfile_progress iprog; + int ret; + + if (!a->report_progress) + return NULL; + if (packfile_size > 0 && a->sent_ready) + return NULL; + + memset(&iprog, 0, sizeof(iprog)); + iprog.ncolored = ncolored; + iprog.nfound = nfound; + iprog.ntrees = ntrees; + iprog.packfile_size = packfile_size; + iprog.ncommits = ncommits; + iprog.nobj_total = nobj_total; + iprog.nobj_deltify = nobj_deltify; + iprog.nobj_written = nobj_written; + + /* Using synchronous writes since we are blocking the event loop. */ + if (packfile_size == 0) { + ret = imsg_compose(a->ibuf, GOTD_IMSG_PACKFILE_PROGRESS, + PROC_REPO_READ, repo_read.pid, -1, &iprog, sizeof(iprog)); + if (ret == -1) { + return got_error_from_errno("imsg compose " + "PACKFILE_PROGRESS"); + } + } else { + a->sent_ready = 1; + ret = imsg_compose(a->ibuf, GOTD_IMSG_PACKFILE_READY, + PROC_REPO_READ, repo_read.pid, -1, &iprog, sizeof(iprog)); + if (ret == -1) { + return got_error_from_errno("imsg compose " + "PACKFILE_READY"); + } + } + + return gotd_imsg_flush(a->ibuf); +} + +static const struct got_error * +receive_delta_cache_fd(struct repo_read_client **client, struct imsg *imsg, + struct gotd_imsgev *iev) +{ + struct gotd_imsg_send_packfile ireq; + size_t datalen; + + log_debug("receving delta cache file"); + + if (imsg->fd == -1) + return got_error(GOT_ERR_PRIVSEP_NO_FD); + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(ireq)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&ireq, imsg->data, sizeof(ireq)); + + *client = find_client(ireq.client_id); + if (*client == NULL) + return got_error(GOT_ERR_CLIENT_ID); + + if ((*client)->delta_cache_fd != -1) + return got_error(GOT_ERR_PRIVSEP_MSG); + + (*client)->delta_cache_fd = imsg->fd; + (*client)->report_progress = ireq.report_progress; + return NULL; +} + +static const struct got_error * +receive_pack_pipe(struct repo_read_client **client, struct imsg *imsg, + struct gotd_imsgev *iev) +{ + struct gotd_imsg_packfile_pipe ireq; + size_t datalen; + + log_debug("receving pack pipe descriptor"); + + if (imsg->fd == -1) + return got_error(GOT_ERR_PRIVSEP_NO_FD); + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(ireq)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&ireq, imsg->data, sizeof(ireq)); + + *client = find_client(ireq.client_id); + if (*client == NULL) + return got_error(GOT_ERR_CLIENT_ID); + if ((*client)->pack_pipe[1] != -1) + return got_error(GOT_ERR_PRIVSEP_MSG); + + if ((*client)->pack_pipe[0] == -1) + (*client)->pack_pipe[0] = imsg->fd; + else + (*client)->pack_pipe[1] = imsg->fd; + + return NULL; +} + +static const struct got_error * +send_packfile(struct repo_read_client *client, struct imsg *imsg, + struct gotd_imsgev *iev) +{ + const struct got_error *err = NULL; + struct gotd_imsg_packfile_done idone; + uint8_t packsha1[SHA1_DIGEST_LENGTH]; + char hex[SHA1_DIGEST_STRING_LENGTH]; + FILE *delta_cache = NULL; + struct imsgbuf ibuf; + struct repo_read_pack_progress_arg pa; + struct got_ratelimit rl; + + log_debug("packfile request received"); + + got_ratelimit_init(&rl, 2, 0); + + if (client->delta_cache_fd == -1 || + client->pack_pipe[0] == -1 || + client->pack_pipe[1] == -1) + return got_error(GOT_ERR_PRIVSEP_NO_FD); + + imsg_init(&ibuf, client->fd); + + /* Send pack file pipe to gotsh(1). */ + if (imsg_compose(&ibuf, GOTD_IMSG_PACKFILE_PIPE, PROC_REPO_READ, + repo_read.pid, client->pack_pipe[1], NULL, 0) == -1) { + err = got_error_from_errno("imsg_compose ACK"); + if (err) + goto done; + } + client->pack_pipe[1] = -1; + err = gotd_imsg_flush(&ibuf); + if (err) + goto done; + + delta_cache = fdopen(client->delta_cache_fd, "w+"); + if (delta_cache == NULL) { + err = got_error_from_errno("fdopen"); + goto done; + } + client->delta_cache_fd = -1; + + memset(&pa, 0, sizeof(pa)); + pa.ibuf = &ibuf; + pa.report_progress = client->report_progress; + + err = got_pack_create(packsha1, client->pack_pipe[0], delta_cache, + client->have_ids.ids, client->have_ids.nids, + client->want_ids.ids, client->want_ids.nids, + repo_read.repo, 0, 1, pack_progress, &pa, &rl, + check_cancelled, NULL); + if (err) + goto done; + + if (log_getverbose() > 0 && + got_sha1_digest_to_str(packsha1, hex, sizeof(hex))) + log_debug("sent pack-%s.pack", hex); + + memset(&idone, 0, sizeof(idone)); + idone.client_id = client->id; + if (gotd_imsg_compose_event(iev, GOTD_IMSG_PACKFILE_DONE, + PROC_REPO_READ, -1, &idone, sizeof(idone)) == -1) + err = got_error_from_errno("imsg compose PACKFILE_DONE"); +done: + if (delta_cache != NULL && fclose(delta_cache) == EOF && err == NULL) + err = got_error_from_errno("fclose"); + imsg_clear(&ibuf); + return err; +} + +static const struct got_error * +recv_disconnect(struct imsg *imsg) +{ + const struct got_error *err = NULL; + struct gotd_imsg_disconnect idisconnect; + size_t datalen; + int client_fd, delta_cache_fd, pipe[2]; + struct repo_read_client *client = NULL; + uint64_t slot; + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(idisconnect)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&idisconnect, imsg->data, sizeof(idisconnect)); + + log_debug("client disconnecting"); + + client = find_client(idisconnect.client_id); + if (client == NULL) + return got_error(GOT_ERR_CLIENT_ID); + + slot = client_hash(client->id) % nitems(repo_read_clients); + STAILQ_REMOVE(&repo_read_clients[slot], client, repo_read_client, + entry); + free_object_ids(&client->have_ids); + free_object_ids(&client->want_ids); + client_fd = client->fd; + delta_cache_fd = client->delta_cache_fd; + pipe[0] = client->pack_pipe[0]; + pipe[1] = client->pack_pipe[1]; + free(client); + if (close(client_fd) == -1) + err = got_error_from_errno("close"); + if (delta_cache_fd != -1 && close(delta_cache_fd) == -1 && err == NULL) + return got_error_from_errno("close"); + if (pipe[0] != -1 && close(pipe[0]) == -1 && err == NULL) + return got_error_from_errno("close"); + if (pipe[1] != -1 && close(pipe[1]) == -1 && err == NULL) + return got_error_from_errno("close"); + return err; +} + +static void +repo_read_dispatch(int fd, short event, void *arg) +{ + const struct got_error *err = NULL; + struct gotd_imsgev *iev = arg; + struct imsgbuf *ibuf = &iev->ibuf; + struct imsg imsg; + ssize_t n; + int shut = 0; + struct repo_read_client *client = NULL; + + if (event & EV_READ) { + if ((n = imsg_read(ibuf)) == -1 && errno != EAGAIN) + fatal("imsg_read error"); + if (n == 0) /* Connection closed. */ + shut = 1; + } + + if (event & EV_WRITE) { + n = msgbuf_write(&ibuf->w); + if (n == -1 && errno != EAGAIN) + fatal("msgbuf_write"); + if (n == 0) /* Connection closed. */ + shut = 1; + } + + while (err == NULL && check_cancelled(NULL) == NULL) { + client = NULL; + if ((n = imsg_get(ibuf, &imsg)) == -1) + fatal("%s: imsg_get", __func__); + if (n == 0) /* No more messages. */ + break; + + switch (imsg.hdr.type) { + case GOTD_IMSG_LIST_REFS_INTERNAL: + err = list_refs(&client, &imsg); + if (err) + log_warnx("%s: ls-refs: %s", repo_read.title, + err->msg); + break; + case GOTD_IMSG_WANT: + err = recv_want(&client, &imsg); + if (err) + log_warnx("%s: want-line: %s", repo_read.title, + err->msg); + break; + case GOTD_IMSG_HAVE: + err = recv_have(&client, &imsg); + if (err) + log_warnx("%s: have-line: %s", repo_read.title, + err->msg); + break; + case GOTD_IMSG_SEND_PACKFILE: + err = receive_delta_cache_fd(&client, &imsg, iev); + if (err) + log_warnx("%s: receiving delta cache: %s", + repo_read.title, err->msg); + break; + case GOTD_IMSG_PACKFILE_PIPE: + err = receive_pack_pipe(&client, &imsg, iev); + if (err) { + log_warnx("%s: receiving pack pipe: %s", + repo_read.title, err->msg); + break; + } + if (client->pack_pipe[1] == -1) + break; + err = send_packfile(client, &imsg, iev); + if (err) + log_warnx("%s: sending packfile: %s", + repo_read.title, err->msg); + break; + case GOTD_IMSG_DISCONNECT: + err = recv_disconnect(&imsg); + if (err) + log_warnx("%s: disconnect: %s", + repo_read.title, err->msg); + break; + default: + log_debug("%s: unexpected imsg %d", repo_read.title, + imsg.hdr.type); + break; + } + + imsg_free(&imsg); + } + + if (!shut && check_cancelled(NULL) == NULL) { + if (err && + gotd_imsg_send_error_event(iev, PROC_REPO_READ, + client ? client->id : 0, err) == -1) { + log_warnx("could not send error to parent: %s", + err->msg); + } + gotd_imsg_event_add(iev); + } else { + /* This pipe is dead. Remove its event handler */ + event_del(&iev->ev); + event_loopexit(NULL); + } +} + +void +repo_read_main(const char *title, int *pack_fds, int *temp_fds) +{ + const struct got_error *err = NULL; + struct gotd_imsgev iev; + + repo_read.title = title; + repo_read.pid = getpid(); + repo_read.pack_fds = pack_fds; + repo_read.temp_fds = temp_fds; + + arc4random_buf(&clients_hash_key, sizeof(clients_hash_key)); + + /* + * Open a repository in the root directory. + * We are already in chroot at this point. + */ + err = got_repo_open(&repo_read.repo, "/", NULL, pack_fds); + if (err) + goto done; + if (!got_repo_is_bare(repo_read.repo)) { + err = got_error_msg(GOT_ERR_NOT_GIT_REPO, + "bare git repository required"); + goto done; + } + + got_repo_temp_fds_set(repo_read.repo, temp_fds); + + signal(SIGINT, catch_sigint); + signal(SIGTERM, catch_sigterm); + signal(SIGPIPE, SIG_IGN); + signal(SIGHUP, SIG_IGN); + + imsg_init(&iev.ibuf, GOTD_SOCK_FILENO); + iev.handler = repo_read_dispatch; + iev.events = EV_READ; + iev.handler_arg = NULL; + event_set(&iev.ev, iev.ibuf.fd, EV_READ, repo_read_dispatch, &iev); + if (event_add(&iev.ev, NULL) == -1) { + err = got_error_from_errno("event_add"); + goto done; + } + + event_dispatch(); +done: + if (err) + log_warnx("%s: %s", title, err->msg); + repo_read_shutdown(); +} + +void +repo_read_shutdown(void) +{ + log_debug("%s: shutting down", repo_read.title); + if (repo_read.repo) + got_repo_close(repo_read.repo); + got_repo_pack_fds_close(repo_read.pack_fds); + got_repo_temp_fds_close(repo_read.temp_fds); + exit(0); +} blob - /dev/null blob + 8dc9ffd3453f593a9b4aae0566ce57742feb0f97 (mode 644) --- /dev/null +++ gotd/repo_read.h @@ -0,0 +1,18 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +void repo_read_main(const char *, int *, int *); +void repo_read_shutdown(void); blob - /dev/null blob + 51a31d0b8608b9351cb8b63bfa93e3d5ab4f87f8 (mode 644) --- /dev/null +++ gotd/repo_write.c @@ -0,0 +1,1474 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "buf.h" + +#include "got_error.h" +#include "got_repository.h" +#include "got_object.h" +#include "got_reference.h" +#include "got_path.h" + +#include "got_lib_delta.h" +#include "got_lib_delta_cache.h" +#include "got_lib_object.h" +#include "got_lib_object_cache.h" +#include "got_lib_ratelimit.h" +#include "got_lib_pack.h" +#include "got_lib_pack_index.h" +#include "got_lib_repository.h" +#include "got_lib_poll.h" + +#include "got_lib_sha1.h" /* XXX temp include for debugging */ + +#include "log.h" +#include "gotd.h" +#include "repo_write.h" + +#ifndef nitems +#define nitems(_a) (sizeof((_a)) / sizeof((_a)[0])) +#endif + +static struct repo_write { + pid_t pid; + const char *title; + struct got_repository *repo; + int *pack_fds; + int *temp_fds; +} repo_write; + +struct gotd_ref_update { + STAILQ_ENTRY(gotd_ref_update) entry; + struct got_reference *ref; + int ref_is_new; + struct got_object_id old_id; + struct got_object_id new_id; +}; +STAILQ_HEAD(gotd_ref_updates, gotd_ref_update); + +struct repo_write_client { + STAILQ_ENTRY(repo_write_client) entry; + uint32_t id; + int fd; + int pack_pipe[2]; + struct got_pack pack; + uint8_t pack_sha1[SHA1_DIGEST_LENGTH]; + int packidx_fd; + struct gotd_ref_updates ref_updates; + int nref_updates; +}; +STAILQ_HEAD(repo_write_clients, repo_write_client); + +static struct repo_write_clients repo_write_clients[GOTD_CLIENT_TABLE_SIZE]; +static SIPHASH_KEY clients_hash_key; + +static uint64_t +client_hash(uint32_t client_id) +{ + return SipHash24(&clients_hash_key, &client_id, sizeof(client_id)); +} + +static void +add_client(struct repo_write_client *client, uint32_t client_id, int fd) +{ + uint64_t slot; + + client->id = client_id; + client->fd = fd; + client->pack_pipe[0] = -1; + client->pack_pipe[1] = -1; + client->packidx_fd = -1; + STAILQ_INIT(&client->ref_updates); + client->nref_updates = 0; + slot = client_hash(client->id) % nitems(repo_write_clients); + STAILQ_INSERT_HEAD(&repo_write_clients[slot], client, entry); +} + +static struct repo_write_client * +find_client(uint32_t client_id) +{ + uint64_t slot; + struct repo_write_client *c; + + slot = client_hash(client_id) % nitems(repo_write_clients); + STAILQ_FOREACH(c, &repo_write_clients[slot], entry) { + if (c->id == client_id) + return c; + } + + return NULL; +} + +static volatile sig_atomic_t sigint_received; +static volatile sig_atomic_t sigterm_received; + +static void +catch_sigint(int signo) +{ + sigint_received = 1; +} + +static void +catch_sigterm(int signo) +{ + sigterm_received = 1; +} + +static const struct got_error * +check_cancelled(void *arg) +{ + if (sigint_received || sigterm_received) + return got_error(GOT_ERR_CANCELLED); + + return NULL; +} + +static const struct got_error * +send_peeled_tag_ref(struct got_reference *ref, struct got_object *obj, + struct imsgbuf *ibuf) +{ + const struct got_error *err = NULL; + struct got_tag_object *tag; + size_t namelen, len; + char *peeled_refname = NULL; + struct got_object_id *id; + struct ibuf *wbuf; + + err = got_object_tag_open(&tag, repo_write.repo, obj); + if (err) + return err; + + if (asprintf(&peeled_refname, "%s^{}", got_ref_get_name(ref)) == -1) { + err = got_error_from_errno("asprintf"); + goto done; + } + + id = got_object_tag_get_object_id(tag); + namelen = strlen(peeled_refname); + + len = sizeof(struct gotd_imsg_ref) + namelen; + if (len > MAX_IMSGSIZE - IMSG_HEADER_SIZE) { + err = got_error(GOT_ERR_NO_SPACE); + goto done; + } + + wbuf = imsg_create(ibuf, GOTD_IMSG_REF, PROC_REPO_WRITE, + repo_write.pid, len); + if (wbuf == NULL) { + err = got_error_from_errno("imsg_create REF"); + goto done; + } + + /* Keep in sync with struct gotd_imsg_ref definition. */ + if (imsg_add(wbuf, id->sha1, SHA1_DIGEST_LENGTH) == -1) { + err = got_error_from_errno("imsg_add REF"); + goto done; + } + if (imsg_add(wbuf, &namelen, sizeof(namelen)) == -1) { + err = got_error_from_errno("imsg_add REF"); + goto done; + } + if (imsg_add(wbuf, peeled_refname, namelen) == -1) { + err = got_error_from_errno("imsg_add REF"); + goto done; + } + + wbuf->fd = -1; + imsg_close(ibuf, wbuf); +done: + got_object_tag_close(tag); + return err; +} + +static const struct got_error * +send_ref(struct got_reference *ref, struct imsgbuf *ibuf) +{ + const struct got_error *err; + const char *refname = got_ref_get_name(ref); + size_t namelen; + struct got_object_id *id = NULL; + struct got_object *obj = NULL; + size_t len; + struct ibuf *wbuf; + + namelen = strlen(refname); + + len = sizeof(struct gotd_imsg_ref) + namelen; + if (len > MAX_IMSGSIZE - IMSG_HEADER_SIZE) + return got_error(GOT_ERR_NO_SPACE); + + err = got_ref_resolve(&id, repo_write.repo, ref); + if (err) + return err; + + wbuf = imsg_create(ibuf, GOTD_IMSG_REF, PROC_REPO_WRITE, + repo_write.pid, len); + if (wbuf == NULL) { + err = got_error_from_errno("imsg_create REF"); + goto done; + } + + /* Keep in sync with struct gotd_imsg_ref definition. */ + if (imsg_add(wbuf, id->sha1, SHA1_DIGEST_LENGTH) == -1) + return got_error_from_errno("imsg_add REF"); + if (imsg_add(wbuf, &namelen, sizeof(namelen)) == -1) + return got_error_from_errno("imsg_add REF"); + if (imsg_add(wbuf, refname, namelen) == -1) + return got_error_from_errno("imsg_add REF"); + + wbuf->fd = -1; + imsg_close(ibuf, wbuf); + + err = got_object_open(&obj, repo_write.repo, id); + if (err) + goto done; + if (obj->type == GOT_OBJ_TYPE_TAG) + err = send_peeled_tag_ref(ref, obj, ibuf); +done: + if (obj) + got_object_close(obj); + free(id); + return err; +} + +static const struct got_error * +list_refs(struct repo_write_client **client, struct imsg *imsg) +{ + const struct got_error *err; + struct got_reflist_head refs; + struct got_reflist_entry *re; + struct gotd_imsg_list_refs_internal ireq; + size_t datalen; + struct gotd_imsg_reflist irefs; + struct imsgbuf ibuf; + int client_fd = imsg->fd; + + TAILQ_INIT(&refs); + + if (client_fd == -1) + return got_error(GOT_ERR_PRIVSEP_NO_FD); + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(ireq)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&ireq, imsg->data, sizeof(ireq)); + + *client = find_client(ireq.client_id); + if (*client) + return got_error_msg(GOT_ERR_CLIENT_ID, "duplicate client ID"); + + *client = calloc(1, sizeof(**client)); + if (*client == NULL) + return got_error_from_errno("calloc"); + add_client(*client, ireq.client_id, client_fd); + + imsg_init(&ibuf, client_fd); + + err = got_ref_list(&refs, repo_write.repo, "", + got_ref_cmp_by_name, NULL); + if (err) + return err; + + memset(&irefs, 0, sizeof(irefs)); + TAILQ_FOREACH(re, &refs, entry) { + struct got_object_id *id; + int obj_type; + + if (got_ref_is_symbolic(re->ref)) + continue; + + irefs.nrefs++; + + /* Account for a peeled tag refs. */ + err = got_ref_resolve(&id, repo_write.repo, re->ref); + if (err) + goto done; + err = got_object_get_type(&obj_type, repo_write.repo, id); + free(id); + if (err) + goto done; + if (obj_type == GOT_OBJ_TYPE_TAG) + irefs.nrefs++; + } + + if (imsg_compose(&ibuf, GOTD_IMSG_REFLIST, PROC_REPO_WRITE, + repo_write.pid, -1, &irefs, sizeof(irefs)) == -1) { + err = got_error_from_errno("imsg_compose REFLIST"); + goto done; + } + + TAILQ_FOREACH(re, &refs, entry) { + if (got_ref_is_symbolic(re->ref)) + continue; + err = send_ref(re->ref, &ibuf); + if (err) + goto done; + } + + err = gotd_imsg_flush(&ibuf); +done: + got_ref_list_free(&refs); + imsg_clear(&ibuf); + return err; +} + +static const struct got_error * +protect_ref_namespace(struct got_reference *ref, const char *namespace) +{ + size_t len = strlen(namespace); + + if (len < 5 || strncmp("refs/", namespace, 5) != 0 || + namespace[len -1] != '/') { + return got_error_fmt(GOT_ERR_BAD_REF_NAME, + "reference namespace '%s'", namespace); + } + + if (strncmp(namespace, got_ref_get_name(ref), len) == 0) + return got_error_fmt(GOT_ERR_REFS_PROTECTED, "%s", namespace); + + return NULL; +} + +static const struct got_error * +recv_ref_update(struct repo_write_client **client, struct imsg *imsg) +{ + const struct got_error *err = NULL; + struct gotd_imsg_ref_update iref; + size_t datalen; + char *refname = NULL; + struct got_reference *ref = NULL; + struct got_object_id *id = NULL; + struct imsgbuf ibuf; + struct gotd_ref_update *ref_update = NULL; + + log_debug("ref-update received"); + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen < sizeof(iref)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&iref, imsg->data, sizeof(iref)); + if (datalen != sizeof(iref) + iref.name_len) + return got_error(GOT_ERR_PRIVSEP_LEN); + + *client = find_client(iref.client_id); + if (*client == NULL) + return got_error(GOT_ERR_CLIENT_ID); + + imsg_init(&ibuf, (*client)->fd); + + refname = malloc(iref.name_len + 1); + if (refname == NULL) + return got_error_from_errno("malloc"); + memcpy(refname, imsg->data + sizeof(iref), iref.name_len); + refname[iref.name_len] = '\0'; + + ref_update = calloc(1, sizeof(*ref_update)); + if (ref_update == NULL) { + err = got_error_from_errno("malloc"); + goto done; + } + + memcpy(ref_update->old_id.sha1, iref.old_id, SHA1_DIGEST_LENGTH); + memcpy(ref_update->new_id.sha1, iref.new_id, SHA1_DIGEST_LENGTH); + + err = got_ref_open(&ref, repo_write.repo, refname, 0); + if (err) { + if (err->code != GOT_ERR_NOT_REF) + goto done; + err = got_ref_alloc(&ref, refname, &ref_update->new_id); + if (err) + goto done; + ref_update->ref_is_new = 1; + } + if (got_ref_is_symbolic(ref)) { + err = got_error_fmt(GOT_ERR_BAD_REF_TYPE, + "'%s' is a symbolic reference and cannot " + "be updated", got_ref_get_name(ref)); + goto done; + } + if (strncmp("refs/", got_ref_get_name(ref), 5) != 0) { + err = got_error_fmt(GOT_ERR_BAD_REF_NAME, + "%s: does not begin with 'refs/'", + got_ref_get_name(ref)); + goto done; + } + + err = protect_ref_namespace(ref, "refs/got/"); + if (err) + goto done; + err = protect_ref_namespace(ref, "refs/remotes/"); + if (err) + goto done; + + if (!ref_update->ref_is_new) { + /* + * Ensure the client's idea of this update is still valid. + * At this point we can only return an error, to prevent + * the client from uploading a pack file which will likely + * have to be discarded. + */ + err = got_ref_resolve(&id, repo_write.repo, ref); + if (err) + goto done; + + if (got_object_id_cmp(id, &ref_update->old_id) != 0) { + err = got_error_fmt(GOT_ERR_REF_BUSY, + "%s has been modified by someone else " + "while transaction was in progress", + got_ref_get_name(ref)); + goto done; + } + } + + gotd_imsg_send_ack(&ref_update->new_id, &ibuf, PROC_REPO_WRITE, + repo_write.pid); + + ref_update->ref = ref; + STAILQ_INSERT_HEAD(&(*client)->ref_updates, ref_update, entry); + (*client)->nref_updates++; + ref = NULL; + ref_update = NULL; +done: + if (ref) + got_ref_close(ref); + free(ref_update); + free(refname); + free(id); + return err; +} + +static const struct got_error * +pack_index_progress(void *arg, uint32_t nobj_total, uint32_t nobj_indexed, + uint32_t nobj_loose, uint32_t nobj_resolved) +{ + int p_indexed = 0, p_resolved = 0; + int nobj_delta = nobj_total - nobj_loose; + + if (nobj_total > 0) + p_indexed = (nobj_indexed * 100) / nobj_total; + + if (nobj_delta > 0) + p_resolved = (nobj_resolved * 100) / nobj_delta; + + if (p_resolved > 0) { + log_debug("indexing %d objects %d%%; resolving %d deltas %d%%", + nobj_total, p_indexed, nobj_delta, p_resolved); + } else + log_debug("indexing %d objects %d%%", nobj_total, p_indexed); + + return NULL; +} + +static const struct got_error * +read_more_pack_stream(int infd, BUF *buf, size_t minsize) +{ + const struct got_error *err = NULL; + uint8_t readahead[65536]; + size_t have, newlen; + + err = got_poll_read_full(infd, &have, + readahead, sizeof(readahead), minsize); + if (err) + return err; + + err = buf_append(&newlen, buf, readahead, have); + if (err) + return err; + return NULL; +} + +static const struct got_error * +copy_object_type_and_size(uint8_t *type, uint64_t *size, int infd, int outfd, + off_t *outsize, BUF *buf, size_t *buf_pos, SHA1_CTX *ctx) +{ + const struct got_error *err = NULL; + uint8_t t = 0; + uint64_t s = 0; + uint8_t sizebuf[8]; + size_t i = 0; + off_t obj_offset = *outsize; + + do { + /* We do not support size values which don't fit in 64 bit. */ + if (i > 9) + return got_error_fmt(GOT_ERR_OBJ_TOO_LARGE, + "packfile offset %llu", obj_offset); + + if (buf_len(buf) - *buf_pos < sizeof(sizebuf[0])) { + err = read_more_pack_stream(infd, buf, + sizeof(sizebuf[0])); + if (err) + return err; + } + + sizebuf[i] = buf_getc(buf, *buf_pos); + *buf_pos += sizeof(sizebuf[i]); + + if (i == 0) { + t = (sizebuf[i] & GOT_PACK_OBJ_SIZE0_TYPE_MASK) >> + GOT_PACK_OBJ_SIZE0_TYPE_MASK_SHIFT; + s = (sizebuf[i] & GOT_PACK_OBJ_SIZE0_VAL_MASK); + } else { + size_t shift = 4 + 7 * (i - 1); + s |= ((sizebuf[i] & GOT_PACK_OBJ_SIZE_VAL_MASK) << + shift); + } + i++; + } while (sizebuf[i - 1] & GOT_PACK_OBJ_SIZE_MORE); + + err = got_pack_hwrite(outfd, sizebuf, i, ctx); + if (err) + return err; + *outsize += i; + + *type = t; + *size = s; + return NULL; +} + +static const struct got_error * +copy_ref_delta(int infd, int outfd, off_t *outsize, BUF *buf, size_t *buf_pos, + SHA1_CTX *ctx) +{ + const struct got_error *err = NULL; + size_t remain = buf_len(buf) - *buf_pos; + + if (remain < SHA1_DIGEST_LENGTH) { + err = read_more_pack_stream(infd, buf, + SHA1_DIGEST_LENGTH - remain); + if (err) + return err; + } + + err = got_pack_hwrite(outfd, buf_get(buf) + *buf_pos, + SHA1_DIGEST_LENGTH, ctx); + if (err) + return err; + + *buf_pos += SHA1_DIGEST_LENGTH; + return NULL; +} + +static const struct got_error * +copy_offset_delta(int infd, int outfd, off_t *outsize, BUF *buf, size_t *buf_pos, + SHA1_CTX *ctx) +{ + const struct got_error *err = NULL; + uint64_t o = 0; + uint8_t offbuf[8]; + size_t i = 0; + off_t obj_offset = *outsize; + + do { + /* We do not support offset values which don't fit in 64 bit. */ + if (i > 8) + return got_error_fmt(GOT_ERR_OBJ_TOO_LARGE, + "packfile offset %llu", obj_offset); + + if (buf_len(buf) - *buf_pos < sizeof(offbuf[0])) { + err = read_more_pack_stream(infd, buf, + sizeof(offbuf[0])); + if (err) + return err; + } + + offbuf[i] = buf_getc(buf, *buf_pos); + *buf_pos += sizeof(offbuf[i]); + + if (i == 0) + o = (offbuf[i] & GOT_PACK_OBJ_DELTA_OFF_VAL_MASK); + else { + o++; + o <<= 7; + o += (offbuf[i] & GOT_PACK_OBJ_DELTA_OFF_VAL_MASK); + } + i++; + } while (offbuf[i - 1] & GOT_PACK_OBJ_DELTA_OFF_MORE); + + if (o < sizeof(struct got_packfile_hdr) || o > *outsize) + return got_error(GOT_ERR_PACK_OFFSET); + + err = got_pack_hwrite(outfd, offbuf, i, ctx); + if (err) + return err; + + *outsize += i; + return NULL; +} + +static const struct got_error * +copy_zstream(int infd, int outfd, off_t *outsize, BUF *buf, size_t *buf_pos, + SHA1_CTX *ctx) +{ + const struct got_error *err = NULL; + z_stream z; + int zret; + char voidbuf[1024]; + size_t consumed_total = 0; + off_t zstream_offset = *outsize; + + memset(&z, 0, sizeof(z)); + + z.zalloc = Z_NULL; + z.zfree = Z_NULL; + zret = inflateInit(&z); + if (zret != Z_OK) { + if (zret == Z_ERRNO) + return got_error_from_errno("inflateInit"); + if (zret == Z_MEM_ERROR) { + errno = ENOMEM; + return got_error_from_errno("inflateInit"); + } + return got_error_msg(GOT_ERR_DECOMPRESSION, + "inflateInit failed"); + } + + while (zret != Z_STREAM_END) { + size_t last_total_in, consumed; + + /* + * Decompress into the void. Object data will be parsed + * later, when the pack file is indexed. For now, we just + * want to locate the end of the compressed stream. + */ + while (zret != Z_STREAM_END && buf_len(buf) - *buf_pos > 0) { + last_total_in = z.total_in; + z.next_in = buf_get(buf) + *buf_pos; + z.avail_in = buf_len(buf) - *buf_pos; + z.next_out = voidbuf; + z.avail_out = sizeof(voidbuf); + + zret = inflate(&z, Z_SYNC_FLUSH); + if (zret != Z_OK && zret != Z_BUF_ERROR && + zret != Z_STREAM_END) { + err = got_error_fmt(GOT_ERR_DECOMPRESSION, + "packfile offset %llu", zstream_offset); + goto done; + } + consumed = z.total_in - last_total_in; + + err = got_pack_hwrite(outfd, buf_get(buf) + *buf_pos, + consumed, ctx); + if (err) + goto done; + + err = buf_discard(buf, *buf_pos + consumed); + if (err) + goto done; + *buf_pos = 0; + + consumed_total += consumed; + } + + if (zret != Z_STREAM_END) { + err = read_more_pack_stream(infd, buf, 1); + if (err) + goto done; + } + } + + if (err == NULL) + *outsize += consumed_total; +done: + inflateEnd(&z); + return err; +} + +static const struct got_error * +validate_object_type(int obj_type) +{ + switch (obj_type) { + case GOT_OBJ_TYPE_BLOB: + case GOT_OBJ_TYPE_COMMIT: + case GOT_OBJ_TYPE_TREE: + case GOT_OBJ_TYPE_TAG: + case GOT_OBJ_TYPE_REF_DELTA: + case GOT_OBJ_TYPE_OFFSET_DELTA: + return NULL; + default: + break; + } + + return got_error(GOT_ERR_OBJ_TYPE); +} + +static const struct got_error * +recv_packdata(off_t *outsize, uint8_t *sha1, int infd, int outfd) +{ + const struct got_error *err; + struct got_packfile_hdr hdr; + size_t have; + uint32_t nobj, nhave = 0; + SHA1_CTX ctx; + uint8_t expected_sha1[SHA1_DIGEST_LENGTH]; + char hex[SHA1_DIGEST_STRING_LENGTH]; + BUF *buf = NULL; + size_t buf_pos = 0, remain; + ssize_t w; + + *outsize = 0; + SHA1Init(&ctx); + + err = got_poll_read_full(infd, &have, &hdr, sizeof(hdr), sizeof(hdr)); + if (err) + return err; + if (have != sizeof(hdr)) + return got_error_msg(GOT_ERR_BAD_PACKFILE, "short pack file"); + *outsize += have; + + if (hdr.signature != htobe32(GOT_PACKFILE_SIGNATURE)) + return got_error_msg(GOT_ERR_BAD_PACKFILE, + "bad packfile signature"); + if (hdr.version != htobe32(GOT_PACKFILE_VERSION)) + return got_error_msg(GOT_ERR_BAD_PACKFILE, + "bad packfile version"); + + nobj = be32toh(hdr.nobjects); + if (nobj == 0) + return got_error_msg(GOT_ERR_BAD_PACKFILE, + "bad packfile with zero objects"); + + log_debug("expecting %d objects", nobj); + + err = got_pack_hwrite(outfd, &hdr, sizeof(hdr), &ctx); + if (err) + return err; + + err = buf_alloc(&buf, 65536); + if (err) + return err; + + while (nhave != nobj) { + uint8_t obj_type; + uint64_t obj_size; + + err = copy_object_type_and_size(&obj_type, &obj_size, + infd, outfd, outsize, buf, &buf_pos, &ctx); + if (err) + goto done; + + err = validate_object_type(obj_type); + if (err) + goto done; + + if (obj_type == GOT_OBJ_TYPE_REF_DELTA) { + err = copy_ref_delta(infd, outfd, outsize, + buf, &buf_pos, &ctx); + if (err) + goto done; + } else if (obj_type == GOT_OBJ_TYPE_OFFSET_DELTA) { + err = copy_offset_delta(infd, outfd, outsize, + buf, &buf_pos, &ctx); + if (err) + goto done; + } + + err = copy_zstream(infd, outfd, outsize, buf, &buf_pos, &ctx); + if (err) + goto done; + + nhave++; + } + + log_debug("received %u objects", nobj); + + SHA1Final(expected_sha1, &ctx); + + remain = buf_len(buf) - buf_pos; + if (remain < SHA1_DIGEST_LENGTH) { + err = read_more_pack_stream(infd, buf, + SHA1_DIGEST_LENGTH - remain); + if (err) + return err; + } + + got_sha1_digest_to_str(expected_sha1, hex, sizeof(hex)); + log_debug("expect SHA1: %s", hex); + got_sha1_digest_to_str(buf_get(buf) + buf_pos, hex, sizeof(hex)); + log_debug("actual SHA1: %s", hex); + + if (memcmp(buf_get(buf) + buf_pos, expected_sha1, + SHA1_DIGEST_LENGTH) != 0) { + err = got_error(GOT_ERR_PACKFILE_CSUM); + goto done; + } + + memcpy(sha1, expected_sha1, SHA1_DIGEST_LENGTH); + + w = write(outfd, expected_sha1, SHA1_DIGEST_LENGTH); + if (w == -1) { + err = got_error_from_errno("write"); + goto done; + } + if (w != SHA1_DIGEST_LENGTH) { + err = got_error(GOT_ERR_IO); + goto done; + } + + *outsize += SHA1_DIGEST_LENGTH; + + if (fsync(outfd) == -1) { + err = got_error_from_errno("fsync"); + goto done; + } + if (lseek(outfd, 0L, SEEK_SET) == -1) { + err = got_error_from_errno("lseek"); + goto done; + } +done: + buf_free(buf); + return err; +} + +static const struct got_error * +report_pack_status(struct repo_write_client *client, + const struct got_error *unpack_err) +{ + const struct got_error *err = NULL; + struct gotd_imsg_packfile_status istatus; + struct ibuf *wbuf; + struct imsgbuf ibuf; + const char *unpack_ok = "unpack ok\n"; + size_t len; + + imsg_init(&ibuf, client->fd); + + if (unpack_err) + istatus.reason_len = strlen(unpack_err->msg); + else + istatus.reason_len = strlen(unpack_ok); + + len = sizeof(istatus) + istatus.reason_len; + wbuf = imsg_create(&ibuf, GOTD_IMSG_PACKFILE_STATUS, PROC_REPO_WRITE, + repo_write.pid, len); + if (wbuf == NULL) { + err = got_error_from_errno("imsg_create PACKFILE_STATUS"); + goto done; + } + + if (imsg_add(wbuf, &istatus, sizeof(istatus)) == -1) { + err = got_error_from_errno("imsg_add PACKFILE_STATUS"); + goto done; + } + + if (imsg_add(wbuf, err ? err->msg : unpack_ok, + istatus.reason_len) == -1) { + err = got_error_from_errno("imsg_add PACKFILE_STATUS"); + goto done; + } + + wbuf->fd = -1; + imsg_close(&ibuf, wbuf); + + err = gotd_imsg_flush(&ibuf); +done: + imsg_clear(&ibuf); + return err; +} + +static const struct got_error * +recv_packfile(struct repo_write_client **client, struct imsg *imsg) +{ + const struct got_error *err = NULL, *unpack_err; + struct gotd_imsg_recv_packfile ireq; + FILE *tempfiles[3] = { NULL, NULL, NULL }; + struct repo_tempfile { + int fd; + int idx; + } repo_tempfiles[3] = { { - 1, - 1 }, { - 1, - 1 }, { - 1, - 1 }, }; + int i; + size_t datalen; + struct imsgbuf ibuf; + struct got_ratelimit rl; + struct got_pack *pack = NULL; + off_t pack_filesize = 0; + + log_debug("packfile request received"); + + got_ratelimit_init(&rl, 2, 0); + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(ireq)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&ireq, imsg->data, sizeof(ireq)); + + *client = find_client(ireq.client_id); + if (*client == NULL || STAILQ_EMPTY(&(*client)->ref_updates)) + return got_error(GOT_ERR_CLIENT_ID); + + if ((*client)->pack_pipe[0] == -1 || + (*client)->pack_pipe[1] == -1 || + (*client)->packidx_fd == -1) + return got_error(GOT_ERR_PRIVSEP_NO_FD); + + imsg_init(&ibuf, (*client)->fd); + + if (imsg->fd == -1) + return got_error(GOT_ERR_PRIVSEP_NO_FD); + + pack = &(*client)->pack; + memset(pack, 0, sizeof(*pack)); + pack->fd = imsg->fd; + err = got_delta_cache_alloc(&pack->delta_cache); + if (err) + return err; + + for (i = 0; i < nitems(repo_tempfiles); i++) { + struct repo_tempfile *t = &repo_tempfiles[i]; + err = got_repo_temp_fds_get(&t->fd, &t->idx, repo_write.repo); + if (err) + goto done; + } + + for (i = 0; i < nitems(tempfiles); i++) { + int fd = dup(repo_tempfiles[i].fd); + FILE *f; + if (fd == -1) { + err = got_error_from_errno("dup"); + goto done; + } + f = fdopen(fd, "w+"); + if (f == NULL) { + err = got_error_from_errno("dup"); + close(fd); + goto done; + } + tempfiles[i] = f; + } + + /* Send pack file pipe to gotsh(1). */ + if (imsg_compose(&ibuf, GOTD_IMSG_RECV_PACKFILE, PROC_REPO_WRITE, + repo_write.pid, (*client)->pack_pipe[1], NULL, 0) == -1) { + (*client)->pack_pipe[1] = -1; + err = got_error_from_errno("imsg_compose ACK"); + if (err) + goto done; + } + (*client)->pack_pipe[1] = -1; + err = gotd_imsg_flush(&ibuf); + if (err) + goto done; + + log_debug("receiving pack data"); + unpack_err = recv_packdata(&pack_filesize, (*client)->pack_sha1, + (*client)->pack_pipe[0], pack->fd); + if (ireq.report_status) { + err = report_pack_status(*client, unpack_err); + if (err) { + /* Git clients hang up after sending the pack file. */ + if (err->code == GOT_ERR_EOF) + err = NULL; + } + } + if (unpack_err) + err = unpack_err; + if (err) + goto done; + + log_debug("pack data received"); + + /* XXX size_t vs off_t, both should be off_t */ + if (pack_filesize >= SIZE_MAX) { + err = got_error_msg(GOT_ERR_BAD_PACKFILE, + "pack file too large"); + goto done; + } + pack->filesize = pack_filesize; + + log_debug("begin indexing pack (%zu bytes in size)", pack->filesize); + err = got_pack_index(pack, (*client)->packidx_fd, + tempfiles[0], tempfiles[1], tempfiles[2], (*client)->pack_sha1, + pack_index_progress, NULL, &rl); + if (err) + goto done; + log_debug("done indexing pack"); + + if (fsync((*client)->packidx_fd) == -1) { + err = got_error_from_errno("fsync"); + goto done; + } + if (lseek((*client)->packidx_fd, 0L, SEEK_SET) == -1) + err = got_error_from_errno("lseek"); +done: + if (close((*client)->pack_pipe[0]) == -1 && err == NULL) + err = got_error_from_errno("close"); + (*client)->pack_pipe[0] = -1; + for (i = 0; i < nitems(repo_tempfiles); i++) { + struct repo_tempfile *t = &repo_tempfiles[i]; + if (t->idx != -1) + got_repo_temp_fds_put(t->idx, repo_write.repo); + } + for (i = 0; i < nitems(tempfiles); i++) { + if (tempfiles[i] && fclose(tempfiles[i]) == EOF && err == NULL) + err = got_error_from_errno("fclose"); + } + if (err) + got_pack_close(pack); + imsg_clear(&ibuf); + return err; +} + +static const struct got_error * +verify_packfile(struct repo_write_client *client) +{ + const struct got_error *err = NULL, *close_err; + struct gotd_ref_update *ref_update; + struct got_packidx *packidx = NULL; + struct stat sb; + char *id_str = NULL; + int idx = -1; + + if (STAILQ_EMPTY(&client->ref_updates)) { + return got_error_msg(GOT_ERR_BAD_REQUEST, + "cannot verify pack file without any ref-updates"); + } + + if (client->pack.fd == -1) { + return got_error_msg(GOT_ERR_BAD_REQUEST, + "invalid pack file handle during pack verification"); + } + if (client->packidx_fd == -1) { + return got_error_msg(GOT_ERR_BAD_REQUEST, + "invalid pack index handle during pack verification"); + } + + if (fstat(client->packidx_fd, &sb) == -1) + return got_error_from_errno("pack index fstat"); + + packidx = malloc(sizeof(*packidx)); + memset(packidx, 0, sizeof(*packidx)); + packidx->fd = client->packidx_fd; + client->packidx_fd = -1; + packidx->len = sb.st_size; + + err = got_packidx_init_hdr(packidx, 1, client->pack.filesize); + if (err) + return err; + + STAILQ_FOREACH(ref_update, &client->ref_updates, entry) { + err = got_object_id_str(&id_str, &ref_update->new_id); + if (err) + goto done; + + idx = got_packidx_get_object_idx(packidx, &ref_update->new_id); + if (idx == -1) { + err = got_error_fmt(GOT_ERR_BAD_PACKFILE, + "advertised object %s is missing from pack file", + id_str); + goto done; + } + } + +done: + close_err = got_packidx_close(packidx); + if (close_err && err == NULL) + err = close_err; + free(id_str); + return err; +} + +static const struct got_error * +install_packfile(struct repo_write_client *client, struct gotd_imsgev *iev) +{ + struct gotd_imsg_packfile_install inst; + int ret; + + memset(&inst, 0, sizeof(inst)); + inst.client_id = client->id; + memcpy(inst.pack_sha1, client->pack_sha1, SHA1_DIGEST_LENGTH); + + ret = gotd_imsg_compose_event(iev, GOTD_IMSG_PACKFILE_INSTALL, + PROC_REPO_WRITE, -1, &inst, sizeof(inst)); + if (ret == -1) + return got_error_from_errno("imsg_compose PACKFILE_INSTALL"); + + return NULL; +} + +static const struct got_error * +send_ref_updates_start(struct repo_write_client *client, int nref_updates, + struct gotd_imsgev *iev) +{ + struct gotd_imsg_ref_updates_start istart; + int ret; + + memset(&istart, 0, sizeof(istart)); + istart.nref_updates = nref_updates; + istart.client_id = client->id; + + ret = gotd_imsg_compose_event(iev, GOTD_IMSG_REF_UPDATES_START, + PROC_REPO_WRITE, -1, &istart, sizeof(istart)); + if (ret == -1) + return got_error_from_errno("imsg_compose REF_UPDATES_START"); + + return NULL; +} + + +static const struct got_error * +send_ref_update(struct repo_write_client *client, + struct gotd_ref_update *ref_update, struct gotd_imsgev *iev) +{ + struct gotd_imsg_ref_update iref; + const char *refname = got_ref_get_name(ref_update->ref); + struct ibuf *wbuf; + size_t len; + + memset(&iref, 0, sizeof(iref)); + memcpy(iref.old_id, ref_update->old_id.sha1, SHA1_DIGEST_LENGTH); + memcpy(iref.new_id, ref_update->new_id.sha1, SHA1_DIGEST_LENGTH); + iref.ref_is_new = ref_update->ref_is_new; + iref.client_id = client->id; + iref.name_len = strlen(refname); + + len = sizeof(iref) + iref.name_len; + wbuf = imsg_create(&iev->ibuf, GOTD_IMSG_REF_UPDATE, PROC_REPO_WRITE, + repo_write.pid, len); + if (wbuf == NULL) + return got_error_from_errno("imsg_create REF_UPDATE"); + + if (imsg_add(wbuf, &iref, sizeof(iref)) == -1) + return got_error_from_errno("imsg_add REF_UPDATE"); + if (imsg_add(wbuf, refname, iref.name_len) == -1) + return got_error_from_errno("imsg_add REF_UPDATE"); + + wbuf->fd = -1; + imsg_close(&iev->ibuf, wbuf); + + gotd_imsg_event_add(iev); + return NULL; +} + +static const struct got_error * +update_refs(struct repo_write_client *client, struct gotd_imsgev *iev) +{ + const struct got_error *err = NULL; + struct gotd_ref_update *ref_update; + + err = send_ref_updates_start(client, client->nref_updates, iev); + if (err) + return err; + + STAILQ_FOREACH(ref_update, &client->ref_updates, entry) { + err = send_ref_update(client, ref_update, iev); + if (err) + goto done; + } +done: + return err; +} + +static const struct got_error * +recv_disconnect(struct imsg *imsg) +{ + const struct got_error *err = NULL; + struct gotd_imsg_disconnect idisconnect; + size_t datalen; + int client_fd = -1, pipe0 = -1, pipe1 = - 1, idxfd = -1; + struct repo_write_client *client = NULL; + uint64_t slot; + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(idisconnect)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&idisconnect, imsg->data, sizeof(idisconnect)); + + log_debug("client disconnecting"); + + client = find_client(idisconnect.client_id); + if (client == NULL) + return got_error(GOT_ERR_CLIENT_ID); + + slot = client_hash(client->id) % nitems(repo_write_clients); + STAILQ_REMOVE(&repo_write_clients[slot], client, repo_write_client, + entry); + while (!STAILQ_EMPTY(&client->ref_updates)) { + struct gotd_ref_update *ref_update; + ref_update = STAILQ_FIRST(&client->ref_updates); + STAILQ_REMOVE_HEAD(&client->ref_updates, entry); + got_ref_close(ref_update->ref); + free(ref_update); + } + err = got_pack_close(&client->pack); + client_fd = client->fd; + pipe0 = client->pack_pipe[0]; + pipe1 = client->pack_pipe[1]; + idxfd = client->packidx_fd; + free(client); + if (client_fd != -1 && close(client_fd) == -1) + err = got_error_from_errno("close"); + if (pipe0 != -1 && close(pipe0) == -1 && err == NULL) + err = got_error_from_errno("close"); + if (pipe1 != -1 && close(pipe1) == -1 && err == NULL) + err = got_error_from_errno("close"); + if (idxfd != -1 && close(idxfd) == -1 && err == NULL) + err = got_error_from_errno("close"); + return err; +} + +static const struct got_error * +receive_pack_pipe(struct repo_write_client **client, struct imsg *imsg, + struct gotd_imsgev *iev) +{ + struct gotd_imsg_packfile_pipe ireq; + size_t datalen; + + log_debug("receving pack pipe descriptor"); + + if (imsg->fd == -1) + return got_error(GOT_ERR_PRIVSEP_NO_FD); + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(ireq)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&ireq, imsg->data, sizeof(ireq)); + + *client = find_client(ireq.client_id); + if (*client == NULL) + return got_error(GOT_ERR_CLIENT_ID); + if ((*client)->pack_pipe[1] != -1) + return got_error(GOT_ERR_PRIVSEP_MSG); + + if ((*client)->pack_pipe[0] == -1) + (*client)->pack_pipe[0] = imsg->fd; + else + (*client)->pack_pipe[1] = imsg->fd; + + return NULL; +} + +static const struct got_error * +receive_pack_idx(struct repo_write_client **client, struct imsg *imsg, + struct gotd_imsgev *iev) +{ + struct gotd_imsg_packidx_file ireq; + size_t datalen; + + log_debug("receving pack index output file"); + + if (imsg->fd == -1) + return got_error(GOT_ERR_PRIVSEP_NO_FD); + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(ireq)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&ireq, imsg->data, sizeof(ireq)); + + *client = find_client(ireq.client_id); + if (*client == NULL) + return got_error(GOT_ERR_CLIENT_ID); + if ((*client)->packidx_fd != -1) + return got_error(GOT_ERR_PRIVSEP_MSG); + + (*client)->packidx_fd = imsg->fd; + return NULL; +} + +static void +repo_write_dispatch(int fd, short event, void *arg) +{ + const struct got_error *err = NULL; + struct gotd_imsgev *iev = arg; + struct imsgbuf *ibuf = &iev->ibuf; + struct imsg imsg; + struct repo_write_client *client = NULL; + ssize_t n; + int shut = 0; + + if (event & EV_READ) { + if ((n = imsg_read(ibuf)) == -1 && errno != EAGAIN) + fatal("imsg_read error"); + if (n == 0) /* Connection closed. */ + shut = 1; + } + + if (event & EV_WRITE) { + n = msgbuf_write(&ibuf->w); + if (n == -1 && errno != EAGAIN) + fatal("msgbuf_write"); + if (n == 0) /* Connection closed. */ + shut = 1; + } + + for (;;) { + if ((n = imsg_get(ibuf, &imsg)) == -1) + fatal("%s: imsg_get error", __func__); + if (n == 0) /* No more messages. */ + break; + + switch (imsg.hdr.type) { + case GOTD_IMSG_LIST_REFS_INTERNAL: + err = list_refs(&client, &imsg); + if (err) + log_warnx("%s: ls-refs: %s", repo_write.title, + err->msg); + break; + case GOTD_IMSG_REF_UPDATE: + err = recv_ref_update(&client, &imsg); + if (err) + log_warnx("%s: ref-update: %s", + repo_write.title, err->msg); + break; + case GOTD_IMSG_PACKFILE_PIPE: + err = receive_pack_pipe(&client, &imsg, iev); + if (err) { + log_warnx("%s: receiving pack pipe: %s", + repo_write.title, err->msg); + break; + } + break; + case GOTD_IMSG_PACKIDX_FILE: + err = receive_pack_idx(&client, &imsg, iev); + if (err) { + log_warnx("%s: receiving pack index: %s", + repo_write.title, err->msg); + break; + } + break; + case GOTD_IMSG_RECV_PACKFILE: + err = recv_packfile(&client, &imsg); + if (err) { + log_warnx("%s: receive packfile: %s", + repo_write.title, err->msg); + break; + } + err = verify_packfile(client); + if (err) { + log_warnx("%s: verify packfile: %s", + repo_write.title, err->msg); + break; + } + err = install_packfile(client, iev); + if (err) { + log_warnx("%s: install packfile: %s", + repo_write.title, err->msg); + break; + } + err = update_refs(client, iev); + if (err) { + log_warnx("%s: update refs: %s", + repo_write.title, err->msg); + } + break; + case GOTD_IMSG_DISCONNECT: + err = recv_disconnect(&imsg); + if (err) + log_warnx("%s: disconnect: %s", + repo_write.title, err->msg); + break; + default: + log_debug("%s: unexpected imsg %d", repo_write.title, + imsg.hdr.type); + break; + } + + imsg_free(&imsg); + } + + if (!shut && check_cancelled(NULL) == NULL) { + if (err && + gotd_imsg_send_error_event(iev, PROC_REPO_WRITE, + client ? client->id : 0, err) == -1) { + log_warnx("could not send error to parent: %s", + err->msg); + } + gotd_imsg_event_add(iev); + } else { + /* This pipe is dead. Remove its event handler */ + event_del(&iev->ev); + event_loopexit(NULL); + } +} + +void +repo_write_main(const char *title, int *pack_fds, int *temp_fds) +{ + const struct got_error *err = NULL; + struct gotd_imsgev iev; + + repo_write.title = title; + repo_write.pid = getpid(); + repo_write.pack_fds = pack_fds; + repo_write.temp_fds = temp_fds; + + arc4random_buf(&clients_hash_key, sizeof(clients_hash_key)); + + /* + * Open a repository in the root directory. + * We are already in chroot at this point. + */ + err = got_repo_open(&repo_write.repo, "/", NULL, pack_fds); + if (err) + goto done; + if (!got_repo_is_bare(repo_write.repo)) { + err = got_error_msg(GOT_ERR_NOT_GIT_REPO, + "bare git repository required"); + goto done; + } + + got_repo_temp_fds_set(repo_write.repo, temp_fds); + + signal(SIGINT, catch_sigint); + signal(SIGTERM, catch_sigterm); + signal(SIGPIPE, SIG_IGN); + signal(SIGHUP, SIG_IGN); + + imsg_init(&iev.ibuf, GOTD_SOCK_FILENO); + iev.handler = repo_write_dispatch; + iev.events = EV_READ; + iev.handler_arg = NULL; + event_set(&iev.ev, iev.ibuf.fd, EV_READ, repo_write_dispatch, &iev); + if (event_add(&iev.ev, NULL) == -1) { + err = got_error_from_errno("event_add"); + goto done; + } + + event_dispatch(); +done: + if (err) + log_warnx("%s: %s", title, err->msg); + repo_write_shutdown(); +} + +void +repo_write_shutdown(void) +{ + log_debug("%s: shutting down", repo_write.title); + if (repo_write.repo) + got_repo_close(repo_write.repo); + got_repo_pack_fds_close(repo_write.pack_fds); + got_repo_temp_fds_close(repo_write.pack_fds); + exit(0); +} blob - /dev/null blob + 7e5a7a9839979cbe97d2571a61acbc00cc378898 (mode 644) --- /dev/null +++ gotd/repo_write.h @@ -0,0 +1,18 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +void repo_write_main(const char *, int *, int *); +void repo_write_shutdown(void); blob - 8f99e7a2489fce25802d4360c54fd0110b3b6827 blob + 8744ffb84ade1da058f9e9837fdbbd91cd5130fc --- gotwebd/Makefile +++ gotwebd/Makefile @@ -19,7 +19,7 @@ SRCS += blame.c commit_graph.c delta.c diff.c \ diff_output_edscript.c diff_patience.c bloom.c murmurhash2.c \ worktree_open.c patch.c sigs.c date.c sockaddr.c \ object_open_privsep.c read_gitconfig_privsep.c \ - read_gotconfig_privsep.c + read_gotconfig_privsep.c pollfd.c reference_parse.c MAN = ${PROG}.conf.5 ${PROG}.8 blob - 8a3f38ce4c45ce1386bdc180e342b043f8abe809 blob + 40a8a89af35bb2ad537815f6c18fd5fd38f1d013 --- gotwebd/libexec/got-read-blob/Makefile +++ gotwebd/libexec/got-read-blob/Makefile @@ -4,7 +4,7 @@ PROG= got-read-blob SRCS= got-read-blob.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c + path.c privsep.c sha1.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../../include -I${.CURDIR}/../../../lib LDADD = -lutil -lz blob - 0996068f0f3d0bb71779d03c1c1a7ec355644166 blob + cf8660bd4f6b01319d7acb1039c734dfa629da91 --- gotwebd/libexec/got-read-commit/Makefile +++ gotwebd/libexec/got-read-commit/Makefile @@ -4,7 +4,7 @@ PROG= got-read-commit SRCS= got-read-commit.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c + path.c privsep.c sha1.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../../include -I${.CURDIR}/../../../lib LDADD = -lutil -lz blob - 77cc7852cae9199c991f2908f2d9e8a27da650ee blob + 72c1dd13680cea2b0ea7f5c2e72f7eaa2496f77c --- gotwebd/libexec/got-read-gitconfig/Makefile +++ gotwebd/libexec/got-read-gitconfig/Makefile @@ -4,7 +4,7 @@ PROG= got-read-gitconfig SRCS= got-read-gitconfig.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c gitconfig.c + path.c privsep.c sha1.c gitconfig.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../../include -I${.CURDIR}/../../../lib LDADD = -lutil -lz blob - 29605918a07b9e1969e3a2722605c3424e77d13f blob + 4b42a2accc0b3f70acfa45cbe4996b1eb9298e2f --- gotwebd/libexec/got-read-gotconfig/Makefile +++ gotwebd/libexec/got-read-gotconfig/Makefile @@ -4,7 +4,7 @@ PROG= got-read-gotconfig SRCS= got-read-gotconfig.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c parse.y + path.c privsep.c sha1.c parse.y pollfd.c CPPFLAGS = -I${.CURDIR}/../../../include -I${.CURDIR}/../../../lib \ -I${.CURDIR}/../../../libexec/got-read-gotconfig blob - 4889fe0bab46bbcdb20610ea9255916cd11d0e0d blob + 5de9e23dff74ce4b6002d8f839c2f1daaf5db9bf --- gotwebd/libexec/got-read-object/Makefile +++ gotwebd/libexec/got-read-object/Makefile @@ -4,7 +4,7 @@ PROG= got-read-object SRCS= got-read-object.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c + path.c privsep.c sha1.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../../include -I${.CURDIR}/../../../lib LDADD = -lutil -lz blob - a28b9cfd2d1c5d17ffcbe771790c2183e406f924 blob + f2b80f8b19809b9120fa5cd3263b48fac0ba01b1 --- gotwebd/libexec/got-read-pack/Makefile +++ gotwebd/libexec/got-read-pack/Makefile @@ -5,7 +5,7 @@ PROG= got-read-pack SRCS= got-read-pack.c delta.c error.c inflate.c object_cache.c \ object_idset.c object_parse.c opentemp.c pack.c path.c \ - privsep.c sha1.c delta_cache.c + privsep.c sha1.c delta_cache.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../../include -I${.CURDIR}/../../../lib LDADD = -lutil -lz blob - 3a0b798c57ea849bde67efe66e3b721f7486e287 blob + 203f0dabf48adc72a071b923ee0ca33f8cb7cc14 --- gotwebd/libexec/got-read-tag/Makefile +++ gotwebd/libexec/got-read-tag/Makefile @@ -4,7 +4,7 @@ PROG= got-read-tag SRCS= got-read-tag.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c + path.c privsep.c sha1.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../../include -I${.CURDIR}/../../../lib LDADD = -lutil -lz blob - 19a4c9cfa3379ca5fec4fafdc691d38c80c996e8 blob + 5e2f4bda5f9910c7dd4473e99d088fe961a387ec --- gotwebd/libexec/got-read-tree/Makefile +++ gotwebd/libexec/got-read-tree/Makefile @@ -4,7 +4,7 @@ PROG= got-read-tree SRCS= got-read-tree.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c + path.c privsep.c sha1.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../../include -I${.CURDIR}/../../../lib LDADD = -lutil -lz blob - /dev/null blob + c1a000c41f1170b02f9c21852f7ae112290e359a (mode 644) --- /dev/null +++ gotsh/Makefile @@ -0,0 +1,31 @@ +.PATH:${.CURDIR}/../lib ${.CURDIR}/../gotd + +.include "../got-version.mk" + +PREFIX ?= /usr/local +BINDIR ?= ${PREFIX}/bin + +PROG= gotsh +SRCS= gotsh.c error.c pkt.c sha1.c serve.c path.c gitproto.c \ + imsg.c pollfd.c reference_parse.c + +MAN = ${PROG}.1 + +CPPFLAGS = -I${.CURDIR}/../include -I${.CURDIR}/../lib -I${.CURDIR}/../gotd + +.if defined(PROFILE) +LDADD = -lutil_p -lz_p -lm_p -lc_p -levent_p +.else +LDADD = -lutil -lz -lm -levent +.endif +DPADD = ${LIBZ} ${LIBUTIL} + +.if ${GOT_RELEASE} != "Yes" +NOMAN = Yes +.endif + +realinstall: + ${INSTALL} ${INSTALL_COPY} -o ${BINOWN} -g ${BINGRP} \ + -m ${BINMODE} ${PROG} ${BINDIR}/${PROG} + +.include blob - /dev/null blob + b12b0ba51bbe016f648cf3e84bd1c9bd1923fa51 (mode 644) --- /dev/null +++ gotsh/gotsh.1 @@ -0,0 +1,113 @@ +.\" +.\" Copyright (c) 2022 Stefan Sperling +.\" +.\" Permission to use, copy, modify, and distribute this software for any +.\" purpose with or without fee is hereby granted, provided that the above +.\" copyright notice and this permission notice appear in all copies. +.\" +.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +.\" +.Dd $Mdocdate$ +.Dt GOTSH 1 +.Os +.Sh NAME +.Nm gotsh +.Nd Game of Trees Shell +.Sh SYNOPSIS +.Nm Fl c Sq Cm git-receive-pack Ar repository-path +.Nm Fl c Sq Cm git-upload-pack Ar repository-path +.Sh DESCRIPTION +.Nm +is the network-facing interface to +.Xr gotd 8 . +It implements the server-side part of the Git network protocol used by +.Xr git 1 +and +.Xr got 1 . +.Pp +.Nm +is not an interactive shell. +.Nm +is intended to be configured as the login shell of Git repository +user accounts on servers running +.Xr gotd 8 . +The users can then interact with +.Xr gotd 8 +over the network. +When users invoke commands such as +.Cm got send +and +.Cm got fetch +on client machines, +.Xr got 1 +will connect to the server with +.Xr ssh 1 . +.Nm +will facilitate communication between +.Xr gotd 8 +running on the server machine and the +.Xr got 1 +or +.Xr git 1 +program running on the client machine. +.Pp +Users running +.Nm +must be members of the group which has read/write permission to the +.Xr gotd 8 +unix socket. +The group used for this purpose can be configured in +.Xr gotd.conf 5 . +Users running +.Nm +should not have access to Git repositories by means other than +accessing the unix socket of +.Xr gotd 8 +via +.Nm . +.Pp +It is recommended to restrict +.Xr ssh 1 +features available to users of +.Nm . +See the EXAMPLES section for details. +.Sh ENVIRONMENT +.Bl -tag -width GOTD_UNIX_SOCKET +.It Ev GOTD_UNIX_SOCKET +Set the path to the unix socket which +.Xr gotd 8 +is listening on. +If not specified, the default path +.Pa /var/run/gotd.sock +will be used. +.El +.Sh EXAMPLES +The following +.Xr sshd_config 5 +directives are recommended to protect the server machine and any systems +reachable from it via +.Xr ssh 1 +forwarding features. +This example assumes the group called +.Dq _gotsh +has read/write access to the +.Xr gotd 8 +unix socket. +.Bd -literal -offset indent +Match Group _gotsh + DisableForwarding + PermitTTY no +.Sh SEE ALSO +.Xr got 1 , +.Xr ssh 1 , +.Xr gotd.conf 5 , +.Xr sshd_config 5 , +.Xr gotd 8 +.Sh AUTHORS +.An Stefan Sperling Aq Mt stsp@openbsd.org blob - /dev/null blob + bcad03d7f0b1bac247ec4be71f63f13760cefef8 (mode 644) --- /dev/null +++ gotsh/gotsh.c @@ -0,0 +1,126 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "got_error.h" +#include "got_serve.h" + +#include "gotd.h" + +static int chattygot; + +__dead static void +usage() +{ + fprintf(stderr, "usage: %s -c '%s|%s repository-path'\n", + getprogname(), GOT_SERVE_CMD_SEND, GOT_SERVE_CMD_FETCH); + exit(1); +} + +static const struct got_error * +apply_unveil(const char *unix_socket_path) +{ +#ifdef PROFILE + if (unveil("gmon.out", "rwc") != 0) + return got_error_from_errno2("unveil", "gmon.out"); +#endif + if (unveil(unix_socket_path, "w") != 0) + return got_error_from_errno2("unveil", unix_socket_path); + + if (unveil(NULL, NULL) != 0) + return got_error_from_errno("unveil"); + + return NULL; +} + +int +main(int argc, char *argv[]) +{ + const struct got_error *error; + char unix_socket_path[PATH_MAX]; + char *unix_socket_path_env = getenv("GOTD_UNIX_SOCKET"); + int gotd_sock = -1; + struct sockaddr_un sun; + +#ifndef PROFILE + if (pledge("stdio recvfd unix unveil", NULL) == -1) + err(1, "pledge"); +#endif + if (argc != 3 || + strcmp(argv[1], "-c") != 0 || + (strncmp(argv[2], GOT_SERVE_CMD_SEND, + strlen(GOT_SERVE_CMD_SEND)) != 0 && + (strncmp(argv[2], GOT_SERVE_CMD_FETCH, + strlen(GOT_SERVE_CMD_FETCH)) != 0))) + usage(); + + if (unix_socket_path_env) { + if (strlcpy(unix_socket_path, unix_socket_path_env, + sizeof(unix_socket_path)) >= sizeof(unix_socket_path)) + errx(1, "gotd socket path too long"); + } else { + strlcpy(unix_socket_path, GOTD_UNIX_SOCKET, + sizeof(unix_socket_path)); + } + + error = apply_unveil(unix_socket_path); + if (error) + goto done; + +#ifndef PROFILE + if (pledge("stdio recvfd unix", NULL) == -1) + err(1, "pledge"); +#endif + if ((gotd_sock = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) + err(1, "socket"); + + memset(&sun, 0, sizeof(sun)); + sun.sun_family = AF_UNIX; + if (strlcpy(sun.sun_path, unix_socket_path, + sizeof(sun.sun_path)) >= sizeof(sun.sun_path)) + errx(1, "gotd socket path too long"); + if (connect(gotd_sock, (struct sockaddr *)&sun, sizeof(sun)) == -1) + err(1, "connect: %s", unix_socket_path); + +#ifndef PROFILE + if (pledge("stdio recvfd", NULL) == -1) + err(1, "pledge"); +#endif + error = got_serve(STDIN_FILENO, STDOUT_FILENO, argv[2], gotd_sock, + chattygot); +done: + if (gotd_sock != -1) + close(gotd_sock); + if (error) { + fprintf(stderr, "%s: %s\n", getprogname(), error->msg); + return 1; + } + + return 0; +} blob - 3ef43e397f0c76093e7db235561aa095240895e8 blob + f0c22e965e0b95671ae7f6667830714b714ecbe9 --- include/got_error.h +++ include/got_error.h @@ -113,7 +113,7 @@ #define GOT_ERR_HISTEDIT_BUSY 96 #define GOT_ERR_HISTEDIT_CMD 97 #define GOT_ERR_HISTEDIT_PATH 98 -/* 99 is currently unused */ +#define GOT_ERR_PACKFILE_CSUM 99 #define GOT_ERR_COMMIT_BRANCH 100 #define GOT_ERR_FILE_STAGED 101 #define GOT_ERR_STAGE_NO_CHANGE 102 @@ -175,12 +175,20 @@ #define GOT_ERR_COMMIT_REDUNDANT_AUTHOR 157 #define GOT_ERR_BAD_QUERYSTRING 158 #define GOT_ERR_INTEGRATE_BRANCH 159 +#define GOT_ERR_BAD_REQUEST 160 +#define GOT_ERR_CLIENT_ID 161 +#define GOT_ERR_REPO_TEMPFILE 162 +#define GOT_ERR_REFS_PROTECTED 163 +#define GOT_ERR_REF_PROTECTED 164 +#define GOT_ERR_REF_BUSY 165 struct got_error { int code; const char *msg; }; +#define GOT_ERR_MAX_MSG_SIZE 4080 /* includes '\0' */ + /* * Get an error object from the above list, for a given error code. * The error message is fixed. blob - 4f8b24e0e19f1af4e21707388cf9bbf9bb54cd18 blob + b507b207a3c163a0492722f3e42ea689dacbf7e6 --- include/got_repository.h +++ include/got_repository.h @@ -186,3 +186,8 @@ const struct got_error *got_repo_pack_fds_open(int **) /* Close the array of file descriptors handed over to got_repo_open for pack */ const struct got_error *got_repo_pack_fds_close(int *); + +/* Open/set/close temporary files for internal use. Needed by gotd(8). */ +const struct got_error *got_repo_temp_fds_open(int **); +void got_repo_temp_fds_set(struct got_repository *, int *); +const struct got_error *got_repo_temp_fds_close(int *); blob - /dev/null blob + 180c362c726680bac4fbe3b5fd47dca9a5e166ae (mode 644) --- /dev/null +++ include/got_serve.h @@ -0,0 +1,21 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#define GOT_SERVE_CMD_SEND "git-receive-pack" +#define GOT_SERVE_CMD_FETCH "git-upload-pack" + +const struct got_error *got_serve(int infd, int outfd, const char *gitcmd, + int gotd_sock, int chattygot); blob - 1c914ee420f9d1fd24cc811a67c7c81a690b9d2a blob + 050225122b5b7622f11da5c0a7196021b719fe68 --- lib/buf.c +++ lib/buf.c @@ -192,6 +192,23 @@ buf_empty(BUF *b) b->cb_len = 0; } +/* Discard the leading bytes from the buffer. */ +const struct got_error * +buf_discard(BUF *b, size_t n) +{ + if (n > b->cb_len) + return got_error(GOT_ERR_RANGE); + + if (n == b->cb_len) + buf_empty(b); + else { + memmove(b->cb_buf, b->cb_buf + n, b->cb_len - n); + b->cb_len -= n; + } + + return NULL; +} + /* * Append a single character to the end of the buffer . */ blob - aff1492306e97d3fdd255c836d083eea96b30e19 blob + 6dba5533403059ec66da01b5d67892dfd4002f88 --- lib/buf.h +++ lib/buf.h @@ -56,6 +56,7 @@ void buf_free(BUF *); void *buf_release(BUF *); u_char buf_getc(BUF *, size_t); void buf_empty(BUF *); +const struct got_error *buf_discard(BUF *, size_t); const struct got_error *buf_append(size_t *, BUF *, const void *, size_t); const struct got_error *buf_putc(BUF *, int); const struct got_error *buf_puts(size_t *, BUF *b, const char *str); blob - bed54dcd68f16ffd5582f7f477c963c86ca93fc6 blob + 3ce3c75e0f8d23779fd0bd90f5d81d7953cf433e --- lib/deflate.c +++ lib/deflate.c @@ -29,6 +29,7 @@ #include "got_path.h" #include "got_lib_deflate.h" +#include "got_lib_poll.h" #ifndef MIN #define MIN(_a,_b) ((_a) < (_b) ? (_a) : (_b)) @@ -255,15 +256,9 @@ got_deflate_to_fd(off_t *outlen, FILE *infile, off_t l goto done; len -= consumed; if (avail > 0) { - ssize_t w; - w = write(outfd, zb.outbuf, avail); - if (w == -1) { - err = got_error_from_errno("write"); - goto done; - } else if (w != avail) { - err = got_error(GOT_ERR_IO); + err = got_poll_write_full(outfd, zb.outbuf, avail); + if (err) goto done; - } if (csum) csum_output(csum, zb.outbuf, avail); *outlen += avail; @@ -296,15 +291,9 @@ got_deflate_to_fd_mmap(off_t *outlen, uint8_t *map, si offset += consumed; len -= consumed; if (avail > 0) { - ssize_t w; - w = write(outfd, zb.outbuf, avail); - if (w == -1) { - err = got_error_from_errno("write"); + err = got_poll_write_full(outfd, zb.outbuf, avail); + if (err) goto done; - } else if (w != avail) { - err = got_error(GOT_ERR_IO); - goto done; - } if (csum) csum_output(csum, zb.outbuf, avail); *outlen += avail; blob - bdf99a0a32ad209551673841dbb8c303028eb9e1 blob + d12ab759e6d259c67d2d70004b3aab5e63c0210c --- lib/error.c +++ lib/error.c @@ -147,7 +147,7 @@ static const struct got_error got_errors[] = { { GOT_ERR_HISTEDIT_CMD, "bad histedit command" }, { GOT_ERR_HISTEDIT_PATH, "cannot edit branch history which contains " "changes outside of this work tree's path prefix" }, - { 99, "unused error code" }, + { GOT_ERR_PACKFILE_CSUM, "pack file checksum error" }, { GOT_ERR_COMMIT_BRANCH, "will not commit to a branch outside the " "\"refs/heads/\" reference namespace" }, { GOT_ERR_FILE_STAGED, "file is staged" }, @@ -225,11 +225,17 @@ static const struct got_error got_errors[] = { { GOT_ERR_BAD_QUERYSTRING, "invalid query string" }, { GOT_ERR_INTEGRATE_BRANCH, "will not integrate into a reference " "outside the \"refs/heads/\" reference namespace" }, + { GOT_ERR_BAD_REQUEST, "unexpected request received" }, + { GOT_ERR_CLIENT_ID, "unknown client identifier" }, + { GOT_ERR_REPO_TEMPFILE, "no repository tempfile available" }, + { GOT_ERR_REFS_PROTECTED, "reference namespace may not be modified" }, + { GOT_ERR_REF_PROTECTED," reference may not be modified" }, + { GOT_ERR_REF_BUSY, "reference cannot be updated; please try again" }, }; static struct got_custom_error { struct got_error err; - char msg[4080]; + char msg[GOT_ERR_MAX_MSG_SIZE]; } custom_errors[16]; static struct got_custom_error * blob - dec248671df1f10f622bd433e33a46fc2d10f627 blob + a487dffb531fb051aa859f3c1ac13816fdfcd926 --- lib/gitproto.c +++ lib/gitproto.c @@ -32,8 +32,19 @@ #define nitems(_a) (sizeof((_a)) / sizeof((_a)[0])) #endif +static void +free_tokens(char **tokens, size_t ntokens) +{ + int i; + + for (i = 0; i < ntokens; i++) { + free(tokens[i]); + tokens[i] = NULL; + } +} + static const struct got_error * -tokenize_refline(char **tokens, char *line, int len, int maxtokens) +tokenize_line(char **tokens, char *line, int len, int mintokens, int maxtokens) { const struct got_error *err = NULL; char *p; @@ -64,16 +75,12 @@ tokenize_refline(char **tokens, char *line, int len, i n++; } } - if (i <= 2) - err = got_error(GOT_ERR_BAD_PACKET); + if (i < mintokens) + err = got_error_msg(GOT_ERR_BAD_PACKET, + "pkt-line contains too few tokens"); done: - if (err) { - int j; - for (j = 0; j < i; j++) { - free(tokens[j]); - tokens[j] = NULL; - } - } + if (err) + free_tokens(tokens, i); return err; } @@ -88,7 +95,7 @@ got_gitproto_parse_refline(char **id_str, char **refna *refname = NULL; /* don't reset *server_capabilities */ - err = tokenize_refline(tokens, line, len, nitems(tokens)); + err = tokenize_line(tokens, line, len, 2, nitems(tokens)); if (err) return err; @@ -110,6 +117,115 @@ got_gitproto_parse_refline(char **id_str, char **refna return NULL; } +const struct got_error * +got_gitproto_parse_want_line(char **id_str, + char **capabilities, char *line, int len) +{ + const struct got_error *err = NULL; + char *tokens[3]; + + *id_str = NULL; + /* don't reset *capabilities */ + + err = tokenize_line(tokens, line, len, 2, nitems(tokens)); + if (err) + return err; + + if (tokens[0] == NULL) { + free_tokens(tokens, nitems(tokens)); + return got_error_msg(GOT_ERR_BAD_PACKET, "empty want-line"); + } + + if (strcmp(tokens[0], "want") != 0) { + free_tokens(tokens, nitems(tokens)); + return got_error_msg(GOT_ERR_BAD_PACKET, "bad want-line"); + } + + free(tokens[0]); + if (tokens[1]) + *id_str = tokens[1]; + if (tokens[2]) { + if (*capabilities == NULL) { + char *p; + *capabilities = tokens[2]; + p = strrchr(*capabilities, '\n'); + if (p) + *p = '\0'; + } else + free(tokens[2]); + } + + return NULL; +} + +const struct got_error * +got_gitproto_parse_have_line(char **id_str, char *line, int len) +{ + const struct got_error *err = NULL; + char *tokens[2]; + + *id_str = NULL; + + err = tokenize_line(tokens, line, len, 2, nitems(tokens)); + if (err) + return err; + + if (tokens[0] == NULL) { + free_tokens(tokens, nitems(tokens)); + return got_error_msg(GOT_ERR_BAD_PACKET, "empty have-line"); + } + + if (strcmp(tokens[0], "have") != 0) { + free_tokens(tokens, nitems(tokens)); + return got_error_msg(GOT_ERR_BAD_PACKET, "bad have-line"); + } + + free(tokens[0]); + if (tokens[1]) + *id_str = tokens[1]; + + return NULL; +} + +const struct got_error * +got_gitproto_parse_ref_update_line(char **old_id_str, char **new_id_str, + char **refname, char **capabilities, char *line, size_t len) +{ + const struct got_error *err = NULL; + char *tokens[4]; + + *old_id_str = NULL; + *new_id_str = NULL; + *refname = NULL; + + /* don't reset *capabilities */ + + err = tokenize_line(tokens, line, len, 3, nitems(tokens)); + if (err) + return err; + + if (tokens[0] == NULL || tokens[1] == NULL || tokens[2] == NULL) { + free_tokens(tokens, nitems(tokens)); + return got_error_msg(GOT_ERR_BAD_PACKET, "empty ref-update"); + } + + *old_id_str = tokens[0]; + *new_id_str = tokens[1]; + *refname = tokens[2]; + if (tokens[3]) { + if (*capabilities == NULL) { + char *p; + *capabilities = tokens[3]; + p = strrchr(*capabilities, '\n'); + if (p) + *p = '\0'; + } else + free(tokens[3]); + } + + return NULL; +} + static const struct got_error * match_capability(char **my_capabilities, const char *capa, const struct got_capability *mycapa) @@ -175,7 +291,7 @@ done: const struct got_error * got_gitproto_match_capabilities(char **common_capabilities, - struct got_pathlist_head *symrefs, char *server_capabilities, + struct got_pathlist_head *symrefs, char *capabilities, const struct got_capability my_capabilities[], size_t ncapa) { const struct got_error *err = NULL; @@ -184,7 +300,7 @@ got_gitproto_match_capabilities(char **common_capabili *common_capabilities = NULL; do { - capa = strsep(&server_capabilities, " "); + capa = strsep(&capabilities, " "); if (capa == NULL) return NULL; @@ -212,3 +328,112 @@ got_gitproto_match_capabilities(char **common_capabili } return err; } + +const struct got_error * +got_gitproto_append_capabilities(size_t *capalen, char *buf, size_t offset, + size_t bufsize, const struct got_capability my_capabilities[], size_t ncapa) +{ + char *p = buf + offset; + size_t i, len, remain = bufsize - offset; + + *capalen = 0; + + if (offset >= bufsize || remain < 1) + return got_error(GOT_ERR_NO_SPACE); + + /* Capabilities are hidden behind a NUL byte. */ + *p = '\0'; + p++; + remain--; + *capalen += 1; + + for (i = 0; i < ncapa; i++) { + len = strlcat(p, " ", remain); + if (len >= remain) + return got_error(GOT_ERR_NO_SPACE); + remain -= len; + *capalen += 1; + + len = strlcat(p, my_capabilities[i].key, remain); + if (len >= remain) + return got_error(GOT_ERR_NO_SPACE); + remain -= len; + *capalen += strlen(my_capabilities[i].key); + + if (my_capabilities[i].value == NULL) + continue; + + len = strlcat(p, "=", remain); + if (len >= remain) + return got_error(GOT_ERR_NO_SPACE); + remain -= len; + *capalen += 1; + + len = strlcat(p, my_capabilities[i].value, remain); + if (len >= remain) + return got_error(GOT_ERR_NO_SPACE); + remain -= len; + *capalen += strlen(my_capabilities[i].value); + } + + return NULL; +} + +const struct got_error * +got_gitproto_split_capabilities_str(struct got_capability **capabilities, + size_t *ncapabilities, char *capabilities_str) +{ + char *capastr, *capa; + size_t i; + + *capabilities = NULL; + *ncapabilities = 0; + + /* Compute number of capabilities on a copy of the input string. */ + capastr = strdup(capabilities_str); + if (capastr == NULL) + return got_error_from_errno("strdup"); + do { + capa = strsep(&capastr, " "); + if (capa && *capa != '\0') + (*ncapabilities)++; + } while (capa); + free(capastr); + + *capabilities = calloc(*ncapabilities, sizeof(**capabilities)); + if (*capabilities == NULL) + return got_error_from_errno("calloc"); + + /* Modify input string in place, splitting it into key/value tuples. */ + i = 0; + for (;;) { + char *key = NULL, *value = NULL, *equalsign; + + capa = strsep(&capabilities_str, " "); + if (capa == NULL) + break; + if (*capa == '\0') + continue; + + if (i >= *ncapabilities) { /* should not happen */ + free(*capabilities); + *capabilities = NULL; + *ncapabilities = 0; + return got_error(GOT_ERR_NO_SPACE); + } + + key = capa; + + equalsign = strchr(capa, '='); + if (equalsign != NULL) { + *equalsign = '\0'; + value = equalsign + 1; + } + + (*capabilities)[i].key = key; + (*capabilities)[i].value = value; + i++; + } + + return NULL; +} blob - cffe349e03b4a8a0cd9126b511ed9198af1c3c57 blob + 09426d878e3a3c32836fe6d798cf82b8f46702e5 --- lib/got_lib_gitproto.h +++ lib/got_lib_gitproto.h @@ -20,11 +20,14 @@ #define GOT_CAPA_SIDE_BAND_64K "side-band-64k" #define GOT_CAPA_REPORT_STATUS "report-status" #define GOT_CAPA_DELETE_REFS "delete-refs" +#define GOT_CAPA_NO_THIN "no-thin" #define GOT_SIDEBAND_PACKFILE_DATA 1 #define GOT_SIDEBAND_PROGRESS_INFO 2 #define GOT_SIDEBAND_ERROR_INFO 3 +#define GOT_SIDEBAND_64K_PACKFILE_DATA_MAX (GOT_PKT_MAX - 17) + struct got_capability { const char *key; const char *value; @@ -34,7 +37,20 @@ struct got_pathlist_head; const struct got_error *got_gitproto_parse_refline(char **id_str, char **refname, char **server_capabilities, char *line, int len); +const struct got_error *got_gitproto_parse_want_line(char **id_str, + char **capabilities, char *line, int len); +const struct got_error *got_gitproto_parse_have_line(char **id_str, + char *line, int len); +const struct got_error *got_gitproto_parse_ref_update_line(char **old_id_str, + char **new_id_str, char **refname, char **client_capabilities, + char *line, size_t len); const struct got_error *got_gitproto_match_capabilities( char **common_capabilities, - struct got_pathlist_head *symrefs, char *server_capabilities, + struct got_pathlist_head *symrefs, char *capabilities, const struct got_capability my_capabilities[], size_t ncapa); +const struct got_error *got_gitproto_append_capabilities(size_t *capalen, + char *buf, size_t offset, size_t bufsize, + const struct got_capability my_capabilities[], size_t ncapa); +const struct got_error *got_gitproto_split_capabilities_str( + struct got_capability **capabilities, size_t *ncapabilities, + char *capabilities_str); blob - 2ed7fd6f24b77b5c61137092e927591fc46d903d blob + f0d614fbd97c672d8ebb6442ea09ea11e0d3e929 --- lib/got_lib_object.h +++ lib/got_lib_object.h @@ -31,13 +31,22 @@ struct got_object { int refcnt; /* > 0 if open and/or cached */ }; + +/* A callback function which is invoked when a raw object is closed. */ +struct got_raw_object; +typedef void (got_object_raw_close_cb)(struct got_raw_object *); + struct got_raw_object { FILE *f; /* NULL if data buffer is being used */ int fd; /* -1 unless data buffer is memory-mapped */ + int tempfile_idx; /* -1 unless using a repository-tempfile */ uint8_t *data; off_t size; size_t hdrlen; int refcnt; /* > 0 if open and/or cached */ + + got_object_raw_close_cb *close_cb; + void *close_arg; }; struct got_commit_object { @@ -144,3 +153,6 @@ const struct got_error *got_object_enumerate(int *, got_object_enumerate_commit_cb, got_object_enumerate_tree_cb, void *, struct got_object_id **, int, struct got_object_id **, int, struct got_packidx *, struct got_repository *); + +const struct got_error *got_object_raw_alloc(struct got_raw_object **, + uint8_t *, int *, size_t, off_t); blob - 723508cf635f39153b1b47082dca8bf6b3bb09ca blob + 8498bedda0021930f8e588d1ef4adb98a845c1f0 --- lib/got_lib_object_parse.h +++ lib/got_lib_object_parse.h @@ -22,6 +22,8 @@ struct got_tree_entry *got_alloc_tree_entry_partial(vo const struct got_error *got_object_parse_commit(struct got_commit_object **, char *, size_t); +const struct got_error *got_object_read_commit(struct got_commit_object **, int, + struct got_object_id *, size_t); struct got_parsed_tree_entry { const char *name; /* Points to name in parsed buffer */ @@ -31,9 +33,13 @@ struct got_parsed_tree_entry { }; const struct got_error *got_object_parse_tree(struct got_parsed_tree_entry **, size_t *, size_t *, uint8_t *, size_t); +const struct got_error *got_object_read_tree(struct got_parsed_tree_entry **, + size_t *, size_t *, uint8_t **, int, struct got_object_id *); const struct got_error *got_object_parse_tag(struct got_tag_object **, uint8_t *, size_t); +const struct got_error *got_object_read_tag(struct got_tag_object **, int, + struct got_object_id *, size_t); const struct got_error *got_read_file_to_mem(uint8_t **, size_t *, FILE *); struct got_pack; @@ -43,3 +49,5 @@ struct got_inflate_checksum; const struct got_error *got_object_parse_header(struct got_object **, char *, size_t); const struct got_error *got_object_read_header(struct got_object **, int); +const struct got_error *got_object_read_raw(uint8_t **, off_t *, + size_t *, size_t, int, struct got_object_id *, int); blob - 0aaa7ce0d80e279d0cc620502c79b4671a3ae724 blob + bec94759fda7eb63db8c6a38520291cd5c1a8ad1 --- lib/got_lib_pack_index.h +++ lib/got_lib_pack_index.h @@ -18,6 +18,8 @@ typedef const struct got_error *(got_pack_index_progre uint32_t nobj_total, uint32_t nobj_indexed, uint32_t nobj_loose, uint32_t nobj_resolved); +const struct got_error *got_pack_hwrite(int, void *, int, SHA1_CTX *); + const struct got_error * got_pack_index(struct got_pack *pack, int idxfd, FILE *tmpfile, FILE *delta_base_file, FILE *delta_accum_file, blob - /dev/null blob + 4f41b7577755f2ed3b7dcbb4e217ed32d08d0bde (mode 644) --- /dev/null +++ lib/got_lib_poll.h @@ -0,0 +1,20 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +const struct got_error *got_poll_fd(int fd, int events, int timeout); +const struct got_error *got_poll_read_full(int, size_t *, void *, size_t, + size_t); +const struct got_error *got_poll_write_full(int, const void *, off_t); blob - 3e9ad2f8e37f7198f75a99a817a5549bc8daf4f1 blob + ced490e96ad476ac78d59ed49d235863982116c3 --- lib/got_lib_repository.h +++ lib/got_lib_repository.h @@ -30,6 +30,13 @@ #define GOT_PACK_CACHE_SIZE 32 +/* + * While in gotd(8) chroot, a repository needs this many temporary files. + * This limit sets an upper bound on how many raw objects or blobs can + * be kept open in parallel. + */ +#define GOT_REPO_NUM_TEMPFILES 32 + struct got_packidx_bloom_filter { RB_ENTRY(got_packidx_bloom_filter) entry; char path[PATH_MAX]; /* on-disk path */ @@ -73,6 +80,10 @@ struct got_repository { /* Open file handles for pack files. */ struct got_pack packs[GOT_PACK_CACHE_SIZE]; + /* Open file handles for storing temporary data in gotd(8) chroot. */ + int tempfiles[GOT_REPO_NUM_TEMPFILES]; + uint32_t tempfile_use_mask; + /* * The cache size limit may be lower than GOT_PACK_CACHE_SIZE, * depending on resource limits. @@ -161,3 +172,7 @@ void got_repo_unpin_pack(struct got_repository *); const struct got_error *got_repo_read_gitconfig(int *, char **, char **, struct got_remote_repo **, int *, char **, char ***, int *, const char *); + +const struct got_error *got_repo_temp_fds_get(int *, int *, + struct got_repository *); +void got_repo_temp_fds_put(int, struct got_repository *); blob - 7fb3bb5f6697a122e8f1b2698501306782d33bdb blob + 806b5542c5fcee17a27336702f83c72c94a50070 --- lib/inflate.c +++ lib/inflate.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -30,6 +31,7 @@ #include "got_path.h" #include "got_lib_inflate.h" +#include "got_lib_poll.h" #ifndef MIN #define MIN(_a,_b) ((_a) < (_b) ? (_a) : (_b)) @@ -167,6 +169,7 @@ const struct got_error * got_inflate_read_fd(struct got_inflate_buf *zb, int fd, size_t *outlenp, size_t *consumed) { + const struct got_error *err = NULL; size_t last_total_out = zb->z.total_out; size_t last_total_in = zb->z.total_in; z_stream *z = &zb->z; @@ -183,7 +186,16 @@ got_inflate_read_fd(struct got_inflate_buf *zb, int fd size_t csum_avail_in = 0, csum_avail_out = 0; if (z->avail_in == 0) { - ssize_t n = read(fd, zb->inbuf, zb->inlen); + ssize_t n; + err = got_poll_fd(fd, POLLIN, INFTIM); + if (err) { + if (err->code == GOT_ERR_EOF) { + ret = Z_STREAM_END; + break; + } + return err; + } + n = read(fd, zb->inbuf, zb->inlen); if (n < 0) return got_error_from_errno("read"); else if (n == 0) { blob - a29d9aed8e74dfe0af6e0d402daa550461fe4ca5 blob + f8e685448165384cfc578c0f089b4aa6fb6219c8 --- lib/object.c +++ lib/object.c @@ -920,3 +920,68 @@ got_object_commit_retain(struct got_commit_object *com { commit->refcnt++; } + +const struct got_error * +got_object_raw_alloc(struct got_raw_object **obj, uint8_t *outbuf, int *outfd, + size_t hdrlen, off_t size) +{ + const struct got_error *err = NULL; + + *obj = calloc(1, sizeof(**obj)); + if (*obj == NULL) { + err = got_error_from_errno("calloc"); + goto done; + } + (*obj)->fd = -1; + (*obj)->tempfile_idx = -1; + + if (outbuf) { + (*obj)->data = outbuf; + } else { + struct stat sb; + if (fstat(*outfd, &sb) == -1) { + err = got_error_from_errno("fstat"); + goto done; + } + + if (sb.st_size != hdrlen + size) { + err = got_error(GOT_ERR_PRIVSEP_LEN); + goto done; + } +#ifndef GOT_PACK_NO_MMAP + if (hdrlen + size > 0) { + (*obj)->data = mmap(NULL, hdrlen + size, PROT_READ, + MAP_PRIVATE, *outfd, 0); + if ((*obj)->data == MAP_FAILED) { + if (errno != ENOMEM) { + err = got_error_from_errno("mmap"); + goto done; + } + (*obj)->data = NULL; + } else { + (*obj)->fd = *outfd; + *outfd = -1; + } + } +#endif + if (*outfd != -1) { + (*obj)->f = fdopen(*outfd, "r"); + if ((*obj)->f == NULL) { + err = got_error_from_errno("fdopen"); + goto done; + } + *outfd = -1; + } + } + (*obj)->hdrlen = hdrlen; + (*obj)->size = size; +done: + if (err) { + if (*obj) { + got_object_raw_close(*obj); + *obj = NULL; + } + } else + (*obj)->refcnt++; + return err; +} blob - be7da9be78909deb2cfae4068d9ea7a25d5f2964 blob + 07661321d0cf3745053ed8ace0d4e6bacf83d615 --- lib/object_cache.c +++ lib/object_cache.c @@ -43,7 +43,7 @@ #define GOT_OBJECT_CACHE_SIZE_TREE 256 #define GOT_OBJECT_CACHE_SIZE_COMMIT 64 #define GOT_OBJECT_CACHE_SIZE_TAG 2048 -#define GOT_OBJECT_CACHE_SIZE_RAW 64 +#define GOT_OBJECT_CACHE_SIZE_RAW 16 #define GOT_OBJECT_CACHE_MAX_ELEM_SIZE 1048576 /* 1 MB */ const struct got_error * blob - 135e7ee1cfac145ac32e4f2556498a2a472f3cbd blob + e8c16fd8ba83fc71b04a6214300dcc4261fea75e --- lib/object_open_privsep.c +++ lib/object_open_privsep.c @@ -509,53 +509,10 @@ got_object_raw_open(struct got_raw_object **obj, int * goto done; } - *obj = calloc(1, sizeof(**obj)); - if (*obj == NULL) { - err = got_error_from_errno("calloc"); + err = got_object_raw_alloc(obj, outbuf, outfd, hdrlen, size); + if (err) goto done; - } - (*obj)->fd = -1; - if (outbuf) { - (*obj)->data = outbuf; - } else { - struct stat sb; - if (fstat(*outfd, &sb) == -1) { - err = got_error_from_errno("fstat"); - goto done; - } - - if (sb.st_size != hdrlen + size) { - err = got_error(GOT_ERR_PRIVSEP_LEN); - goto done; - } -#ifndef GOT_PACK_NO_MMAP - if (hdrlen + size > 0) { - (*obj)->data = mmap(NULL, hdrlen + size, PROT_READ, - MAP_PRIVATE, *outfd, 0); - if ((*obj)->data == MAP_FAILED) { - if (errno != ENOMEM) { - err = got_error_from_errno("mmap"); - goto done; - } - (*obj)->data = NULL; - } else { - (*obj)->fd = *outfd; - *outfd = -1; - } - } -#endif - if (*outfd != -1) { - (*obj)->f = fdopen(*outfd, "r"); - if ((*obj)->f == NULL) { - err = got_error_from_errno("fdopen"); - goto done; - } - *outfd = -1; - } - } - (*obj)->hdrlen = hdrlen; - (*obj)->size = size; err = got_repo_cache_raw_object(repo, id, *obj); done: free(path_packfile); blob - 1e7ebaf7fd5125d8622a43d7387611208e41209c blob + b2f06f459a64bbfe86dc489ba4e83470590c3872 --- lib/object_parse.c +++ lib/object_parse.c @@ -134,6 +134,9 @@ got_object_raw_close(struct got_raw_object *obj) return NULL; } + if (obj->close_cb) + obj->close_cb(obj); + if (obj->f == NULL) { if (obj->fd != -1) { if (munmap(obj->data, obj->hdrlen + obj->size) == -1) @@ -271,7 +274,86 @@ done: got_inflate_end(&zb); return err; } + +const struct got_error * +got_object_read_raw(uint8_t **outbuf, off_t *size, size_t *hdrlen, + size_t max_in_mem_size, int outfd, struct got_object_id *expected_id, + int infd) +{ + const struct got_error *err = NULL; + struct got_object *obj; + struct got_inflate_checksum csum; + uint8_t sha1[SHA1_DIGEST_LENGTH]; + SHA1_CTX sha1_ctx; + size_t len, consumed; + FILE *f = NULL; + + *outbuf = NULL; + *size = 0; + *hdrlen = 0; + + SHA1Init(&sha1_ctx); + memset(&csum, 0, sizeof(csum)); + csum.output_sha1 = &sha1_ctx; + + if (lseek(infd, SEEK_SET, 0) == -1) + return got_error_from_errno("lseek"); + + err = got_object_read_header(&obj, infd); + if (err) + return err; + if (lseek(infd, SEEK_SET, 0) == -1) + return got_error_from_errno("lseek"); + + if (obj->size + obj->hdrlen <= max_in_mem_size) { + err = got_inflate_to_mem_fd(outbuf, &len, &consumed, &csum, + obj->size + obj->hdrlen, infd); + } else { + int fd; + /* + * XXX This uses an extra file descriptor for no good reason. + * We should have got_inflate_fd_to_fd(). + */ + fd = dup(infd); + if (fd == -1) + return got_error_from_errno("dup"); + f = fdopen(fd, "r"); + if (f == NULL) { + err = got_error_from_errno("fdopen"); + abort(); + close(fd); + goto done; + } + err = got_inflate_to_fd(&len, f, &csum, outfd); + } + if (err) + goto done; + + if (len < obj->hdrlen || len != obj->hdrlen + obj->size) { + err = got_error(GOT_ERR_BAD_OBJ_HDR); + goto done; + } + + SHA1Final(sha1, &sha1_ctx); + if (memcmp(expected_id->sha1, sha1, SHA1_DIGEST_LENGTH) != 0) { + char buf[SHA1_DIGEST_STRING_LENGTH]; + err = got_error_fmt(GOT_ERR_OBJ_CSUM, + "checksum failure for object %s", + got_sha1_digest_to_str(expected_id->sha1, buf, + sizeof(buf))); + goto done; + } + + *size = obj->size; + *hdrlen = obj->hdrlen; +done: + got_object_close(obj); + if (f && fclose(f) == EOF && err == NULL) + err = got_error_from_errno("fclose"); + return err; +} + struct got_commit_object * got_object_commit_alloc_partial(void) { @@ -653,7 +735,61 @@ done: } return err; } + +const struct got_error * +got_object_read_commit(struct got_commit_object **commit, int fd, + struct got_object_id *expected_id, size_t expected_size) +{ + struct got_object *obj = NULL; + const struct got_error *err = NULL; + size_t len; + uint8_t *p; + struct got_inflate_checksum csum; + SHA1_CTX sha1_ctx; + struct got_object_id id; + SHA1Init(&sha1_ctx); + memset(&csum, 0, sizeof(csum)); + csum.output_sha1 = &sha1_ctx; + + err = got_inflate_to_mem_fd(&p, &len, NULL, &csum, expected_size, fd); + if (err) + return err; + + SHA1Final(id.sha1, &sha1_ctx); + if (memcmp(expected_id->sha1, id.sha1, SHA1_DIGEST_LENGTH) != 0) { + char buf[SHA1_DIGEST_STRING_LENGTH]; + err = got_error_fmt(GOT_ERR_OBJ_CSUM, + "checksum failure for object %s", + got_sha1_digest_to_str(expected_id->sha1, buf, + sizeof(buf))); + goto done; + } + + err = got_object_parse_header(&obj, p, len); + if (err) + goto done; + + if (len < obj->hdrlen + obj->size) { + err = got_error(GOT_ERR_BAD_OBJ_DATA); + goto done; + } + + if (obj->type != GOT_OBJ_TYPE_COMMIT) { + err = got_error(GOT_ERR_OBJ_TYPE); + goto done; + } + + /* Skip object header. */ + len -= obj->hdrlen; + err = got_object_parse_commit(commit, p + obj->hdrlen, len); +done: + free(p); + if (obj) + got_object_close(obj); + return err; +} + void got_object_tree_close(struct got_tree_object *tree) { @@ -773,6 +909,55 @@ done: return err; } +const struct got_error * +got_object_read_tree(struct got_parsed_tree_entry **entries, size_t *nentries, + size_t *nentries_alloc, uint8_t **p, int fd, + struct got_object_id *expected_id) +{ + const struct got_error *err = NULL; + struct got_object *obj = NULL; + size_t len; + struct got_inflate_checksum csum; + SHA1_CTX sha1_ctx; + struct got_object_id id; + + SHA1Init(&sha1_ctx); + memset(&csum, 0, sizeof(csum)); + csum.output_sha1 = &sha1_ctx; + + err = got_inflate_to_mem_fd(p, &len, NULL, &csum, 0, fd); + if (err) + return err; + + SHA1Final(id.sha1, &sha1_ctx); + if (memcmp(expected_id->sha1, id.sha1, SHA1_DIGEST_LENGTH) != 0) { + char buf[SHA1_DIGEST_STRING_LENGTH]; + err = got_error_fmt(GOT_ERR_OBJ_CSUM, + "checksum failure for object %s", + got_sha1_digest_to_str(expected_id->sha1, buf, + sizeof(buf))); + goto done; + } + + err = got_object_parse_header(&obj, *p, len); + if (err) + goto done; + + if (len < obj->hdrlen + obj->size) { + err = got_error(GOT_ERR_BAD_OBJ_DATA); + goto done; + } + + /* Skip object header. */ + len -= obj->hdrlen; + err = got_object_parse_tree(entries, nentries, nentries_alloc, + *p + obj->hdrlen, len); +done: + if (obj) + got_object_close(obj); + return err; +} + void got_object_tag_close(struct got_tag_object *tag) { @@ -963,7 +1148,57 @@ done: if (err) { got_object_tag_close(*tag); *tag = NULL; + } + return err; +} + +const struct got_error * +got_object_read_tag(struct got_tag_object **tag, int fd, + struct got_object_id *expected_id, size_t expected_size) +{ + const struct got_error *err = NULL; + struct got_object *obj = NULL; + size_t len; + uint8_t *p; + struct got_inflate_checksum csum; + SHA1_CTX sha1_ctx; + struct got_object_id id; + + SHA1Init(&sha1_ctx); + memset(&csum, 0, sizeof(csum)); + csum.output_sha1 = &sha1_ctx; + + err = got_inflate_to_mem_fd(&p, &len, NULL, &csum, + expected_size, fd); + if (err) + return err; + + SHA1Final(id.sha1, &sha1_ctx); + if (memcmp(expected_id->sha1, id.sha1, SHA1_DIGEST_LENGTH) != 0) { + char buf[SHA1_DIGEST_STRING_LENGTH]; + err = got_error_fmt(GOT_ERR_OBJ_CSUM, + "checksum failure for object %s", + got_sha1_digest_to_str(expected_id->sha1, buf, + sizeof(buf))); + goto done; } + + err = got_object_parse_header(&obj, p, len); + if (err) + goto done; + + if (len < obj->hdrlen + obj->size) { + err = got_error(GOT_ERR_BAD_OBJ_DATA); + goto done; + } + + /* Skip object header. */ + len -= obj->hdrlen; + err = got_object_parse_tag(tag, p + obj->hdrlen, len); +done: + free(p); + if (obj) + got_object_close(obj); return err; } blob - /dev/null blob + d9f3910f84cec758f9f922c95b27d9d4d35e5151 (mode 644) --- /dev/null +++ lib/object_open_io.c @@ -0,0 +1,699 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "got_error.h" +#include "got_object.h" +#include "got_repository.h" +#include "got_path.h" + +#include "got_lib_delta.h" +#include "got_lib_object.h" +#include "got_lib_object_cache.h" +#include "got_lib_object_parse.h" +#include "got_lib_pack.h" +#include "got_lib_repository.h" + +const struct got_error * +got_object_open_packed(struct got_object **obj, struct got_object_id *id, + struct got_repository *repo) +{ + const struct got_error *err = NULL; + struct got_pack *pack = NULL; + struct got_packidx *packidx = NULL; + int idx; + char *path_packfile; + + err = got_repo_search_packidx(&packidx, &idx, repo, id); + if (err) + return err; + + err = got_packidx_get_packfile_path(&path_packfile, + packidx->path_packidx); + if (err) + return err; + + pack = got_repo_get_cached_pack(repo, path_packfile); + if (pack == NULL) { + err = got_repo_cache_pack(&pack, repo, path_packfile, packidx); + if (err) + goto done; + } + + err = got_packfile_open_object(obj, pack, packidx, idx, id); + if (err) + return err; + (*obj)->refcnt++; + + err = got_repo_cache_object(repo, id, *obj); + if (err) { + if (err->code == GOT_ERR_OBJ_EXISTS || + err->code == GOT_ERR_OBJ_TOO_LARGE) + err = NULL; + } +done: + free(path_packfile); + return err; +} + +const struct got_error * +got_object_open_from_packfile(struct got_object **obj, struct got_object_id *id, + struct got_pack *pack, struct got_packidx *packidx, int obj_idx, + struct got_repository *repo) +{ + return got_error(GOT_ERR_NOT_IMPL); +} + +const struct got_error * +got_object_read_raw_delta(uint64_t *base_size, uint64_t *result_size, + off_t *delta_size, off_t *delta_compressed_size, off_t *delta_offset, + off_t *delta_out_offset, struct got_object_id **base_id, int delta_cache_fd, + struct got_packidx *packidx, int obj_idx, struct got_object_id *id, + struct got_repository *repo) +{ + return got_error(GOT_ERR_NOT_IMPL); +} + +const struct got_error * +got_object_open(struct got_object **obj, struct got_repository *repo, + struct got_object_id *id) +{ + const struct got_error *err = NULL; + int fd; + + *obj = got_repo_get_cached_object(repo, id); + if (*obj != NULL) { + (*obj)->refcnt++; + return NULL; + } + + err = got_object_open_packed(obj, id, repo); + if (err) { + if (err->code != GOT_ERR_NO_OBJ) + return err; + } else + return NULL; + + err = got_object_open_loose_fd(&fd, id, repo); + if (err) { + if (err->code == GOT_ERR_ERRNO && errno == ENOENT) + err = got_error_no_obj(id); + return err; + } + + err = got_object_read_header(obj, fd); + if (err) + goto done; + + memcpy(&(*obj)->id, id, sizeof((*obj)->id)); + (*obj)->refcnt++; + + err = got_repo_cache_object(repo, id, *obj); + if (err) { + if (err->code == GOT_ERR_OBJ_EXISTS || + err->code == GOT_ERR_OBJ_TOO_LARGE) + err = NULL; + } +done: + if (close(fd) == -1 && err == NULL) + err = got_error_from_errno("close"); + return err; +} + +static const struct got_error * +wrap_fd(FILE **f, int wrapped_fd) +{ + const struct got_error *err = NULL; + int fd; + + if (ftruncate(wrapped_fd, 0L) == -1) + return got_error_from_errno("ftruncate"); + + if (lseek(wrapped_fd, 0L, SEEK_SET) == -1) + return got_error_from_errno("lseek"); + + fd = dup(wrapped_fd); + if (fd == -1) + return got_error_from_errno("dup"); + + *f = fdopen(fd, "w+"); + if (*f == NULL) { + err = got_error_from_errno("fdopen"); + close(fd); + } + return err; +} + +static const struct got_error * +read_packed_object_raw(uint8_t **outbuf, off_t *size, size_t *hdrlen, + int outfd, struct got_pack *pack, struct got_packidx *packidx, int idx, + struct got_object_id *id) +{ + const struct got_error *err = NULL; + uint64_t raw_size = 0; + struct got_object *obj; + FILE *outfile = NULL, *basefile = NULL, *accumfile = NULL; + + *outbuf = NULL; + *size = 0; + *hdrlen = 0; + + err = got_packfile_open_object(&obj, pack, packidx, idx, id); + if (err) + return err; + + if (obj->flags & GOT_OBJ_FLAG_DELTIFIED) { + err = got_pack_get_max_delta_object_size(&raw_size, obj, pack); + if (err) + goto done; + } else + raw_size = obj->size; + + if (raw_size <= GOT_DELTA_RESULT_SIZE_CACHED_MAX) { + size_t len; + err = got_packfile_extract_object_to_mem(outbuf, &len, + obj, pack); + if (err) + goto done; + *size = (off_t)len; + } else { + /* + * XXX This uses 3 file extra descriptors for no good reason. + * We should have got_packfile_extract_object_to_fd(). + */ + err = wrap_fd(&outfile, outfd); + if (err) + goto done; + err = wrap_fd(&basefile, pack->basefd); + if (err) + goto done; + err = wrap_fd(&accumfile, pack->accumfd); + if (err) + goto done; + err = got_packfile_extract_object(pack, obj, outfile, basefile, + accumfile); + if (err) + goto done; + } + + *hdrlen = obj->hdrlen; +done: + got_object_close(obj); + if (outfile && fclose(outfile) == EOF && err == NULL) + err = got_error_from_errno("fclose"); + if (basefile && fclose(basefile) == EOF && err == NULL) + err = got_error_from_errno("fclose"); + if (accumfile && fclose(accumfile) == EOF && err == NULL) + err = got_error_from_errno("fclose"); + return err; + +} + +static void +put_raw_object_tempfile(struct got_raw_object *obj) +{ + struct got_repository *repo = obj->close_arg; + + if (obj->tempfile_idx != -1) + got_repo_temp_fds_put(obj->tempfile_idx, repo); +} + +/* *outfd must be initialized to -1 by caller */ +const struct got_error * +got_object_raw_open(struct got_raw_object **obj, int *outfd, + struct got_repository *repo, struct got_object_id *id) +{ + const struct got_error *err = NULL; + struct got_packidx *packidx = NULL; + int idx, tempfile_idx = -1; + uint8_t *outbuf = NULL; + off_t size = 0; + size_t hdrlen = 0; + char *path_packfile = NULL; + + *obj = got_repo_get_cached_raw_object(repo, id); + if (*obj != NULL) { + (*obj)->refcnt++; + return NULL; + } + + if (*outfd == -1) { + int tempfd; + + err = got_repo_temp_fds_get(&tempfd, &tempfile_idx, repo); + if (err) + return err; + + /* Duplicate tempfile descriptor to allow use of fdopen(3). */ + *outfd = dup(tempfd); + if (*outfd == -1) { + got_repo_temp_fds_put(tempfile_idx, repo); + return got_error_from_errno("dup"); + } + } + + err = got_repo_search_packidx(&packidx, &idx, repo, id); + if (err == NULL) { + struct got_pack *pack = NULL; + + err = got_packidx_get_packfile_path(&path_packfile, + packidx->path_packidx); + if (err) + goto done; + + pack = got_repo_get_cached_pack(repo, path_packfile); + if (pack == NULL) { + err = got_repo_cache_pack(&pack, repo, path_packfile, + packidx); + if (err) + goto done; + } + err = read_packed_object_raw(&outbuf, &size, &hdrlen, + *outfd, pack, packidx, idx, id); + if (err) + goto done; + } else if (err->code == GOT_ERR_NO_OBJ) { + int fd; + + err = got_object_open_loose_fd(&fd, id, repo); + if (err) + goto done; + err = got_object_read_raw(&outbuf, &size, &hdrlen, + GOT_DELTA_RESULT_SIZE_CACHED_MAX, *outfd, id, fd); + if (close(fd) == -1 && err == NULL) + err = got_error_from_errno("close"); + if (err) + goto done; + } + + err = got_object_raw_alloc(obj, outbuf, outfd, hdrlen, size); + if (err) + goto done; + + err = got_repo_cache_raw_object(repo, id, *obj); + if (err) { + if (err->code == GOT_ERR_OBJ_EXISTS || + err->code == GOT_ERR_OBJ_TOO_LARGE) + err = NULL; + } +done: + free(path_packfile); + if (err) { + if (*obj) { + got_object_raw_close(*obj); + *obj = NULL; + } + free(outbuf); + if (tempfile_idx != -1) + got_repo_temp_fds_put(tempfile_idx, repo); + } else { + (*obj)->tempfile_idx = tempfile_idx; + (*obj)->close_cb = put_raw_object_tempfile; + (*obj)->close_arg = repo; + } + return err; +} + +static const struct got_error * +open_commit(struct got_commit_object **commit, + struct got_repository *repo, struct got_object_id *id, int check_cache) +{ + const struct got_error *err = NULL; + struct got_packidx *packidx = NULL; + int idx; + char *path_packfile = NULL; + + if (check_cache) { + *commit = got_repo_get_cached_commit(repo, id); + if (*commit != NULL) { + (*commit)->refcnt++; + return NULL; + } + } else + *commit = NULL; + + err = got_repo_search_packidx(&packidx, &idx, repo, id); + if (err == NULL) { + struct got_pack *pack = NULL; + struct got_object *obj; + uint8_t *buf; + size_t len; + + err = got_packidx_get_packfile_path(&path_packfile, + packidx->path_packidx); + if (err) + return err; + + pack = got_repo_get_cached_pack(repo, path_packfile); + if (pack == NULL) { + err = got_repo_cache_pack(&pack, repo, path_packfile, + packidx); + if (err) + goto done; + } + err = got_packfile_open_object(&obj, pack, packidx, idx, id); + if (err) + goto done; + err = got_packfile_extract_object_to_mem(&buf, &len, + obj, pack); + got_object_close(obj); + if (err) + goto done; + err = got_object_parse_commit(commit, buf, len); + free(buf); + } else if (err->code == GOT_ERR_NO_OBJ) { + int fd; + + err = got_object_open_loose_fd(&fd, id, repo); + if (err) + return err; + err = got_object_read_commit(commit, fd, id, 0); + if (close(fd) == -1 && err == NULL) + err = got_error_from_errno("close"); + if (err) + return err; + } + + if (err == NULL) { + (*commit)->refcnt++; + err = got_repo_cache_commit(repo, id, *commit); + if (err) { + if (err->code == GOT_ERR_OBJ_EXISTS || + err->code == GOT_ERR_OBJ_TOO_LARGE) + err = NULL; + } + } +done: + free(path_packfile); + return err; +} + +const struct got_error * +got_object_open_as_commit(struct got_commit_object **commit, + struct got_repository *repo, struct got_object_id *id) +{ + *commit = got_repo_get_cached_commit(repo, id); + if (*commit != NULL) { + (*commit)->refcnt++; + return NULL; + } + + return open_commit(commit, repo, id, 0); +} + +const struct got_error * +got_object_commit_open(struct got_commit_object **commit, + struct got_repository *repo, struct got_object *obj) +{ + return open_commit(commit, repo, got_object_get_id(obj), 1); +} + +static const struct got_error * +open_tree(struct got_tree_object **tree, + struct got_repository *repo, struct got_object_id *id, int check_cache) +{ + const struct got_error *err = NULL; + struct got_packidx *packidx = NULL; + int idx; + char *path_packfile = NULL; + struct got_parsed_tree_entry *entries = NULL; + size_t nentries = 0, nentries_alloc = 0, i; + uint8_t *buf = NULL; + + if (check_cache) { + *tree = got_repo_get_cached_tree(repo, id); + if (*tree != NULL) { + (*tree)->refcnt++; + return NULL; + } + } else + *tree = NULL; + + err = got_repo_search_packidx(&packidx, &idx, repo, id); + if (err == NULL) { + struct got_pack *pack = NULL; + struct got_object *obj; + size_t len; + + err = got_packidx_get_packfile_path(&path_packfile, + packidx->path_packidx); + if (err) + return err; + + pack = got_repo_get_cached_pack(repo, path_packfile); + if (pack == NULL) { + err = got_repo_cache_pack(&pack, repo, path_packfile, + packidx); + if (err) + goto done; + } + err = got_packfile_open_object(&obj, pack, packidx, idx, id); + if (err) + goto done; + err = got_packfile_extract_object_to_mem(&buf, &len, + obj, pack); + got_object_close(obj); + if (err) + goto done; + err = got_object_parse_tree(&entries, &nentries, + &nentries_alloc, buf, len); + if (err) + goto done; + } else if (err->code == GOT_ERR_NO_OBJ) { + int fd; + + err = got_object_open_loose_fd(&fd, id, repo); + if (err) + return err; + err = got_object_read_tree(&entries, &nentries, + &nentries_alloc, &buf, fd, id); + if (close(fd) == -1 && err == NULL) + err = got_error_from_errno("close"); + if (err) + goto done; + } else + goto done; + + *tree = malloc(sizeof(**tree)); + if (*tree == NULL) { + err = got_error_from_errno("malloc"); + goto done; + } + (*tree)->entries = calloc(nentries, sizeof(struct got_tree_entry)); + if ((*tree)->entries == NULL) { + err = got_error_from_errno("malloc"); + goto done; + } + (*tree)->nentries = nentries; + (*tree)->refcnt = 0; + + for (i = 0; i < nentries; i++) { + struct got_parsed_tree_entry *pe = &entries[i]; + struct got_tree_entry *te = &(*tree)->entries[i]; + + if (strlcpy(te->name, pe->name, + sizeof(te->name)) >= sizeof(te->name)) { + err = got_error(GOT_ERR_NO_SPACE); + goto done; + } + memcpy(te->id.sha1, pe->id, SHA1_DIGEST_LENGTH); + te->mode = pe->mode; + te->idx = i; + } +done: + free(path_packfile); + free(entries); + free(buf); + if (err == NULL) { + (*tree)->refcnt++; + err = got_repo_cache_tree(repo, id, *tree); + if (err) { + if (err->code == GOT_ERR_OBJ_EXISTS || + err->code == GOT_ERR_OBJ_TOO_LARGE) + err = NULL; + } + } + if (err) { + if (*tree) + free((*tree)->entries); + free(*tree); + *tree = NULL; + } + return err; +} + +const struct got_error * +got_object_open_as_tree(struct got_tree_object **tree, + struct got_repository *repo, struct got_object_id *id) +{ + *tree = got_repo_get_cached_tree(repo, id); + if (*tree != NULL) { + (*tree)->refcnt++; + return NULL; + } + + return open_tree(tree, repo, id, 0); +} + +const struct got_error * +got_object_tree_open(struct got_tree_object **tree, + struct got_repository *repo, struct got_object *obj) +{ + return open_tree(tree, repo, got_object_get_id(obj), 1); +} + +const struct got_error * +got_object_open_as_blob(struct got_blob_object **blob, + struct got_repository *repo, struct got_object_id *id, size_t blocksize, + int outfd) +{ + return got_error(GOT_ERR_NOT_IMPL); +} + +const struct got_error * +got_object_blob_open(struct got_blob_object **blob, + struct got_repository *repo, struct got_object *obj, size_t blocksize, + int outfd) +{ + return got_error(GOT_ERR_NOT_IMPL); +} + +static const struct got_error * +open_tag(struct got_tag_object **tag, struct got_repository *repo, + struct got_object_id *id, int check_cache) +{ + const struct got_error *err = NULL; + struct got_packidx *packidx = NULL; + int idx; + char *path_packfile = NULL; + struct got_object *obj = NULL; + int obj_type = GOT_OBJ_TYPE_ANY; + + if (check_cache) { + *tag = got_repo_get_cached_tag(repo, id); + if (*tag != NULL) { + (*tag)->refcnt++; + return NULL; + } + } else + *tag = NULL; + + err = got_repo_search_packidx(&packidx, &idx, repo, id); + if (err == NULL) { + struct got_pack *pack = NULL; + uint8_t *buf = NULL; + size_t len; + + err = got_packidx_get_packfile_path(&path_packfile, + packidx->path_packidx); + if (err) + return err; + + pack = got_repo_get_cached_pack(repo, path_packfile); + if (pack == NULL) { + err = got_repo_cache_pack(&pack, repo, path_packfile, + packidx); + if (err) + goto done; + } + + /* Beware of "lightweight" tags: Check object type first. */ + err = got_packfile_open_object(&obj, pack, packidx, idx, id); + if (err) + goto done; + obj_type = obj->type; + if (obj_type != GOT_OBJ_TYPE_TAG) { + err = got_error(GOT_ERR_OBJ_TYPE); + got_object_close(obj); + goto done; + } + err = got_packfile_extract_object_to_mem(&buf, &len, + obj, pack); + got_object_close(obj); + if (err) + goto done; + err = got_object_parse_tag(tag, buf, len); + free(buf); + } else if (err->code == GOT_ERR_NO_OBJ) { + int fd; + + err = got_object_open_loose_fd(&fd, id, repo); + if (err) + return err; + err = got_object_read_header(&obj, fd); + if (close(fd) == -1 && err == NULL) + err = got_error_from_errno("close"); + if (err) + return err; + obj_type = obj->type; + got_object_close(obj); + if (obj_type != GOT_OBJ_TYPE_TAG) + return got_error(GOT_ERR_OBJ_TYPE); + + err = got_object_open_loose_fd(&fd, id, repo); + if (err) + return err; + err = got_object_read_tag(tag, fd, id, 0); + if (close(fd) == -1 && err == NULL) + err = got_error_from_errno("close"); + if (err) + return err; + } + + if (err == NULL) { + (*tag)->refcnt++; + err = got_repo_cache_tag(repo, id, *tag); + if (err) { + if (err->code == GOT_ERR_OBJ_EXISTS || + err->code == GOT_ERR_OBJ_TOO_LARGE) + err = NULL; + } + } +done: + free(path_packfile); + return err; +} + +const struct got_error * +got_object_open_as_tag(struct got_tag_object **tag, + struct got_repository *repo, struct got_object_id *id) +{ + *tag = got_repo_get_cached_tag(repo, id); + if (*tag != NULL) { + (*tag)->refcnt++; + return NULL; + } + + return open_tag(tag, repo, id, 0); +} + +const struct got_error * +got_object_tag_open(struct got_tag_object **tag, + struct got_repository *repo, struct got_object *obj) +{ + return open_tag(tag, repo, got_object_get_id(obj), 1); +} blob - c3e801e9c0d7b67f66bf77579aba98535cf054db blob + 44ccdb3791fa3057c6bda8686d44d57d48169e2e --- lib/pack_create.c +++ lib/pack_create.c @@ -28,6 +28,7 @@ #include #include #include +#include #include #include #include @@ -55,6 +56,7 @@ #include "got_lib_pack_create.h" #include "got_lib_repository.h" #include "got_lib_inflate.h" +#include "got_lib_poll.h" #include "murmurhash2.h" @@ -1419,24 +1421,17 @@ done: static const struct got_error * hwrite(int fd, const void *buf, off_t len, SHA1_CTX *ctx) { - ssize_t w; - SHA1Update(ctx, buf, len); - w = write(fd, buf, len); - if (w == -1) - return got_error_from_errno("write"); - else if (w != len) - return got_error(GOT_ERR_IO); - return NULL; + return got_poll_write_full(fd, buf, len); } static const struct got_error * hcopy(FILE *fsrc, int fd_dst, off_t len, SHA1_CTX *ctx) { + const struct got_error *err; unsigned char buf[65536]; off_t remain = len; size_t n; - ssize_t w; while (remain > 0) { size_t copylen = MIN(sizeof(buf), remain); @@ -1444,11 +1439,9 @@ hcopy(FILE *fsrc, int fd_dst, off_t len, SHA1_CTX *ctx if (n != copylen) return got_ferror(fsrc, GOT_ERR_IO); SHA1Update(ctx, buf, copylen); - w = write(fd_dst, buf, copylen); - if (w == -1) - return got_error_from_errno("write"); - else if (w != copylen) - return got_error(GOT_ERR_IO); + err = got_poll_write_full(fd_dst, buf, copylen); + if (err) + return err; remain -= copylen; } @@ -1459,18 +1452,11 @@ static const struct got_error * hcopy_mmap(uint8_t *src, off_t src_offset, size_t src_size, int fd, off_t len, SHA1_CTX *ctx) { - ssize_t w; - if (src_offset + len > src_size) return got_error(GOT_ERR_RANGE); SHA1Update(ctx, src + src_offset, len); - w = write(fd, src + src_offset, len); - if (w == -1) - return got_error_from_errno("write"); - else if (w != len) - return got_error(GOT_ERR_IO); - return NULL; + return got_poll_write_full(fd, src + src_offset, len); } static void @@ -1706,7 +1692,6 @@ genpack(uint8_t *pack_sha1, int packfd, FILE *delta_ca SHA1_CTX ctx; struct got_pack_meta *m; char buf[32]; - ssize_t w; off_t packfile_size = 0; int outfd = -1; int delta_cache_fd = -1; @@ -1782,11 +1767,7 @@ genpack(uint8_t *pack_sha1, int packfd, FILE *delta_ca } SHA1Final(pack_sha1, &ctx); - w = write(packfd, pack_sha1, SHA1_DIGEST_LENGTH); - if (w == -1) - err = got_error_from_errno("write"); - else if (w != SHA1_DIGEST_LENGTH) - err = got_error(GOT_ERR_IO); + err = got_poll_write_full(packfd, pack_sha1, SHA1_DIGEST_LENGTH); if (err) goto done; packfile_size += SHA1_DIGEST_LENGTH; @@ -1915,6 +1896,20 @@ got_pack_create(uint8_t *packsha1, int packfd, FILE *d err = got_error_from_errno("fflush"); goto done; } + + if (progress_cb) { + /* + * Report a 1-byte packfile write to indicate we are about + * to start sending packfile data. gotd(8) needs this. + */ + err = progress_cb(progress_arg, ncolored, nfound, ntrees, + 1 /* packfile_size */, nours, + got_object_idset_num_elements(idset), + deltify.nmeta + reuse.nmeta, 0); + if (err) + goto done; + } + err = genpack(packsha1, packfd, delta_cache, deltify.meta, deltify.nmeta, reuse.meta, reuse.nmeta, ncolored, nfound, ntrees, nours, repo, progress_cb, progress_arg, rl, blob - e85d5490e5500a2547fce3244c397bc70e0d1dfe blob + b1c742e556836c8ebe34fdc0737b73b17fa2a931 --- lib/pack_index.c +++ lib/pack_index.c @@ -354,8 +354,8 @@ read_packed_object(struct got_pack *pack, struct got_i return err; } -static const struct got_error * -hwrite(int fd, void *buf, int len, SHA1_CTX *ctx) +const struct got_error * +got_pack_hwrite(int fd, void *buf, int len, SHA1_CTX *ctx) { ssize_t w; @@ -783,9 +783,9 @@ got_pack_index(struct got_pack *pack, int idxfd, FILE * verify its checksum. */ SHA1Final(pack_sha1, &ctx); + if (memcmp(pack_sha1_expected, pack_sha1, SHA1_DIGEST_LENGTH) != 0) { - err = got_error_msg(GOT_ERR_BAD_PACKFILE, - "pack file checksum mismatch"); + err = got_error(GOT_ERR_PACKFILE_CSUM); goto done; } @@ -913,31 +913,32 @@ got_pack_index(struct got_pack *pack, int idxfd, FILE SHA1Init(&ctx); putbe32(buf, GOT_PACKIDX_V2_MAGIC); putbe32(buf + 4, GOT_PACKIDX_VERSION); - err = hwrite(idxfd, buf, 8, &ctx); + err = got_pack_hwrite(idxfd, buf, 8, &ctx); if (err) goto done; - err = hwrite(idxfd, packidx.hdr.fanout_table, + err = got_pack_hwrite(idxfd, packidx.hdr.fanout_table, GOT_PACKIDX_V2_FANOUT_TABLE_ITEMS * sizeof(uint32_t), &ctx); if (err) goto done; - err = hwrite(idxfd, packidx.hdr.sorted_ids, + err = got_pack_hwrite(idxfd, packidx.hdr.sorted_ids, nobj * SHA1_DIGEST_LENGTH, &ctx); if (err) goto done; - err = hwrite(idxfd, packidx.hdr.crc32, nobj * sizeof(uint32_t), &ctx); + err = got_pack_hwrite(idxfd, packidx.hdr.crc32, + nobj * sizeof(uint32_t), &ctx); if (err) goto done; - err = hwrite(idxfd, packidx.hdr.offsets, nobj * sizeof(uint32_t), - &ctx); + err = got_pack_hwrite(idxfd, packidx.hdr.offsets, + nobj * sizeof(uint32_t), &ctx); if (err) goto done; if (packidx.nlargeobj > 0) { - err = hwrite(idxfd, packidx.hdr.large_offsets, + err = got_pack_hwrite(idxfd, packidx.hdr.large_offsets, packidx.nlargeobj * sizeof(uint64_t), &ctx); if (err) goto done; } - err = hwrite(idxfd, pack_sha1, SHA1_DIGEST_LENGTH, &ctx); + err = got_pack_hwrite(idxfd, pack_sha1, SHA1_DIGEST_LENGTH, &ctx); if (err) goto done; blob - /dev/null blob + 2adcf9e794f2de58f508160da8633a854ebfdef9 (mode 644) --- /dev/null +++ lib/pack_create_io.c @@ -0,0 +1,384 @@ +/* + * Copyright (c) 2020 Ori Bernstein + * Copyright (c) 2021, 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "got_error.h" +#include "got_cancel.h" +#include "got_object.h" +#include "got_reference.h" +#include "got_repository_admin.h" +#include "got_path.h" + +#include "got_lib_delta.h" +#include "got_lib_object.h" +#include "got_lib_object_cache.h" +#include "got_lib_object_idset.h" +#include "got_lib_ratelimit.h" +#include "got_lib_pack.h" +#include "got_lib_pack_create.h" +#include "got_lib_repository.h" + +static const struct got_error * +get_base_object_id(struct got_object_id *base_id, struct got_packidx *packidx, + off_t base_offset) +{ + const struct got_error *err; + int idx; + + err = got_packidx_get_offset_idx(&idx, packidx, base_offset); + if (err) + return err; + if (idx == -1) + return got_error(GOT_ERR_BAD_PACKIDX); + + return got_packidx_get_object_id(base_id, packidx, idx); +} + +struct search_deltas_arg { + struct got_pack_metavec *v; + struct got_packidx *packidx; + struct got_pack *pack; + struct got_object_idset *idset; + int delta_cache_fd; + int ncolored, nfound, ntrees, ncommits; + got_pack_progress_cb progress_cb; + void *progress_arg; + struct got_ratelimit *rl; + got_cancel_cb cancel_cb; + void *cancel_arg; +}; + +static const struct got_error * +search_delta_for_object(struct got_object_id *id, void *data, void *arg) +{ + const struct got_error *err; + struct search_deltas_arg *a = arg; + int obj_idx; + uint8_t *delta_buf = NULL; + uint64_t base_size, result_size; + size_t delta_size, delta_compressed_size; + off_t delta_offset, base_offset; + struct got_object_id base_id; + + if (a->cancel_cb) { + err = a->cancel_cb(a->cancel_arg); + if (err) + return err; + } + + obj_idx = got_packidx_get_object_idx(a->packidx, id); + if (obj_idx == -1) + return NULL; /* object not present in our pack file */ + + err = got_packfile_extract_raw_delta(&delta_buf, &delta_size, + &delta_compressed_size, &delta_offset, &base_offset, &base_id, + &base_size, &result_size, a->pack, a->packidx, obj_idx); + if (err) { + if (err->code == GOT_ERR_OBJ_TYPE) + return NULL; /* object not stored as a delta */ + return err; + } + + /* + * If this is an offset delta we must determine the base + * object ID ourselves. + */ + if (base_offset != 0) { + err = get_base_object_id(&base_id, a->packidx, base_offset); + if (err) + goto done; + } + + if (got_object_idset_contains(a->idset, &base_id)) { + struct got_pack_meta *m, *base; + off_t delta_out_offset; + ssize_t w; + + delta_out_offset = lseek(a->delta_cache_fd, 0, SEEK_CUR); + if (delta_out_offset == -1) { + err = got_error_from_errno("lseek"); + goto done; + } + w = write(a->delta_cache_fd, delta_buf, delta_compressed_size); + if (w != delta_compressed_size) { + err = got_error_from_errno("write"); + goto done; + } + + m = got_object_idset_get(a->idset, id); + if (m == NULL) { + err = got_error_msg(GOT_ERR_NO_OBJ, + "delta object not found"); + goto done; + } + + base = got_object_idset_get(a->idset, &base_id); + if (m == NULL) { + err = got_error_msg(GOT_ERR_NO_OBJ, + "delta base object not found"); + goto done; + } + + m->base_obj_id = got_object_id_dup(&base_id); + if (m->base_obj_id == NULL) { + err = got_error_from_errno("got_object_id_dup"); + goto done; + } + + m->prev = base; + m->size = result_size; + m->delta_len = delta_size; + m->delta_compressed_len = delta_compressed_size; + m->reused_delta_offset = delta_offset; + m->delta_offset = delta_out_offset; + + err = got_pack_add_meta(m, a->v); + if (err) + goto done; + + err = got_pack_report_progress(a->progress_cb, a->progress_arg, + a->rl, a->ncolored, a->nfound, a->ntrees, 0L, a->ncommits, + got_object_idset_num_elements(a->idset), a->v->nmeta, 0); + if (err) + goto done; + } +done: + free(delta_buf); + return err; +} + +const struct got_error * +got_pack_search_deltas(struct got_pack_metavec *v, + struct got_object_idset *idset, int delta_cache_fd, + int ncolored, int nfound, int ntrees, int ncommits, + struct got_repository *repo, + got_pack_progress_cb progress_cb, void *progress_arg, + struct got_ratelimit *rl, got_cancel_cb cancel_cb, void *cancel_arg) +{ + const struct got_error *err = NULL; + struct got_packidx *packidx; + struct got_pack *pack; + struct search_deltas_arg sda; + + err = got_pack_find_pack_for_reuse(&packidx, repo); + if (err) + return err; + + if (packidx == NULL) + return NULL; + + err = got_pack_cache_pack_for_packidx(&pack, packidx, repo); + if (err) + return err; + + memset(&sda, 0, sizeof(sda)); + sda.v = v; + sda.idset = idset; + sda.pack = pack; + sda.packidx = packidx; + sda.delta_cache_fd = delta_cache_fd; + sda.ncolored = ncolored; + sda.nfound = nfound; + sda.ntrees = ntrees; + sda.ncommits = ncommits; + sda.progress_cb = progress_cb; + sda.progress_arg = progress_arg; + sda.rl = rl; + sda.cancel_cb = cancel_cb; + sda.cancel_arg = cancel_arg; + return got_object_idset_for_each(idset, search_delta_for_object, &sda); +} + +const struct got_error * +got_pack_load_packed_object_ids(int *found_all_objects, + struct got_object_id **ours, int nours, + struct got_object_id **theirs, int ntheirs, + int want_meta, uint32_t seed, struct got_object_idset *idset, + struct got_object_idset *idset_exclude, int loose_obj_only, + struct got_repository *repo, struct got_packidx *packidx, + int *ncolored, int *nfound, int *ntrees, + got_pack_progress_cb progress_cb, void *progress_arg, + struct got_ratelimit *rl, got_cancel_cb cancel_cb, void *cancel_arg) +{ + /* We do not need this optimized traversal while using direct I/O. */ + *found_all_objects = 0; + return NULL; +} + +const struct got_error * +got_pack_paint_commits(int *ncolored, struct got_object_id_queue *ids, int nids, + struct got_object_idset *keep, struct got_object_idset *drop, + struct got_object_idset *skip, struct got_repository *repo, + got_pack_progress_cb progress_cb, void *progress_arg, + struct got_ratelimit *rl, got_cancel_cb cancel_cb, void *cancel_arg) +{ + const struct got_error *err = NULL; + struct got_commit_object *commit = NULL; + struct got_packidx *packidx = NULL; + struct got_pack *pack = NULL; + const struct got_object_id_queue *parents; + struct got_object_qid *qid = NULL; + int nqueued = nids, nskip = 0; + + while (!STAILQ_EMPTY(ids) && nskip != nqueued) { + intptr_t color; + + if (cancel_cb) { + err = cancel_cb(cancel_arg); + if (err) + break; + } + + qid = STAILQ_FIRST(ids); + STAILQ_REMOVE_HEAD(ids, entry); + nqueued--; + color = (intptr_t)qid->data; + if (color == COLOR_SKIP) + nskip--; + + if (got_object_idset_contains(skip, &qid->id)) { + got_object_qid_free(qid); + qid = NULL; + continue; + } + if (color == COLOR_KEEP && + got_object_idset_contains(keep, &qid->id)) { + got_object_qid_free(qid); + qid = NULL; + continue; + } + if (color == COLOR_DROP && + got_object_idset_contains(drop, &qid->id)) { + got_object_qid_free(qid); + qid = NULL; + continue; + } + + switch (color) { + case COLOR_KEEP: + if (got_object_idset_contains(drop, &qid->id)) { + err = got_pack_paint_commit(qid, COLOR_SKIP); + if (err) + goto done; + } else + (*ncolored)++; + err = got_object_idset_add(keep, &qid->id, NULL); + if (err) + goto done; + break; + case COLOR_DROP: + if (got_object_idset_contains(keep, &qid->id)) { + err = got_pack_paint_commit(qid, COLOR_SKIP); + if (err) + goto done; + } else + (*ncolored)++; + err = got_object_idset_add(drop, &qid->id, NULL); + if (err) + goto done; + break; + case COLOR_SKIP: + if (!got_object_idset_contains(skip, &qid->id)) { + err = got_object_idset_add(skip, &qid->id, + NULL); + if (err) + goto done; + } + break; + default: + /* should not happen */ + err = got_error_fmt(GOT_ERR_NOT_IMPL, + "%s invalid commit color %"PRIdPTR, __func__, + color); + goto done; + } + + err = got_pack_report_progress(progress_cb, progress_arg, rl, + *ncolored, 0, 0, 0L, 0, 0, 0, 0); + if (err) + break; + + err = got_object_open_as_commit(&commit, repo, &qid->id); + if (err) + break; + + parents = got_object_commit_get_parent_ids(commit); + if (parents) { + struct got_object_qid *pid; + color = (intptr_t)qid->data; + STAILQ_FOREACH(pid, parents, entry) { + err = got_pack_queue_commit_id(ids, &pid->id, + color, repo); + if (err) + break; + nqueued++; + if (color == COLOR_SKIP) + nskip++; + } + } + + if (pack == NULL && (commit->flags & GOT_COMMIT_FLAG_PACKED)) { + /* + * We now know that at least one pack file exists. + * Pin a suitable pack to ensure it remains cached + * while we are churning through commit history. + */ + if (packidx == NULL) { + err = got_pack_find_pack_for_commit_painting( + &packidx, ids, nqueued, repo); + if (err) + goto done; + } + if (packidx != NULL) { + err = got_pack_cache_pack_for_packidx(&pack, + packidx, repo); + if (err) + goto done; + err = got_repo_pin_pack(repo, packidx, pack); + if (err) + goto done; + } + } + + got_object_commit_close(commit); + commit = NULL; + + got_object_qid_free(qid); + qid = NULL; + } +done: + if (commit) + got_object_commit_close(commit); + got_object_qid_free(qid); + got_repo_unpin_pack(repo); + return err; +} blob - 22b5fe916ef3380f1ead6796d6ddd366191190a2 blob + 175c3a0c4138ebc26f9e5160af4f028ccf1f46e7 --- lib/privsep.c +++ lib/privsep.c @@ -33,7 +33,6 @@ #include #include #include -#include #include "got_object.h" #include "got_error.h" @@ -47,6 +46,7 @@ #include "got_lib_object_parse.h" #include "got_lib_privsep.h" #include "got_lib_pack.h" +#include "got_lib_poll.h" #include "got_privsep.h" @@ -59,46 +59,17 @@ #endif static const struct got_error * -poll_fd(int fd, int events, int timeout) -{ - struct pollfd pfd[1]; - struct timespec ts; - sigset_t sigset; - int n; - - pfd[0].fd = fd; - pfd[0].events = events; - - ts.tv_sec = timeout; - ts.tv_nsec = 0; - - if (sigemptyset(&sigset) == -1) - return got_error_from_errno("sigemptyset"); - if (sigaddset(&sigset, SIGWINCH) == -1) - return got_error_from_errno("sigaddset"); - - n = ppoll(pfd, 1, timeout == INFTIM ? NULL : &ts, &sigset); - if (n == -1) - return got_error_from_errno("ppoll"); - if (n == 0) - return got_error(GOT_ERR_TIMEOUT); - if (pfd[0].revents & (POLLERR | POLLNVAL)) - return got_error_from_errno("poll error"); - if (pfd[0].revents & (events | POLLHUP)) - return NULL; - - return got_error(GOT_ERR_INTERRUPT); -} - -static const struct got_error * read_imsg(struct imsgbuf *ibuf) { const struct got_error *err; size_t n; - err = poll_fd(ibuf->fd, POLLIN, INFTIM); - if (err) + err = got_poll_fd(ibuf->fd, POLLIN, INFTIM); + if (err) { + if (err->code == GOT_ERR_EOF) + return got_error(GOT_ERR_PRIVSEP_PIPE); return err; + } n = imsg_read(ibuf); if (n == -1) { @@ -199,7 +170,7 @@ got_privsep_send_error(struct imsgbuf *ibuf, const str return; } - poll_err = poll_fd(ibuf->fd, POLLOUT, INFTIM); + poll_err = got_poll_fd(ibuf->fd, POLLOUT, INFTIM); if (poll_err) { fprintf(stderr, "%s: error %d \"%s\": poll: %s\n", getprogname(), err->code, err->msg, poll_err->msg); @@ -220,7 +191,7 @@ flush_imsg(struct imsgbuf *ibuf) { const struct got_error *err; - err = poll_fd(ibuf->fd, POLLOUT, INFTIM); + err = got_poll_fd(ibuf->fd, POLLOUT, INFTIM); if (err) return err; blob - /dev/null blob + 32efcd7f1a512fe6f1444aaba4ff7e8aa6d051b1 (mode 644) --- /dev/null +++ lib/pollfd.c @@ -0,0 +1,114 @@ +/* + * Copyright (c) 2018, 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#include +#include +#include +#include +#include +#include + +#include "got_error.h" + +#include "got_lib_poll.h" + +const struct got_error * +got_poll_fd(int fd, int events, int timeout) +{ + struct pollfd pfd[1]; + struct timespec ts; + sigset_t sigset; + int n; + + pfd[0].fd = fd; + pfd[0].events = events; + + ts.tv_sec = timeout; + ts.tv_nsec = 0; + + if (sigemptyset(&sigset) == -1) + return got_error_from_errno("sigemptyset"); + if (sigaddset(&sigset, SIGWINCH) == -1) + return got_error_from_errno("sigaddset"); + + n = ppoll(pfd, 1, timeout == INFTIM ? NULL : &ts, &sigset); + if (n == -1) + return got_error_from_errno("ppoll"); + if (n == 0) { + if (pfd[0].revents & POLLHUP) + return got_error(GOT_ERR_EOF); + return got_error(GOT_ERR_TIMEOUT); + } + if (pfd[0].revents & (POLLERR | POLLNVAL)) + return got_error_from_errno("poll error"); + if (pfd[0].revents & events) + return NULL; + if (pfd[0].revents & POLLHUP) + return got_error(GOT_ERR_EOF); + + return got_error(GOT_ERR_INTERRUPT); +} + +const struct got_error * +got_poll_read_full(int fd, size_t *len, void *buf, size_t bufsize, + size_t minbytes) +{ + const struct got_error *err = NULL; + size_t have = 0; + ssize_t r; + + if (minbytes > bufsize) + return got_error(GOT_ERR_NO_SPACE); + + while (have < minbytes) { + err = got_poll_fd(fd, POLLIN, INFTIM); + if (err) + return err; + r = read(fd, buf + have, bufsize - have); + if (r == -1) + return got_error_from_errno("read"); + if (r == 0) + return got_error(GOT_ERR_EOF); + have += r; + } + + *len = have; + return NULL; +} + +const struct got_error * +got_poll_write_full(int fd, const void *buf, off_t len) +{ + const struct got_error *err = NULL; + off_t wlen = 0; + ssize_t w = 0; + + while (wlen != len) { + if (wlen > 0) { + err = got_poll_fd(fd, POLLOUT, INFTIM); + if (err) + return err; + } + w = write(fd, buf + wlen, len - wlen); + if (w == -1) { + if (errno != EAGAIN) + return got_error_from_errno("write"); + } else + wlen += w; + } + + return NULL; +} blob - d6f94939b844a5e06b4b92355a08680544dba996 blob + b5f93d7b7eba07c7f32fdad87614319943f14305 --- lib/reference.c +++ lib/reference.c @@ -19,7 +19,6 @@ #include #include -#include #include #include #include @@ -266,57 +265,6 @@ get_refs_dir_path(struct got_repository *repo, const c return strdup(got_repo_get_path_git_dir(repo)); return got_repo_get_path_refs(repo); -} - -int -got_ref_name_is_valid(const char *name) -{ - const char *s, *seg; - const char forbidden[] = { ' ', '~', '^', ':', '?', '*', '[' , '\\' }; - const char *forbidden_seq[] = { "//", "..", "@{" }; - const char *lfs = GOT_LOCKFILE_SUFFIX; - const size_t lfs_len = sizeof(GOT_LOCKFILE_SUFFIX) - 1; - size_t i; - - if (name[0] == '@' && name[1] == '\0') - return 0; - - s = name; - seg = s; - if (seg[0] == '\0' || seg[0] == '.' || seg[0] == '/') - return 0; - while (*s) { - for (i = 0; i < nitems(forbidden); i++) { - if (*s == forbidden[i]) - return 0; - } - for (i = 0; i < nitems(forbidden_seq); i++) { - if (s[0] == forbidden_seq[i][0] && - s[1] == forbidden_seq[i][1]) - return 0; - } - if (iscntrl((unsigned char)s[0])) - return 0; - if (s[0] == '.' && s[1] == '\0') - return 0; - if (*s == '/') { - const char *nextseg = s + 1; - if (nextseg[0] == '\0' || nextseg[0] == '.' || - nextseg[0] == '/') - return 0; - if (seg <= s - lfs_len && - strncmp(s - lfs_len, lfs, lfs_len) == 0) - return 0; - seg = nextseg; - } - s++; - } - - if (seg <= s - lfs_len && - strncmp(s - lfs_len, lfs, lfs_len) == 0) - return 0; - - return 1; } const struct got_error * blob - c737a5e9ab0c0f7d5eaf7bd9936fd0f21b0edf45 blob + b07bf0c9c711997031a7d3afa3388a27527ef2db --- lib/repository.c +++ lib/repository.c @@ -53,6 +53,7 @@ #include "got_opentemp.h" #include "got_lib_delta.h" +#include "got_lib_delta_cache.h" #include "got_lib_inflate.h" #include "got_lib_object.h" #include "got_lib_object_parse.h" @@ -248,31 +249,43 @@ done: } -const struct got_error * -got_repo_pack_fds_open(int **pack_fds) +static const struct got_error * +close_tempfiles(int *fds, size_t nfds) { const struct got_error *err = NULL; int i; - *pack_fds = calloc(GOT_PACK_NUM_TEMPFILES, sizeof(**pack_fds)); - if (*pack_fds == NULL) - return got_error_from_errno("calloc"); - - /* - * got_repo_pack_fds_close will try to close all of the - * GOT_PACK_NUM_TEMPFILES fds, even the ones that didn't manage to get - * a value from got_opentempfd(), which would result in a close(0) if - * we do not initialize to -1 here. - */ - for (i = 0; i < GOT_PACK_NUM_TEMPFILES; i++) - (*pack_fds)[i] = -1; + for (i = 0; i < nfds; i++) { + if (fds[i] == -1) + continue; + if (close(fds[i]) == -1) { + err = got_error_from_errno("close"); + break; + } + } + free(fds); + return err; +} - for (i = 0; i < GOT_PACK_NUM_TEMPFILES; i++) { - (*pack_fds)[i] = got_opentempfd(); - if ((*pack_fds)[i] == -1) { +static const struct got_error * +open_tempfiles(int **fds, size_t nfds) +{ + const struct got_error *err = NULL; + int i; + + *fds = calloc(nfds, sizeof(**fds)); + if (*fds == NULL) + return got_error_from_errno("calloc"); + + for (i = 0; i < nfds; i++) + (*fds)[i] = -1; + + for (i = 0; i < nfds; i++) { + (*fds)[i] = got_opentempfd(); + if ((*fds)[i] == -1) { err = got_error_from_errno("got_opentempfd"); - got_repo_pack_fds_close(*pack_fds); - *pack_fds = NULL; + close_tempfiles(*fds, nfds); + *fds = NULL; return err; } } @@ -281,21 +294,66 @@ got_repo_pack_fds_open(int **pack_fds) } const struct got_error * +got_repo_pack_fds_open(int **pack_fds) +{ + return open_tempfiles(pack_fds, GOT_PACK_NUM_TEMPFILES); +} + +const struct got_error * got_repo_pack_fds_close(int *pack_fds) { - const struct got_error *err = NULL; - int i; + return close_tempfiles(pack_fds, GOT_PACK_NUM_TEMPFILES); +} - for (i = 0; i < GOT_PACK_NUM_TEMPFILES; i++) { - if (pack_fds[i] == -1) +const struct got_error * +got_repo_temp_fds_open(int **temp_fds) +{ + return open_tempfiles(temp_fds, GOT_REPO_NUM_TEMPFILES); +} + +void +got_repo_temp_fds_set(struct got_repository *repo, int *temp_fds) +{ + int i; + + for (i = 0; i < GOT_REPO_NUM_TEMPFILES; i++) + repo->tempfiles[i] = temp_fds[i]; +} + +const struct got_error * +got_repo_temp_fds_get(int *fd, int *idx, struct got_repository *repo) +{ + int i; + + *fd = -1; + *idx = -1; + + for (i = 0; i < nitems(repo->tempfiles); i++) { + if (repo->tempfile_use_mask & (1 << i)) continue; - if (close(pack_fds[i]) == -1) { - err = got_error_from_errno("close"); - break; + if (repo->tempfiles[i] != -1) { + if (ftruncate(repo->tempfiles[i], 0L) == -1) + return got_error_from_errno("ftruncate"); + *fd = repo->tempfiles[i]; + *idx = i; + repo->tempfile_use_mask |= (1 << i); + return NULL; } } - free(pack_fds); - return err; + + return got_error(GOT_ERR_REPO_TEMPFILE); +} + +void +got_repo_temp_fds_put(int idx, struct got_repository *repo) +{ + repo->tempfile_use_mask &= ~(1 << idx); +} + +const struct got_error * +got_repo_temp_fds_close(int *temp_fds) +{ + return close_tempfiles(temp_fds, GOT_REPO_NUM_TEMPFILES); } const struct got_error * @@ -625,7 +683,7 @@ got_repo_open(struct got_repository **repop, const cha if (repo->pack_cache_size > rl.rlim_cur / 8) repo->pack_cache_size = rl.rlim_cur / 8; for (i = 0; i < nitems(repo->packs); i++) { - if (i < repo->pack_cache_size) { + if (pack_fds != NULL && i < repo->pack_cache_size) { repo->packs[i].basefd = pack_fds[j++]; repo->packs[i].accumfd = pack_fds[j++]; } else { @@ -633,6 +691,8 @@ got_repo_open(struct got_repository **repop, const cha repo->packs[i].accumfd = -1; } } + for (i = 0; i < nitems(repo->tempfiles); i++) + repo->tempfiles[i] = -1; repo->pinned_pack = -1; repo->pinned_packidx = -1; repo->pinned_pid = 0; @@ -1362,6 +1422,10 @@ got_repo_cache_pack(struct got_pack **packp, struct go pack->filesize = sb.st_size; pack->privsep_child = NULL; + + err = got_delta_cache_alloc(&pack->delta_cache); + if (err) + goto done; #ifndef GOT_PACK_NO_MMAP pack->map = mmap(NULL, pack->filesize, PROT_READ, MAP_PRIVATE, @@ -1376,10 +1440,8 @@ got_repo_cache_pack(struct got_pack **packp, struct go #endif done: if (err) { - if (pack) { - free(pack->path_packfile); - memset(pack, 0, sizeof(*pack)); - } + if (pack) + got_pack_close(pack); } else if (packp) *packp = pack; return err; blob - /dev/null blob + d46762d0f8fd908e9beebe23dd87ff3740db4068 (mode 644) --- /dev/null +++ lib/read_gitconfig.c @@ -0,0 +1,274 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "got_error.h" +#include "got_object.h" +#include "got_repository.h" +#include "got_path.h" + +#include "got_lib_gitconfig.h" +#include "got_lib_delta.h" +#include "got_lib_object.h" +#include "got_lib_object_cache.h" +#include "got_lib_privsep.h" +#include "got_lib_pack.h" +#include "got_lib_repository.h" + +static int +get_boolean_val(char *val) +{ + return (strcasecmp(val, "true") == 0 || + strcasecmp(val, "on") == 0 || + strcasecmp(val, "yes") == 0 || + strcmp(val, "1") == 0); +} + +const struct got_error * +got_repo_read_gitconfig(int *gitconfig_repository_format_version, + char **gitconfig_author_name, char **gitconfig_author_email, + struct got_remote_repo **remotes, int *nremotes, + char **gitconfig_owner, char ***extensions, int *nextensions, + const char *gitconfig_path) +{ + const struct got_error *err = NULL; + struct got_gitconfig *gitconfig = NULL; + struct got_gitconfig_list *tags; + struct got_gitconfig_list_node *node; + int fd, i; + const char *author, *email, *owner; + + *gitconfig_repository_format_version = 0; + if (extensions) + *extensions = NULL; + if (nextensions) + *nextensions = 0; + *gitconfig_author_name = NULL; + *gitconfig_author_email = NULL; + if (remotes) + *remotes = NULL; + if (nremotes) + *nremotes = 0; + if (gitconfig_owner) + *gitconfig_owner = NULL; + + fd = open(gitconfig_path, O_RDONLY | O_CLOEXEC); + if (fd == -1) { + if (errno == ENOENT) + return NULL; + return got_error_from_errno2("open", gitconfig_path); + } + + err = got_gitconfig_open(&gitconfig, fd); + if (err) + goto done; + + *gitconfig_repository_format_version = got_gitconfig_get_num(gitconfig, + "core", "repositoryformatversion", 0); + + tags = got_gitconfig_get_tag_list(gitconfig, "extensions"); + if (extensions && nextensions && tags) { + size_t numext = 0; + TAILQ_FOREACH(node, &tags->fields, link) { + char *ext = node->field; + char *val = got_gitconfig_get_str(gitconfig, + "extensions", ext); + if (get_boolean_val(val)) + numext++; + } + *extensions = calloc(numext, sizeof(char *)); + if (*extensions == NULL) { + err = got_error_from_errno("calloc"); + goto done; + } + TAILQ_FOREACH(node, &tags->fields, link) { + char *ext = node->field; + char *val = got_gitconfig_get_str(gitconfig, + "extensions", ext); + if (get_boolean_val(val)) { + char *extstr = strdup(ext); + if (extstr == NULL) { + err = got_error_from_errno("strdup"); + goto done; + } + (*extensions)[(*nextensions)] = extstr; + (*nextensions)++; + } + } + } + + author = got_gitconfig_get_str(gitconfig, "user", "name"); + if (author) { + *gitconfig_author_name = strdup(author); + if (*gitconfig_author_name == NULL) { + err = got_error_from_errno("strdup"); + goto done; + } + } + + email = got_gitconfig_get_str(gitconfig, "user", "email"); + if (email) { + *gitconfig_author_email = strdup(email); + if (*gitconfig_author_email == NULL) { + err = got_error_from_errno("strdup"); + goto done; + } + } + + if (gitconfig_owner) { + owner = got_gitconfig_get_str(gitconfig, "gotweb", "owner"); + if (owner == NULL) + owner = got_gitconfig_get_str(gitconfig, "gitweb", + "owner"); + if (owner) { + *gitconfig_owner = strdup(owner); + if (*gitconfig_owner == NULL) { + err = got_error_from_errno("strdup"); + goto done; + } + + } + } + + if (remotes && nremotes) { + struct got_gitconfig_list *sections; + size_t nalloc = 0; + err = got_gitconfig_get_section_list(§ions, gitconfig); + if (err) + return err; + TAILQ_FOREACH(node, §ions->fields, link) { + if (strncasecmp("remote \"", node->field, 8) != 0) + continue; + nalloc++; + } + + *remotes = recallocarray(NULL, 0, nalloc, sizeof(**remotes)); + if (*remotes == NULL) { + err = got_error_from_errno("recallocarray"); + goto done; + } + + i = 0; + TAILQ_FOREACH(node, §ions->fields, link) { + struct got_remote_repo *remote; + char *name, *end, *mirror; + const char *fetch_url, *send_url; + + if (strncasecmp("remote \"", node->field, 8) != 0) + continue; + + remote = &(*remotes)[i]; + + name = strdup(node->field + 8); + if (name == NULL) { + err = got_error_from_errno("strdup"); + goto done; + } + end = strrchr(name, '"'); + if (end) + *end = '\0'; + remote->name = name; + + fetch_url = got_gitconfig_get_str(gitconfig, + node->field, "url"); + if (fetch_url == NULL) { + err = got_error(GOT_ERR_GITCONFIG_SYNTAX); + free(remote->name); + remote->name = NULL; + goto done; + } + remote->fetch_url = strdup(fetch_url); + if (remote->fetch_url == NULL) { + err = got_error_from_errno("strdup"); + free(remote->name); + remote->name = NULL; + goto done; + } + + send_url = got_gitconfig_get_str(gitconfig, + node->field, "pushurl"); + if (send_url == NULL) + send_url = got_gitconfig_get_str(gitconfig, + node->field, "url"); + if (send_url == NULL) { + err = got_error(GOT_ERR_GITCONFIG_SYNTAX); + free(remote->name); + remote->name = NULL; + free(remote->fetch_url); + remote->fetch_url = NULL; + goto done; + } + remote->send_url = strdup(send_url); + if (remote->send_url == NULL) { + err = got_error_from_errno("strdup"); + free(remote->name); + remote->name = NULL; + free(remote->fetch_url); + remote->fetch_url = NULL; + goto done; + } + + remote->mirror_references = 0; + mirror = got_gitconfig_get_str(gitconfig, node->field, + "mirror"); + if (mirror != NULL && get_boolean_val(mirror)) + remote->mirror_references = 1; + + i++; + (*nremotes)++; + } + } +done: + if (fd != -1) + close(fd); + if (gitconfig) + got_gitconfig_close(gitconfig); + if (err) { + if (extensions && nextensions) { + for (i = 0; i < (*nextensions); i++) + free((*extensions)[i]); + free(*extensions); + *extensions = NULL; + *nextensions = 0; + } + if (remotes && nremotes) { + for (i = 0; i < (*nremotes); i++) { + struct got_remote_repo *remote; + remote = &(*remotes)[i]; + free(remote->name); + free(remote->fetch_url); + free(remote->send_url); + } + free(*remotes); + *remotes = NULL; + *nremotes = 0; + } + } + return err; +} blob - /dev/null blob + a5046c2bf9464fed10977c382b3808acaa482a80 (mode 644) --- /dev/null +++ lib/read_gotconfig.c @@ -0,0 +1,36 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#include +#include + +#include "got_error.h" +#include "got_repository.h" + +#include "got_gotconfig.h" + +#include "got_lib_gotconfig.h" + +const struct got_error * +got_gotconfig_read(struct got_gotconfig **conf, const char *gotconfig_path) +{ + *conf = calloc(1, sizeof(**conf)); + if (*conf == NULL) + return got_error_from_errno("calloc"); + + /* TODO So far, this only used by gotd, where got.conf is irrelevant. */ + return NULL; +} blob - /dev/null blob + 4e7304f8d4535cb142c07c27aa74f1bac2ba2d40 (mode 644) --- /dev/null +++ lib/reference_parse.c @@ -0,0 +1,81 @@ +/* + * Copyright (c) 2018, 2019 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#include +#include + +#include +#include + +#include "got_reference.h" + +#include "got_lib_lockfile.h" + +#ifndef nitems +#define nitems(_a) (sizeof(_a) / sizeof((_a)[0])) +#endif + +int +got_ref_name_is_valid(const char *name) +{ + const char *s, *seg; + const char forbidden[] = { ' ', '~', '^', ':', '?', '*', '[' , '\\' }; + const char *forbidden_seq[] = { "//", "..", "@{" }; + const char *lfs = GOT_LOCKFILE_SUFFIX; + const size_t lfs_len = sizeof(GOT_LOCKFILE_SUFFIX) - 1; + size_t i; + + if (name[0] == '@' && name[1] == '\0') + return 0; + + s = name; + seg = s; + if (seg[0] == '\0' || seg[0] == '.' || seg[0] == '/') + return 0; + while (*s) { + for (i = 0; i < nitems(forbidden); i++) { + if (*s == forbidden[i]) + return 0; + } + for (i = 0; i < nitems(forbidden_seq); i++) { + if (s[0] == forbidden_seq[i][0] && + s[1] == forbidden_seq[i][1]) + return 0; + } + if (iscntrl((unsigned char)s[0])) + return 0; + if (s[0] == '.' && s[1] == '\0') + return 0; + if (*s == '/') { + const char *nextseg = s + 1; + if (nextseg[0] == '\0' || nextseg[0] == '.' || + nextseg[0] == '/') + return 0; + if (seg <= s - lfs_len && + strncmp(s - lfs_len, lfs, lfs_len) == 0) + return 0; + seg = nextseg; + } + s++; + } + + if (seg <= s - lfs_len && + strncmp(s - lfs_len, lfs, lfs_len) == 0) + return 0; + + return 1; +} + blob - /dev/null blob + 87e9f9ab41e1449492b4a1ca2485a8fba48bfbee (mode 644) --- /dev/null +++ lib/serve.c @@ -0,0 +1,1470 @@ +/* + * Copyright (c) 2022 Stefan Sperling + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "got_error.h" +#include "got_serve.h" +#include "got_path.h" +#include "got_version.h" +#include "got_reference.h" + +#include "got_lib_pkt.h" +#include "got_lib_dial.h" +#include "got_lib_gitproto.h" +#include "got_lib_sha1.h" +#include "got_lib_poll.h" + +#include "gotd.h" + +#ifndef nitems +#define nitems(_a) (sizeof((_a)) / sizeof((_a)[0])) +#endif + +static const struct got_capability read_capabilities[] = { + { GOT_CAPA_AGENT, "got/" GOT_VERSION_STR }, + { GOT_CAPA_OFS_DELTA, NULL }, + { GOT_CAPA_SIDE_BAND_64K, NULL }, +}; + +static const struct got_capability write_capabilities[] = { + { GOT_CAPA_AGENT, "got/" GOT_VERSION_STR }, + { GOT_CAPA_OFS_DELTA, NULL }, + { GOT_CAPA_REPORT_STATUS, NULL }, + { GOT_CAPA_NO_THIN, NULL }, +#if 0 + { GOT_CAPA_DELETE_REFS, NULL }, +#endif +}; + +static const struct got_error * +parse_command(char **command, char **repo_path, const char *gitcmd) +{ + const struct got_error *err = NULL; + size_t len, cmdlen, pathlen; + char *path0 = NULL, *path, *abspath = NULL, *canonpath = NULL; + const char *relpath; + + *command = NULL; + *repo_path = NULL; + + len = strlen(gitcmd); + + if (len >= strlen(GOT_SERVE_CMD_SEND) && + strncmp(gitcmd, GOT_SERVE_CMD_SEND, + strlen(GOT_SERVE_CMD_SEND)) == 0) + cmdlen = strlen(GOT_SERVE_CMD_SEND); + else if (len >= strlen(GOT_SERVE_CMD_FETCH) && + strncmp(gitcmd, GOT_SERVE_CMD_FETCH, + strlen(GOT_SERVE_CMD_FETCH)) == 0) + cmdlen = strlen(GOT_SERVE_CMD_FETCH); + else + return got_error(GOT_ERR_BAD_PACKET); + + if (len <= cmdlen + 1 || gitcmd[cmdlen] != ' ') + return got_error(GOT_ERR_BAD_PACKET); + + if (memchr(&gitcmd[cmdlen + 1], '\0', len - cmdlen) == NULL) + return got_error(GOT_ERR_BAD_PATH); + + /* Forbid linefeeds in paths, like Git does. */ + if (memchr(&gitcmd[cmdlen + 1], '\n', len - cmdlen) != NULL) + return got_error(GOT_ERR_BAD_PATH); + + path0 = strdup(&gitcmd[cmdlen + 1]); + if (path0 == NULL) + return got_error_from_errno("strdup"); + path = path0; + pathlen = strlen(path); + + /* + * Git clients send a shell command. + * Trim spaces and quotes around the path. + */ + while (path[0] == '\'' || path[0] == '\"' || path[0] == ' ') { + path++; + pathlen--; + } + while (pathlen > 0 && + (path[pathlen - 1] == '\'' || path[pathlen - 1] == '\"' || + path[pathlen - 1] == ' ')) { + path[pathlen - 1] = '\0'; + pathlen--; + } + + /* Deny an empty repository path. */ + if (path[0] == '\0' || got_path_is_root_dir(path)) { + err = got_error(GOT_ERR_NOT_GIT_REPO); + goto done; + } + + if (asprintf(&abspath, "/%s", path) == -1) { + err = got_error_from_errno("asprintf"); + goto done; + } + pathlen = strlen(abspath); + canonpath = malloc(pathlen); + if (canonpath == NULL) { + err = got_error_from_errno("malloc"); + goto done; + } + err = got_canonpath(abspath, canonpath, pathlen); + if (err) + goto done; + + relpath = canonpath; + while (relpath[0] == '/') + relpath++; + *repo_path = strdup(relpath); + if (*repo_path == NULL) { + err = got_error_from_errno("strdup"); + goto done; + } + *command = strndup(gitcmd, cmdlen); + if (*command == NULL) + err = got_error_from_errno("strndup"); +done: + free(path0); + free(abspath); + free(canonpath); + if (err) { + free(*repo_path); + *repo_path = NULL; + } + return err; +} + +static const struct got_error * +append_read_capabilities(size_t *capalen, size_t len, const char *symrefstr, + uint8_t *buf, size_t bufsize) +{ + struct got_capability capa[nitems(read_capabilities) + 1]; + size_t ncapa; + + memcpy(&capa, read_capabilities, sizeof(read_capabilities)); + if (symrefstr) { + capa[nitems(read_capabilities)].key = "symref"; + capa[nitems(read_capabilities)].value = symrefstr; + ncapa = nitems(capa); + } else + ncapa = nitems(read_capabilities); + + return got_gitproto_append_capabilities(capalen, buf, len, + bufsize, capa, ncapa); +} + +static const struct got_error * +send_ref(int outfd, uint8_t *id, const char *refname, int send_capabilities, + int client_is_reading, const char *symrefstr, int chattygot) +{ + const struct got_error *err = NULL; + char hex[SHA1_DIGEST_STRING_LENGTH]; + char buf[GOT_PKT_MAX]; + size_t len, capalen = 0; + + if (got_sha1_digest_to_str(id, hex, sizeof(hex)) == NULL) + return got_error(GOT_ERR_BAD_OBJ_ID); + + len = snprintf(buf, sizeof(buf), "%s %s", hex, refname); + if (len >= sizeof(buf)) + return got_error(GOT_ERR_NO_SPACE); + + if (send_capabilities) { + if (client_is_reading) { + err = append_read_capabilities(&capalen, len, + symrefstr, buf, sizeof(buf)); + } else { + err = got_gitproto_append_capabilities(&capalen, + buf, len, sizeof(buf), write_capabilities, + nitems(write_capabilities)); + } + if (err) + return err; + len += capalen; + } + + if (len + 1 >= sizeof(buf)) + return got_error(GOT_ERR_NO_SPACE); + buf[len] = '\n'; + len++; + buf[len] = '\0'; + + return got_pkt_writepkt(outfd, buf, len, chattygot); +} + +static const struct got_error * +send_zero_refs(int outfd, int chattygot) +{ + const struct got_error *err = NULL; + char buf[GOT_PKT_MAX]; + uint8_t zero[SHA1_DIGEST_LENGTH]; + char hex[SHA1_DIGEST_STRING_LENGTH]; + size_t len, capalen = 0; + + memset(&zero, 0, sizeof(zero)); + + if (got_sha1_digest_to_str(zero, hex, sizeof(hex)) == NULL) + return got_error(GOT_ERR_BAD_OBJ_ID); + + len = snprintf(buf, sizeof(buf), "%s capabilities^{}", hex); + if (len >= sizeof(buf)) + return got_error(GOT_ERR_NO_SPACE); + + err = got_gitproto_append_capabilities(&capalen, buf, len, + sizeof(buf), read_capabilities, nitems(read_capabilities)); + if (err) + return err; + + return got_pkt_writepkt(outfd, buf, len, chattygot); +} + +static void +echo_error(const struct got_error *err, int outfd, int chattygot) +{ + char buf[4 + GOT_ERR_MAX_MSG_SIZE]; + size_t len; + + /* + * Echo the error to the client on a pkt-line. + * The client should then terminate its session. + */ + buf[0] = 'E'; buf[1] = 'R'; buf[2] = 'R'; buf[3] = ' '; buf[4] = '\0'; + len = strlcat(buf, err->msg, sizeof(buf)); + err = got_pkt_writepkt(outfd, buf, len, chattygot); + abort(); +} + +static const struct got_error * +announce_refs(int outfd, struct imsgbuf *ibuf, int client_is_reading, + const char *repo_path, int chattygot) +{ + const struct got_error *err = NULL; + struct imsg imsg; + size_t datalen; + struct gotd_imsg_list_refs lsref; + struct gotd_imsg_reflist ireflist; + struct gotd_imsg_ref iref; + struct gotd_imsg_symref isymref; + size_t nrefs = 0; + int have_nrefs = 0, sent_capabilities = 0; + char *symrefname = NULL, *symreftarget = NULL, *symrefstr = NULL; + char *refname = NULL; + + memset(&imsg, 0, sizeof(imsg)); + memset(&lsref, 0, sizeof(lsref)); + + if (strlcpy(lsref.repo_name, repo_path, sizeof(lsref.repo_name)) >= + sizeof(lsref.repo_name)) + return got_error(GOT_ERR_NO_SPACE); + lsref.client_is_reading = client_is_reading; + + if (imsg_compose(ibuf, GOTD_IMSG_LIST_REFS, 0, 0, -1, + &lsref, sizeof(lsref)) == -1) + return got_error_from_errno("imsg_compose LIST_REFS"); + + err = gotd_imsg_flush(ibuf); + if (err) + return err; + + while (!have_nrefs || nrefs > 0) { + err = gotd_imsg_poll_recv(&imsg, ibuf, 0); + if (err) + goto done; + datalen = imsg.hdr.len - IMSG_HEADER_SIZE; + switch (imsg.hdr.type) { + case GOTD_IMSG_ERROR: + err = gotd_imsg_recv_error(NULL, &imsg); + goto done; + case GOTD_IMSG_REFLIST: + if (have_nrefs || nrefs > 0) { + err = got_error(GOT_ERR_PRIVSEP_MSG); + goto done; + } + if (datalen != sizeof(ireflist)) { + err = got_error(GOT_ERR_PRIVSEP_MSG); + goto done; + } + memcpy(&ireflist, imsg.data, sizeof(ireflist)); + nrefs = ireflist.nrefs; + have_nrefs = 1; + if (nrefs == 0) + err = send_zero_refs(outfd, chattygot); + break; + case GOTD_IMSG_REF: + if (!have_nrefs || nrefs == 0) { + err = got_error(GOT_ERR_PRIVSEP_MSG); + goto done; + } + if (datalen < sizeof(iref)) { + err = got_error(GOT_ERR_PRIVSEP_MSG); + goto done; + } + memcpy(&iref, imsg.data, sizeof(iref)); + if (datalen != sizeof(iref) + iref.name_len) { + err = got_error(GOT_ERR_PRIVSEP_LEN); + goto done; + } + refname = malloc(iref.name_len + 1); + if (refname == NULL) { + err = got_error_from_errno("malloc"); + goto done; + } + memcpy(refname, imsg.data + sizeof(iref), + iref.name_len); + refname[iref.name_len] = '\0'; + err = send_ref(outfd, iref.id, refname, + !sent_capabilities, client_is_reading, + NULL, chattygot); + free(refname); + refname = NULL; + if (err) + goto done; + sent_capabilities = 1; + if (nrefs > 0) + nrefs--; + break; + case GOTD_IMSG_SYMREF: + if (!have_nrefs || nrefs == 0) { + err = got_error(GOT_ERR_PRIVSEP_MSG); + goto done; + } + if (datalen < sizeof(isymref)) { + err = got_error(GOT_ERR_PRIVSEP_LEN); + goto done; + } + memcpy(&isymref, imsg.data, sizeof(isymref)); + if (datalen != sizeof(isymref) + isymref.name_len + + isymref.target_len) { + err = got_error(GOT_ERR_PRIVSEP_LEN); + goto done; + } + + /* + * For now, we only announce one symbolic ref, + * as part of our capability advertisement. + */ + if (sent_capabilities || symrefstr != NULL || + symrefname != NULL || symreftarget != NULL) + break; + + symrefname = malloc(isymref.name_len + 1); + if (symrefname == NULL) { + err = got_error_from_errno("malloc"); + goto done; + } + memcpy(symrefname, imsg.data + sizeof(isymref), + isymref.name_len); + symrefname[isymref.name_len] = '\0'; + + symreftarget = malloc(isymref.target_len + 1); + if (symreftarget == NULL) { + err = got_error_from_errno("malloc"); + goto done; + } + memcpy(symreftarget, + imsg.data + sizeof(isymref) + isymref.name_len, + isymref.target_len); + symreftarget[isymref.target_len] = '\0'; + + if (asprintf(&symrefstr, "%s:%s", symrefname, + symreftarget) == -1) { + err = got_error_from_errno("asprintf"); + goto done; + } + err = send_ref(outfd, isymref.target_id, symrefname, + !sent_capabilities, client_is_reading, symrefstr, + chattygot); + free(refname); + refname = NULL; + if (err) + goto done; + sent_capabilities = 1; + if (nrefs > 0) + nrefs--; + break; + default: + err = got_error(GOT_ERR_PRIVSEP_MSG); + break; + } + + imsg_free(&imsg); + } + + err = got_pkt_flushpkt(outfd, chattygot); + if (err) + goto done; +done: + free(symrefstr); + free(symrefname); + free(symreftarget); + return err; +} + +static const struct got_error * +parse_want_line(char **common_capabilities, uint8_t *id, char *buf, size_t len) +{ + const struct got_error *err; + char *id_str = NULL, *client_capabilities = NULL; + + err = got_gitproto_parse_want_line(&id_str, + &client_capabilities, buf, len); + if (err) + return err; + + if (!got_parse_sha1_digest(id, id_str)) { + err = got_error_msg(GOT_ERR_BAD_PACKET, + "want-line with bad object ID"); + goto done; + } + + if (client_capabilities) { + err = got_gitproto_match_capabilities(common_capabilities, + NULL, client_capabilities, read_capabilities, + nitems(read_capabilities)); + if (err) + goto done; + } +done: + free(id_str); + free(client_capabilities); + return err; +} + +static const struct got_error * +parse_have_line(uint8_t *id, char *buf, size_t len) +{ + const struct got_error *err; + char *id_str = NULL; + + err = got_gitproto_parse_have_line(&id_str, buf, len); + if (err) + return err; + + if (!got_parse_sha1_digest(id, id_str)) { + err = got_error_msg(GOT_ERR_BAD_PACKET, + "have-line with bad object ID"); + goto done; + } +done: + free(id_str); + return err; +} + +static const struct got_error * +send_capability(struct got_capability *capa, struct imsgbuf *ibuf) +{ + const struct got_error *err = NULL; + struct gotd_imsg_capability icapa; + size_t len; + struct ibuf *wbuf; + + memset(&icapa, 0, sizeof(icapa)); + + icapa.key_len = strlen(capa->key); + len = sizeof(icapa) + icapa.key_len; + if (capa->value) { + icapa.value_len = strlen(capa->value); + len += icapa.value_len; + } + + wbuf = imsg_create(ibuf, GOTD_IMSG_CAPABILITY, 0, 0, len); + if (wbuf == NULL) { + err = got_error_from_errno("imsg_create CAPABILITY"); + return err; + } + + if (imsg_add(wbuf, &icapa, sizeof(icapa)) == -1) + return got_error_from_errno("imsg_add CAPABILITY"); + if (imsg_add(wbuf, capa->key, icapa.key_len) == -1) + return got_error_from_errno("imsg_add CAPABILITY"); + if (capa->value) { + if (imsg_add(wbuf, capa->value, icapa.value_len) == -1) + return got_error_from_errno("imsg_add CAPABILITY"); + } + + wbuf->fd = -1; + imsg_close(ibuf, wbuf); + + return NULL; +} + +static const struct got_error * +send_capabilities(int *use_sidebands, int *report_status, + char *capabilities_str, struct imsgbuf *ibuf) +{ + const struct got_error *err = NULL; + struct gotd_imsg_capabilities icapas; + struct got_capability *capa = NULL; + size_t ncapa, i; + + err = got_gitproto_split_capabilities_str(&capa, &ncapa, + capabilities_str); + if (err) + return err; + + icapas.ncapabilities = ncapa; + if (imsg_compose(ibuf, GOTD_IMSG_CAPABILITIES, 0, 0, -1, + &icapas, sizeof(icapas)) == -1) { + err = got_error_from_errno("imsg_compose IMSG_CAPABILITIES"); + goto done; + } + + for (i = 0; i < ncapa; i++) { + err = send_capability(&capa[i], ibuf); + if (err) + goto done; + if (use_sidebands && + strcmp(capa[i].key, GOT_CAPA_SIDE_BAND_64K) == 0) + *use_sidebands = 1; + if (report_status && + strcmp(capa[i].key, GOT_CAPA_REPORT_STATUS) == 0) + *report_status = 1; + } +done: + free(capa); + return err; +} + +static const struct got_error * +forward_flushpkt(struct imsgbuf *ibuf) +{ + if (imsg_compose(ibuf, GOTD_IMSG_FLUSH, 0, 0, -1, NULL, 0) == -1) + return got_error_from_errno("imsg_compose FLUSH"); + + return gotd_imsg_flush(ibuf); +} + +static const struct got_error * +recv_ack(struct imsg *imsg, uint8_t *expected_id) +{ + struct gotd_imsg_ack iack; + size_t datalen; + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(iack)) + return got_error(GOT_ERR_PRIVSEP_LEN); + + memcpy(&iack, imsg->data, sizeof(iack)); + if (memcmp(iack.object_id, expected_id, SHA1_DIGEST_LENGTH) != 0) + return got_error(GOT_ERR_BAD_OBJ_ID); + + return NULL; +} + +static const struct got_error * +recv_nak(struct imsg *imsg, uint8_t *expected_id) +{ + struct gotd_imsg_ack inak; + size_t datalen; + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != sizeof(inak)) + return got_error(GOT_ERR_PRIVSEP_LEN); + + memcpy(&inak, imsg->data, sizeof(inak)); + if (memcmp(inak.object_id, expected_id, SHA1_DIGEST_LENGTH) != 0) + return got_error(GOT_ERR_BAD_OBJ_ID); + + return NULL; +} + + +static const struct got_error * +recv_want(int *use_sidebands, int outfd, struct imsgbuf *ibuf, + char *buf, size_t len, int expect_capabilities, int chattygot) +{ + const struct got_error *err; + struct gotd_imsg_want iwant; + char *capabilities_str; + int done = 0; + struct imsg imsg; + + memset(&iwant, 0, sizeof(iwant)); + memset(&imsg, 0, sizeof(imsg)); + + err = parse_want_line(&capabilities_str, iwant.object_id, buf, len); + if (err) + return err; + + if (capabilities_str) { + if (!expect_capabilities) { + err = got_error_msg(GOT_ERR_BAD_PACKET, + "unexpected capability announcement received"); + goto done; + } + err = send_capabilities(use_sidebands, NULL, capabilities_str, + ibuf); + if (err) + goto done; + + } + + if (imsg_compose(ibuf, GOTD_IMSG_WANT, 0, 0, -1, + &iwant, sizeof(iwant)) == -1) { + err = got_error_from_errno("imsg_compose WANT"); + goto done; + } + + err = gotd_imsg_flush(ibuf); + if (err) + goto done; + + /* + * Wait for an ACK, or an error in case the desired object + * does not exist. + */ + while (!done && err == NULL) { + err = gotd_imsg_poll_recv(&imsg, ibuf, 0); + if (err) + break; + switch (imsg.hdr.type) { + case GOTD_IMSG_ERROR: + err = gotd_imsg_recv_error(NULL, &imsg); + break; + case GOTD_IMSG_ACK: + err = recv_ack(&imsg, iwant.object_id); + if (err) + break; + done = 1; + break; + default: + err = got_error(GOT_ERR_PRIVSEP_MSG); + break; + } + + imsg_free(&imsg); + } +done: + free(capabilities_str); + return err; +} + +static const struct got_error * +send_ack(int outfd, uint8_t *id, int chattygot) +{ + char hex[SHA1_DIGEST_STRING_LENGTH]; + char buf[GOT_PKT_MAX]; + int len; + + if (got_sha1_digest_to_str(id, hex, sizeof(hex)) == NULL) + return got_error(GOT_ERR_BAD_OBJ_ID); + + len = snprintf(buf, sizeof(buf), "ACK %s\n", hex); + if (len >= sizeof(buf)) + return got_error(GOT_ERR_NO_SPACE); + + return got_pkt_writepkt(outfd, buf, len, chattygot); +} + +static const struct got_error * +send_nak(int outfd, int chattygot) +{ + char buf[5]; + int len; + + len = snprintf(buf, sizeof(buf), "NAK\n"); + if (len >= sizeof(buf)) + return got_error(GOT_ERR_NO_SPACE); + + return got_pkt_writepkt(outfd, buf, len, chattygot); +} + +static const struct got_error * +recv_have(int *have_ack, int outfd, struct imsgbuf *ibuf, char *buf, + size_t len, int chattygot) +{ + const struct got_error *err; + struct gotd_imsg_have ihave; + int done = 0; + struct imsg imsg; + + memset(&ihave, 0, sizeof(ihave)); + memset(&imsg, 0, sizeof(imsg)); + + err = parse_have_line(ihave.object_id, buf, len); + if (err) + return err; + + if (imsg_compose(ibuf, GOTD_IMSG_HAVE, 0, 0, -1, + &ihave, sizeof(ihave)) == -1) + return got_error_from_errno("imsg_compose HAVE"); + + err = gotd_imsg_flush(ibuf); + if (err) + return err; + + /* + * Wait for an ACK or a NAK, indicating whether a common + * commit object has been found. + */ + while (!done && err == NULL) { + err = gotd_imsg_poll_recv(&imsg, ibuf, 0); + if (err) + return err; + switch (imsg.hdr.type) { + case GOTD_IMSG_ERROR: + err = gotd_imsg_recv_error(NULL, &imsg); + break; + case GOTD_IMSG_ACK: + err = recv_ack(&imsg, ihave.object_id); + if (err) + break; + if (!*have_ack) { + err = send_ack(outfd, ihave.object_id, + chattygot); + if (err) + return err; + *have_ack = 1; + } + done = 1; + break; + case GOTD_IMSG_NAK: + err = recv_nak(&imsg, ihave.object_id); + if (err) + break; + done = 1; + break; + default: + err = got_error(GOT_ERR_PRIVSEP_MSG); + break; + } + + imsg_free(&imsg); + } + + return err; +} + +static const struct got_error * +recv_done(int *packfd, int outfd, struct imsgbuf *ibuf, int chattygot) +{ + const struct got_error *err; + struct imsg imsg; + + *packfd = -1; + + if (imsg_compose(ibuf, GOTD_IMSG_DONE, 0, 0, -1, NULL, 0) == -1) + return got_error_from_errno("imsg_compose DONE"); + + err = gotd_imsg_flush(ibuf); + if (err) + return err; + + err = gotd_imsg_poll_recv(&imsg, ibuf, 0); + if (err) + return err; + + if (imsg.hdr.type == GOTD_IMSG_ERROR) + err = gotd_imsg_recv_error(NULL, &imsg); + else if (imsg.hdr.type != GOTD_IMSG_PACKFILE_PIPE) + err = got_error(GOT_ERR_PRIVSEP_MSG); + else if (imsg.fd == -1) + err = got_error(GOT_ERR_PRIVSEP_NO_FD); + if (err == NULL) + *packfd = imsg.fd; + + imsg_free(&imsg); + return err; +} + +static const struct got_error * +relay_progress_reports(struct imsgbuf *ibuf, int outfd, int chattygot) +{ + const struct got_error *err = NULL; + int pack_starting = 0; + struct gotd_imsg_packfile_progress iprog; + char buf[GOT_PKT_MAX]; + struct imsg imsg; + size_t datalen; + int p_deltify = 0, n; + const char *eol = "\r"; + + memset(&imsg, 0, sizeof(imsg)); + + while (!pack_starting && err == NULL) { + err = gotd_imsg_poll_recv(&imsg, ibuf, 0); + if (err) + break; + + datalen = imsg.hdr.len - IMSG_HEADER_SIZE; + switch (imsg.hdr.type) { + case GOTD_IMSG_ERROR: + err = gotd_imsg_recv_error(NULL, &imsg); + break; + case GOTD_IMSG_PACKFILE_READY: + eol = "\n"; + pack_starting = 1; + /* fallthrough */ + case GOTD_IMSG_PACKFILE_PROGRESS: + if (datalen != sizeof(iprog)) { + err = got_error(GOT_ERR_PRIVSEP_LEN); + break; + } + memcpy(&iprog, imsg.data, sizeof(iprog)); + if (iprog.nobj_total > 0) { + p_deltify = (iprog.nobj_deltify * 100) / + iprog.nobj_total; + } + buf[0] = GOT_SIDEBAND_PROGRESS_INFO; + n = snprintf(&buf[1], sizeof(buf) - 1, + "%d commits colored, " + "%d objects found, " + "deltify %d%%%s", + iprog.ncolored, + iprog.nfound, + p_deltify, eol); + if (n >= sizeof(buf) - 1) + break; + err = got_pkt_writepkt(outfd, buf, 1 + n, chattygot); + break; + default: + err = got_error(GOT_ERR_PRIVSEP_MSG); + break; + } + + imsg_free(&imsg); + } + + return err; +} + +static const struct got_error * +serve_read(int infd, int outfd, int gotd_sock, const char *repo_path, + int chattygot) +{ + const struct got_error *err = NULL; + char buf[GOT_PKT_MAX]; + struct imsgbuf ibuf; + enum protostate { + STATE_EXPECT_WANT, + STATE_EXPECT_MORE_WANT, + STATE_EXPECT_HAVE, + STATE_EXPECT_DONE, + STATE_DONE, + }; + enum protostate curstate = STATE_EXPECT_WANT; + int have_ack = 0, use_sidebands = 0, seen_have = 0; + int packfd = -1; + size_t pack_chunksize; + + imsg_init(&ibuf, gotd_sock); + + err = announce_refs(outfd, &ibuf, 1, repo_path, chattygot); + if (err) + goto done; + + while (curstate != STATE_DONE) { + int n; + buf[0] = '\0'; + err = got_pkt_readpkt(&n, infd, buf, sizeof(buf), chattygot); + if (err) + break; + if (n == 0) { + if (curstate != STATE_EXPECT_MORE_WANT && + curstate != STATE_EXPECT_HAVE) { + err = got_error_msg(GOT_ERR_BAD_PACKET, + "unexpected flush packet received"); + goto done; + } + err = forward_flushpkt(&ibuf); + if (err) + goto done; + if (curstate == STATE_EXPECT_HAVE && !have_ack) { + err = send_nak(outfd, chattygot); + if (err) + goto done; + } + if (curstate == STATE_EXPECT_MORE_WANT) + curstate = STATE_EXPECT_HAVE; + else + curstate = STATE_EXPECT_DONE; + } else if (n >= 5 && strncmp(buf, "want ", 5) == 0) { + if (curstate != STATE_EXPECT_WANT && + curstate != STATE_EXPECT_MORE_WANT) { + err = got_error_msg(GOT_ERR_BAD_PACKET, + "unexpected 'want' packet"); + goto done; + } + err = recv_want(&use_sidebands, outfd, &ibuf, buf, n, + curstate == STATE_EXPECT_WANT ? 1 : 0, chattygot); + if (err) + goto done; + if (curstate == STATE_EXPECT_WANT) + curstate = STATE_EXPECT_MORE_WANT; + } else if (n >= 5 && strncmp(buf, "have ", 5) == 0) { + if (curstate != STATE_EXPECT_HAVE) { + err = got_error_msg(GOT_ERR_BAD_PACKET, + "unexpected 'have' packet"); + goto done; + } + err = recv_have(&have_ack, outfd, &ibuf, buf, n, + chattygot); + if (err) + goto done; + seen_have = 1; + } else if (n == 5 && strncmp(buf, "done\n", 5) == 0) { + if (curstate != STATE_EXPECT_HAVE && + curstate != STATE_EXPECT_DONE) { + err = got_error_msg(GOT_ERR_BAD_PACKET, + "unexpected 'done' packet"); + goto done; + } + err = recv_done(&packfd, outfd, &ibuf, chattygot); + if (err) + goto done; + curstate = STATE_DONE; + break; + } else { + err = got_error(GOT_ERR_BAD_PACKET); + goto done; + } + } + + if (!seen_have) { + err = send_nak(outfd, chattygot); + if (err) + goto done; + } + + if (use_sidebands) { + err = relay_progress_reports(&ibuf, outfd, chattygot); + if (err) + goto done; + pack_chunksize = GOT_SIDEBAND_64K_PACKFILE_DATA_MAX; + } else + pack_chunksize = sizeof(buf); + + for (;;) { + ssize_t r, w; + + r = read(packfd, use_sidebands ? &buf[1] : buf, + pack_chunksize); + if (r == -1) { + err = got_error_from_errno("read"); + break; + } else if (r == 0) { + err = got_pkt_flushpkt(outfd, chattygot); + break; + } + + if (use_sidebands) { + buf[0] = GOT_SIDEBAND_PACKFILE_DATA; + err = got_pkt_writepkt(outfd, buf, 1 + r, chattygot); + if (err) + break; + } else { + w = write(outfd, buf, r); + if (w == -1) { + err = got_error_from_errno("write"); + break; + } else if (w != r) { + err = got_error(GOT_ERR_IO); + break; + } + } + } +done: + imsg_clear(&ibuf); + if (packfd != -1 && close(packfd) == -1 && err == NULL) + err = got_error_from_errno("close"); + if (err) + echo_error(err, outfd, chattygot); + return err; +} + +static const struct got_error * +parse_ref_update_line(char **common_capabilities, char **refname, + uint8_t *old_id, uint8_t *new_id, char *buf, size_t len) +{ + const struct got_error *err; + char *old_id_str = NULL, *new_id_str = NULL; + char *client_capabilities = NULL; + + *refname = NULL; + + err = got_gitproto_parse_ref_update_line(&old_id_str, &new_id_str, + refname, &client_capabilities, buf, len); + if (err) + return err; + + if (!got_parse_sha1_digest(old_id, old_id_str) || + !got_parse_sha1_digest(new_id, new_id_str)) { + err = got_error_msg(GOT_ERR_BAD_PACKET, + "ref-update with bad object ID"); + goto done; + } + if (!got_ref_name_is_valid(*refname)) { + err = got_error_msg(GOT_ERR_BAD_PACKET, + "ref-update with bad reference name"); + goto done; + } + + if (client_capabilities) { + err = got_gitproto_match_capabilities(common_capabilities, + NULL, client_capabilities, write_capabilities, + nitems(write_capabilities)); + if (err) + goto done; + } +done: + free(old_id_str); + free(new_id_str); + free(client_capabilities); + if (err) { + free(*refname); + *refname = NULL; + } + return err; +} + +static const struct got_error * +recv_ref_update(int *report_status, int outfd, struct imsgbuf *ibuf, + char *buf, size_t len, int expect_capabilities, int chattygot) +{ + const struct got_error *err; + struct gotd_imsg_ref_update iref; + struct ibuf *wbuf; + char *capabilities_str = NULL, *refname = NULL; + int done = 0; + struct imsg imsg; + + memset(&iref, 0, sizeof(iref)); + memset(&imsg, 0, sizeof(imsg)); + + err = parse_ref_update_line(&capabilities_str, &refname, + iref.old_id, iref.new_id, buf, len); + if (err) + return err; + + if (capabilities_str) { + if (!expect_capabilities) { + err = got_error_msg(GOT_ERR_BAD_PACKET, + "unexpected capability announcement received"); + goto done; + } + err = send_capabilities(NULL, report_status, capabilities_str, + ibuf); + if (err) + goto done; + } + + iref.name_len = strlen(refname); + len = sizeof(iref) + iref.name_len; + wbuf = imsg_create(ibuf, GOTD_IMSG_REF_UPDATE, 0, 0, len); + if (wbuf == NULL) { + err = got_error_from_errno("imsg_create REF_UPDATE"); + goto done; + } + + if (imsg_add(wbuf, &iref, sizeof(iref)) == -1) + return got_error_from_errno("imsg_add REF_UPDATE"); + if (imsg_add(wbuf, refname, iref.name_len) == -1) + return got_error_from_errno("imsg_add REF_UPDATE"); + wbuf->fd = -1; + imsg_close(ibuf, wbuf); + + err = gotd_imsg_flush(ibuf); + if (err) + goto done; + + /* Wait for ACK or an error. */ + while (!done && err == NULL) { + err = gotd_imsg_poll_recv(&imsg, ibuf, 0); + if (err) + break; + switch (imsg.hdr.type) { + case GOTD_IMSG_ERROR: + err = gotd_imsg_recv_error(NULL, &imsg); + break; + case GOTD_IMSG_ACK: + err = recv_ack(&imsg, iref.new_id); + if (err) + break; + done = 1; + break; + default: + err = got_error(GOT_ERR_PRIVSEP_MSG); + break; + } + + imsg_free(&imsg); + } +done: + free(capabilities_str); + free(refname); + return err; +} + +static const struct got_error * +recv_packfile(struct imsg *imsg, int infd) +{ + const struct got_error *err = NULL; + size_t datalen; + int packfd; + char buf[GOT_PKT_MAX]; + int pack_done = 0; + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen != 0) + return got_error(GOT_ERR_PRIVSEP_MSG); + + if (imsg->fd == -1) + return got_error(GOT_ERR_PRIVSEP_NO_FD); + + packfd = imsg->fd; + while (!pack_done) { + ssize_t r = 0; + + err = got_poll_fd(infd, POLLIN, 1); + if (err) { + if (err->code != GOT_ERR_TIMEOUT) + break; + err = NULL; + } else { + r = read(infd, buf, sizeof(buf)); + if (r == -1) { + err = got_error_from_errno("read"); + break; + } + if (r == 0) { + /* + * Git clients hang up their side of the + * connection after sending the pack file. + */ + err = NULL; + pack_done = 1; + break; + } + } + + if (r == 0) { + /* Detect gotd(8) closing the pack pipe when done. */ + err = got_poll_fd(packfd, POLLOUT, 1); + if (err) { + if (err->code != GOT_ERR_EOF) + break; + err = NULL; + pack_done = 1; + } + } else { + /* Write pack data and/or detect pipe being closed. */ + err = got_poll_write_full(packfd, buf, r); + if (err) { + if (err->code == GOT_ERR_EOF) + err = NULL; + break; + } + } + } + + close(packfd); + return err; +} + +static const struct got_error * +report_unpack_status(struct imsg *imsg, int outfd, int chattygot) +{ + const struct got_error *err = NULL; + struct gotd_imsg_packfile_status istatus; + char buf[GOT_PKT_MAX]; + size_t datalen, len; + char *reason = NULL; + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen < sizeof(istatus)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&istatus, imsg->data, sizeof(istatus)); + if (datalen != sizeof(istatus) + istatus.reason_len) + return got_error(GOT_ERR_PRIVSEP_LEN); + + reason = malloc(istatus.reason_len + 1); + if (reason == NULL) { + err = got_error_from_errno("malloc"); + goto done; + } + memcpy(reason, imsg->data + sizeof(istatus), istatus.reason_len); + reason[istatus.reason_len] = '\0'; + + if (err == NULL) + len = snprintf(buf, sizeof(buf), "unpack ok\n"); + else + len = snprintf(buf, sizeof(buf), "unpack %s\n", reason); + if (len >= sizeof(buf)) { + err = got_error(GOT_ERR_NO_SPACE); + goto done; + } + + err = got_pkt_writepkt(outfd, buf, len, chattygot); +done: + free(reason); + return err; +} + +static const struct got_error * +recv_ref_update_ok(struct imsg *imsg, int outfd, int chattygot) +{ + const struct got_error *err = NULL; + struct gotd_imsg_ref_update_ok iok; + size_t datalen, len; + char buf[GOT_PKT_MAX]; + char *refname = NULL; + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen < sizeof(iok)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&iok, imsg->data, sizeof(iok)); + if (datalen != sizeof(iok) + iok.name_len) + return got_error(GOT_ERR_PRIVSEP_LEN); + + memcpy(&iok, imsg->data, sizeof(iok)); + + refname = malloc(iok.name_len + 1); + if (refname == NULL) + return got_error_from_errno("malloc"); + memcpy(refname, imsg->data + sizeof(iok), iok.name_len); + refname[iok.name_len] = '\0'; + + len = snprintf(buf, sizeof(buf), "ok %s\n", refname); + if (len >= sizeof(buf)) { + err = got_error(GOT_ERR_NO_SPACE); + goto done; + } + + err = got_pkt_writepkt(outfd, buf, len, chattygot); +done: + free(refname); + return err; +} + +static const struct got_error * +recv_ref_update_ng(struct imsg *imsg, int outfd, int chattygot) +{ + const struct got_error *err = NULL; + struct gotd_imsg_ref_update_ng ing; + size_t datalen, len; + char buf[GOT_PKT_MAX]; + char *refname = NULL, *reason = NULL; + + datalen = imsg->hdr.len - IMSG_HEADER_SIZE; + if (datalen < sizeof(ing)) + return got_error(GOT_ERR_PRIVSEP_LEN); + memcpy(&ing, imsg->data, sizeof(ing)); + if (datalen != sizeof(ing) + ing.name_len + ing.reason_len) + return got_error(GOT_ERR_PRIVSEP_LEN); + + memcpy(&ing, imsg->data, sizeof(ing)); + + refname = malloc(ing.name_len + 1); + if (refname == NULL) + return got_error_from_errno("malloc"); + memcpy(refname, imsg->data + sizeof(ing), ing.name_len); + refname[ing.name_len] = '\0'; + + reason = malloc(ing.reason_len + 1); + if (reason == NULL) { + err = got_error_from_errno("malloc"); + goto done; + } + memcpy(refname, imsg->data + sizeof(ing) + ing.name_len, + ing.reason_len); + refname[ing.reason_len] = '\0'; + + len = snprintf(buf, sizeof(buf), "ng %s %s\n", refname, reason); + if (len >= sizeof(buf)) { + err = got_error(GOT_ERR_NO_SPACE); + goto done; + } + + err = got_pkt_writepkt(outfd, buf, len, chattygot); +done: + free(refname); + free(reason); + return err; +} + +static const struct got_error * +serve_write(int infd, int outfd, int gotd_sock, const char *repo_path, + int chattygot) +{ + const struct got_error *err = NULL; + char buf[GOT_PKT_MAX]; + struct imsgbuf ibuf; + enum protostate { + STATE_EXPECT_REF_UPDATE, + STATE_EXPECT_MORE_REF_UPDATES, + STATE_EXPECT_PACKFILE, + STATE_PACKFILE_RECEIVED, + STATE_REFS_UPDATED, + }; + enum protostate curstate = STATE_EXPECT_REF_UPDATE; + struct imsg imsg; + int report_status = 0; + + imsg_init(&ibuf, gotd_sock); + memset(&imsg, 0, sizeof(imsg)); + + err = announce_refs(outfd, &ibuf, 0, repo_path, chattygot); + if (err) + goto done; + + while (curstate != STATE_EXPECT_PACKFILE) { + int n; + buf[0] = '\0'; + err = got_pkt_readpkt(&n, infd, buf, sizeof(buf), chattygot); + if (err) + break; + if (n == 0) { + if (curstate != STATE_EXPECT_MORE_REF_UPDATES) { + err = got_error_msg(GOT_ERR_BAD_PACKET, + "unexpected flush packet received"); + goto done; + } + err = forward_flushpkt(&ibuf); + if (err) + goto done; + curstate = STATE_EXPECT_PACKFILE; + } else if (n >= (SHA1_DIGEST_STRING_LENGTH * 2) + 2) { + if (curstate != STATE_EXPECT_REF_UPDATE && + curstate != STATE_EXPECT_MORE_REF_UPDATES) { + err = got_error_msg(GOT_ERR_BAD_PACKET, + "unexpected ref-update packet"); + goto done; + } + if (curstate == STATE_EXPECT_REF_UPDATE) { + err = recv_ref_update(&report_status, + outfd, &ibuf, buf, n, 1, chattygot); + } else { + err = recv_ref_update(NULL, outfd, &ibuf, + buf, n, 0, chattygot); + } + if (err) + goto done; + curstate = STATE_EXPECT_MORE_REF_UPDATES; + } else { + err = got_error(GOT_ERR_BAD_PACKET); + goto done; + } + } + + while (curstate != STATE_PACKFILE_RECEIVED) { + err = gotd_imsg_poll_recv(&imsg, &ibuf, 0); + if (err) + goto done; + switch (imsg.hdr.type) { + case GOTD_IMSG_ERROR: + err = gotd_imsg_recv_error(NULL, &imsg); + goto done; + case GOTD_IMSG_RECV_PACKFILE: + err = recv_packfile(&imsg, infd); + if (err) { + if (err->code != GOT_ERR_EOF) + goto done; + /* + * EOF is reported when the client hangs up, + * which can happen with Git clients. + * The socket should stay half-open so we + * can still send our reports if requested. + */ + err = NULL; + } + curstate = STATE_PACKFILE_RECEIVED; + break; + default: + err = got_error(GOT_ERR_PRIVSEP_MSG); + break; + } + + imsg_free(&imsg); + if (err) + goto done; + } + + while (curstate != STATE_REFS_UPDATED && err == NULL) { + err = gotd_imsg_poll_recv(&imsg, &ibuf, 0); + if (err) + break; + switch (imsg.hdr.type) { + case GOTD_IMSG_ERROR: + err = gotd_imsg_recv_error(NULL, &imsg); + break; + case GOTD_IMSG_PACKFILE_STATUS: + if (!report_status) + break; + err = report_unpack_status(&imsg, outfd, chattygot); + break; + case GOTD_IMSG_REF_UPDATE_OK: + if (!report_status) + break; + err = recv_ref_update_ok(&imsg, outfd, chattygot); + break; + case GOTD_IMSG_REF_UPDATE_NG: + if (!report_status) + break; + err = recv_ref_update_ng(&imsg, outfd, chattygot); + break; + case GOTD_IMSG_REFS_UPDATED: + curstate = STATE_REFS_UPDATED; + err = got_pkt_flushpkt(outfd, chattygot); + break; + default: + err = got_error(GOT_ERR_PRIVSEP_MSG); + break; + } + + imsg_free(&imsg); + } +done: + imsg_clear(&ibuf); + if (err) + echo_error(err, outfd, chattygot); + return err; +} + +const struct got_error * +got_serve(int infd, int outfd, const char *gitcmd, int gotd_sock, int chattygot) +{ + const struct got_error *err = NULL; + char *command = NULL, *repo_path = NULL; + + err = parse_command(&command, &repo_path, gitcmd); + if (err) + return err; + + if (strcmp(command, GOT_SERVE_CMD_FETCH) == 0) + err = serve_read(infd, outfd, gotd_sock, repo_path, chattygot); + else if (strcmp(command, GOT_SERVE_CMD_SEND) == 0) + err = serve_write(infd, outfd, gotd_sock, repo_path, chattygot); + else + err = got_error(GOT_ERR_BAD_PACKET); + + free(command); + free(repo_path); + return err; +} blob - 1cbeffd382561f9eaa87c657fa55701cfc342355 blob + 7158a0e8c0ec73671b1e74d1affd3176244fbe9e --- libexec/got-fetch-pack/Makefile +++ libexec/got-fetch-pack/Makefile @@ -4,7 +4,8 @@ PROG= got-fetch-pack SRCS= got-fetch-pack.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c pkt.c gitproto.c ratelimit.c + path.c privsep.c sha1.c pkt.c gitproto.c ratelimit.c \ + pollfd.c CPPFLAGS = -I${.CURDIR}/../../include -I${.CURDIR}/../../lib blob - af979cbb0b09e185635cf575cb13c269462c358d blob + c2f4952a5ecb70a6ecdb061eef225def26c15c75 --- libexec/got-index-pack/Makefile +++ libexec/got-index-pack/Makefile @@ -5,7 +5,7 @@ PROG= got-index-pack SRCS= got-index-pack.c error.c inflate.c object_parse.c object_idset.c \ delta_cache.c delta.c pack.c path.c privsep.c sha1.c ratelimit.c \ - pack_index.c + pack_index.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../include -I${.CURDIR}/../../lib blob - 427fba8a8a0724eeb90a9853fb783d15cd60f8ae blob + 10fa486bce699c88c8487b34007e1ab262a1d1a8 --- libexec/got-read-blob/Makefile +++ libexec/got-read-blob/Makefile @@ -4,7 +4,7 @@ PROG= got-read-blob SRCS= got-read-blob.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c + path.c privsep.c sha1.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../include -I${.CURDIR}/../../lib blob - a5aaaaee3ffbd172032dc5ba3edf5ed47b257479 blob + dec93a8e7676ff3c05faf46924a6572bf79f53ea --- libexec/got-read-commit/Makefile +++ libexec/got-read-commit/Makefile @@ -4,7 +4,7 @@ PROG= got-read-commit SRCS= got-read-commit.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c + path.c privsep.c sha1.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../include -I${.CURDIR}/../../lib blob - 867dc5954e47c0673e815db7c0e25ae703e11658 blob + 14d25e4015998c43610d92c975fba1da5a3c6818 --- libexec/got-read-commit/got-read-commit.c +++ libexec/got-read-commit/got-read-commit.c @@ -48,60 +48,6 @@ catch_sigint(int signo) sigint_received = 1; } -static const struct got_error * -read_commit_object(struct got_commit_object **commit, FILE *f, - struct got_object_id *expected_id) -{ - struct got_object *obj = NULL; - const struct got_error *err = NULL; - size_t len; - uint8_t *p; - struct got_inflate_checksum csum; - SHA1_CTX sha1_ctx; - struct got_object_id id; - - SHA1Init(&sha1_ctx); - memset(&csum, 0, sizeof(csum)); - csum.output_sha1 = &sha1_ctx; - - err = got_inflate_to_mem(&p, &len, NULL, &csum, f); - if (err) - return err; - - SHA1Final(id.sha1, &sha1_ctx); - if (memcmp(expected_id->sha1, id.sha1, SHA1_DIGEST_LENGTH) != 0) { - char buf[SHA1_DIGEST_STRING_LENGTH]; - err = got_error_fmt(GOT_ERR_OBJ_CSUM, - "checksum failure for object %s", - got_sha1_digest_to_str(expected_id->sha1, buf, - sizeof(buf))); - goto done; - } - - err = got_object_parse_header(&obj, p, len); - if (err) - goto done; - - if (len < obj->hdrlen + obj->size) { - err = got_error(GOT_ERR_BAD_OBJ_DATA); - goto done; - } - - if (obj->type != GOT_OBJ_TYPE_COMMIT) { - err = got_error(GOT_ERR_OBJ_TYPE); - goto done; - } - - /* Skip object header. */ - len -= obj->hdrlen; - err = got_object_parse_commit(commit, p + obj->hdrlen, len); -done: - free(p); - if (obj) - got_object_close(obj); - return err; -} - int main(int argc, char *argv[]) { @@ -124,7 +70,6 @@ main(int argc, char *argv[]) for (;;) { struct imsg imsg; - FILE *f = NULL; struct got_commit_object *commit = NULL; struct got_object_id expected_id; @@ -160,27 +105,15 @@ main(int argc, char *argv[]) goto done; } - /* Always assume file offset zero. */ - f = fdopen(imsg.fd, "rb"); - if (f == NULL) { - err = got_error_from_errno("fdopen"); - goto done; - } - - err = read_commit_object(&commit, f, &expected_id); + err = got_object_read_commit(&commit, imsg.fd, &expected_id, 0); if (err) goto done; err = got_privsep_send_commit(&ibuf, commit); got_object_commit_close(commit); done: - if (f) { - if (fclose(f) == EOF && err == NULL) - err = got_error_from_errno("fclose"); - } else if (imsg.fd != -1) { - if (close(imsg.fd) == -1 && err == NULL) - err = got_error_from_errno("close"); - } + if (imsg.fd != -1 && close(imsg.fd) == -1 && err == NULL) + err = got_error_from_errno("close"); imsg_free(&imsg); if (err) break; blob - 326e9b965f1353cf93baf20aac196f76c0a02ffb blob + 25a30ce385a81305898d16ab8b87121528f7e630 --- libexec/got-read-gitconfig/Makefile +++ libexec/got-read-gitconfig/Makefile @@ -4,7 +4,7 @@ PROG= got-read-gitconfig SRCS= got-read-gitconfig.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c gitconfig.c + path.c privsep.c sha1.c gitconfig.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../include -I${.CURDIR}/../../lib blob - 952f34c95a8c396a19ea90c1e226f183b6707076 blob + ea559ac400e6c69dfb837d062a2dce93c449ba46 --- libexec/got-read-gotconfig/Makefile +++ libexec/got-read-gotconfig/Makefile @@ -4,7 +4,7 @@ PROG= got-read-gotconfig SRCS= got-read-gotconfig.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c parse.y + path.c privsep.c sha1.c parse.y pollfd.c CPPFLAGS = -I${.CURDIR}/../../include -I${.CURDIR}/../../lib -I${.CURDIR} blob - 35015b318e4e72123a2d4ea6acf7bc07923c0239 blob + 1e6ac4a2c151ff6879da233913d87c73a8f74d31 --- libexec/got-read-object/Makefile +++ libexec/got-read-object/Makefile @@ -4,7 +4,7 @@ PROG= got-read-object SRCS= got-read-object.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c + path.c privsep.c sha1.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../include -I${.CURDIR}/../../lib blob - 4c0bf0f3e8dcb076c30c707c610ceca0aa0bdf80 blob + fd68c400e053f3ff8aceb09e2f9a600aefb7b52e --- libexec/got-read-object/got-read-object.c +++ libexec/got-read-object/got-read-object.c @@ -64,59 +64,24 @@ send_raw_obj(struct imsgbuf *ibuf, struct got_object * { const struct got_error *err = NULL; uint8_t *data = NULL; - size_t len = 0, consumed; - FILE *f; - struct got_object_id id; - struct got_inflate_checksum csum; - SHA1_CTX sha1_ctx; + off_t size; + size_t hdrlen; - SHA1Init(&sha1_ctx); - memset(&csum, 0, sizeof(csum)); - csum.output_sha1 = &sha1_ctx; - if (lseek(fd, SEEK_SET, 0) == -1) { err = got_error_from_errno("lseek"); - close(fd); - return err; + goto done; } - f = fdopen(fd, "r"); - if (f == NULL) { - err = got_error_from_errno("fdopen"); - close(fd); - return err; - } - - if (obj->size + obj->hdrlen <= GOT_PRIVSEP_INLINE_OBJECT_DATA_MAX) - err = got_inflate_to_mem(&data, &len, &consumed, &csum, f); - else - err = got_inflate_to_fd(&len, f, &csum, outfd); + err = got_object_read_raw(&data, &size, &hdrlen, + GOT_PRIVSEP_INLINE_BLOB_DATA_MAX, outfd, expected_id, fd); if (err) goto done; - if (len < obj->hdrlen || len != obj->hdrlen + obj->size) { - err = got_error(GOT_ERR_BAD_OBJ_HDR); - goto done; - } - - SHA1Final(id.sha1, &sha1_ctx); - if (memcmp(expected_id->sha1, id.sha1, SHA1_DIGEST_LENGTH) != 0) { - char buf[SHA1_DIGEST_STRING_LENGTH]; - err = got_error_fmt(GOT_ERR_OBJ_CSUM, - "checksum failure for object %s", - got_sha1_digest_to_str(expected_id->sha1, buf, - sizeof(buf))); - goto done; - } - - - err = got_privsep_send_raw_obj(ibuf, obj->size, obj->hdrlen, data); - + err = got_privsep_send_raw_obj(ibuf, size, hdrlen, data); done: free(data); - if (fclose(f) == EOF && err == NULL) - err = got_error_from_errno("fclose"); - + if (close(fd) == -1 && err == NULL) + err = got_error_from_errno("close"); return err; } blob - ceb42b9503fa151a6a92e2037cea299350be312c blob + 2d45d0f0c9ea5186421438239984f9606128b553 --- libexec/got-read-pack/Makefile +++ libexec/got-read-pack/Makefile @@ -5,7 +5,7 @@ PROG= got-read-pack SRCS= got-read-pack.c delta.c error.c inflate.c object_cache.c \ object_idset.c object_parse.c opentemp.c pack.c path.c \ - privsep.c sha1.c delta_cache.c + privsep.c sha1.c delta_cache.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../include -I${.CURDIR}/../../lib blob - 8c3583004ea6301c62af855a5468f303c84c2ddb blob + dbd3afa62f25910a42da355909407617dc6c1be7 --- libexec/got-read-patch/Makefile +++ libexec/got-read-patch/Makefile @@ -4,7 +4,7 @@ PROG= got-read-patch SRCS= got-read-patch.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c + path.c privsep.c sha1.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../include -I${.CURDIR}/../../lib blob - 7598ada201b24d72b210818965a8bac29f296617 blob + 54a41c2feaca0262542136fa53e0e55842246d6a --- libexec/got-read-tag/Makefile +++ libexec/got-read-tag/Makefile @@ -4,7 +4,7 @@ PROG= got-read-tag SRCS= got-read-tag.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c + path.c privsep.c sha1.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../include -I${.CURDIR}/../../lib blob - 54b081f2cd307441ac03be98dfdf41222edcf15c blob + 5949d7c7273e1e1a82c7ec225c4e23e5a24119e8 --- libexec/got-read-tag/got-read-tag.c +++ libexec/got-read-tag/got-read-tag.c @@ -48,55 +48,6 @@ catch_sigint(int signo) sigint_received = 1; } -static const struct got_error * -read_tag_object(struct got_tag_object **tag, FILE *f, - struct got_object_id *expected_id) -{ - const struct got_error *err = NULL; - struct got_object *obj = NULL; - size_t len; - uint8_t *p; - struct got_inflate_checksum csum; - SHA1_CTX sha1_ctx; - struct got_object_id id; - - SHA1Init(&sha1_ctx); - memset(&csum, 0, sizeof(csum)); - csum.output_sha1 = &sha1_ctx; - - err = got_inflate_to_mem(&p, &len, NULL, &csum, f); - if (err) - return err; - - SHA1Final(id.sha1, &sha1_ctx); - if (memcmp(expected_id->sha1, id.sha1, SHA1_DIGEST_LENGTH) != 0) { - char buf[SHA1_DIGEST_STRING_LENGTH]; - err = got_error_fmt(GOT_ERR_OBJ_CSUM, - "checksum failure for object %s", - got_sha1_digest_to_str(expected_id->sha1, buf, - sizeof(buf))); - goto done; - } - - err = got_object_parse_header(&obj, p, len); - if (err) - goto done; - - if (len < obj->hdrlen + obj->size) { - err = got_error(GOT_ERR_BAD_OBJ_DATA); - goto done; - } - - /* Skip object header. */ - len -= obj->hdrlen; - err = got_object_parse_tag(tag, p + obj->hdrlen, len); -done: - free(p); - if (obj) - got_object_close(obj); - return err; -} - int main(int argc, char *argv[]) { @@ -119,7 +70,6 @@ main(int argc, char *argv[]) for (;;) { struct imsg imsg; - FILE *f = NULL; struct got_tag_object *tag = NULL; struct got_object_id expected_id; @@ -156,25 +106,14 @@ main(int argc, char *argv[]) } /* Always assume file offset zero. */ - f = fdopen(imsg.fd, "rb"); - if (f == NULL) { - err = got_error_from_errno("fdopen"); - goto done; - } - - err = read_tag_object(&tag, f, &expected_id); + err = got_object_read_tag(&tag, imsg.fd, &expected_id, 0); if (err) goto done; err = got_privsep_send_tag(&ibuf, tag); done: - if (f) { - if (fclose(f) == EOF && err == NULL) - err = got_error_from_errno("fclose"); - } else if (imsg.fd != -1) { - if (close(imsg.fd) == -1 && err == NULL) - err = got_error_from_errno("close"); - } + if (imsg.fd != -1 && close(imsg.fd) == -1 && err == NULL) + err = got_error_from_errno("close"); imsg_free(&imsg); if (err) break; blob - 368c85121fd8ecdfc00bf5c5858e4a7bfaf57145 blob + 4312c216c70d94a4b1678f9cca0fd1d41608c347 --- libexec/got-read-tree/Makefile +++ libexec/got-read-tree/Makefile @@ -4,7 +4,7 @@ PROG= got-read-tree SRCS= got-read-tree.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c + path.c privsep.c sha1.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../include -I${.CURDIR}/../../lib blob - 70c37485b3b652e8c3c0fd4dfe8a25350f6d1565 blob + d9873e52a568090e2e3a5702e3ba8555119d31e2 --- libexec/got-read-tree/got-read-tree.c +++ libexec/got-read-tree/got-read-tree.c @@ -49,54 +49,6 @@ catch_sigint(int signo) sigint_received = 1; } -static const struct got_error * -read_tree_object(struct got_parsed_tree_entry **entries, size_t *nentries, - size_t *nentries_alloc, uint8_t **p, FILE *f, struct got_object_id *expected_id) -{ - const struct got_error *err = NULL; - struct got_object *obj = NULL; - size_t len; - struct got_inflate_checksum csum; - SHA1_CTX sha1_ctx; - struct got_object_id id; - - SHA1Init(&sha1_ctx); - memset(&csum, 0, sizeof(csum)); - csum.output_sha1 = &sha1_ctx; - - err = got_inflate_to_mem(p, &len, NULL, &csum, f); - if (err) - return err; - - SHA1Final(id.sha1, &sha1_ctx); - if (memcmp(expected_id->sha1, id.sha1, SHA1_DIGEST_LENGTH) != 0) { - char buf[SHA1_DIGEST_STRING_LENGTH]; - err = got_error_fmt(GOT_ERR_OBJ_CSUM, - "checksum failure for object %s", - got_sha1_digest_to_str(expected_id->sha1, buf, - sizeof(buf))); - goto done; - } - - err = got_object_parse_header(&obj, *p, len); - if (err) - goto done; - - if (len < obj->hdrlen + obj->size) { - err = got_error(GOT_ERR_BAD_OBJ_DATA); - goto done; - } - - /* Skip object header. */ - len -= obj->hdrlen; - err = got_object_parse_tree(entries, nentries, nentries_alloc, - *p + obj->hdrlen, len); -done: - if (obj) - got_object_close(obj); - return err; -} - int main(int argc, char *argv[]) { @@ -121,7 +73,6 @@ main(int argc, char *argv[]) for (;;) { struct imsg imsg; - FILE *f = NULL; uint8_t *buf = NULL; struct got_object_id expected_id; @@ -158,27 +109,16 @@ main(int argc, char *argv[]) } /* Always assume file offset zero. */ - f = fdopen(imsg.fd, "rb"); - if (f == NULL) { - err = got_error_from_errno("fdopen"); - goto done; - } - - err = read_tree_object(&entries, &nentries, &nentries_alloc, - &buf, f, &expected_id); + err = got_object_read_tree(&entries, &nentries, &nentries_alloc, + &buf, imsg.fd, &expected_id); if (err) goto done; err = got_privsep_send_tree(&ibuf, entries, nentries); done: free(buf); - if (f) { - if (fclose(f) == EOF && err == NULL) - err = got_error_from_errno("fclose"); - } else if (imsg.fd != -1) { - if (close(imsg.fd) == -1 && err == NULL) - err = got_error_from_errno("close"); - } + if (imsg.fd != -1 && close(imsg.fd) == -1 && err == NULL) + err = got_error_from_errno("close"); imsg_free(&imsg); if (err) break; blob - fb9f61072a4162175d328a55017cc83f71e08837 blob + c773102729c2714c83c3b63b36a5ce9699aba978 --- libexec/got-send-pack/Makefile +++ libexec/got-send-pack/Makefile @@ -4,7 +4,8 @@ PROG= got-send-pack SRCS= got-send-pack.c error.c inflate.c object_parse.c \ - path.c privsep.c sha1.c pkt.c gitproto.c ratelimit.c + path.c privsep.c sha1.c pkt.c gitproto.c ratelimit.c \ + pollfd.c CPPFLAGS = -I${.CURDIR}/../../include -I${.CURDIR}/../../lib blob - b710300b01ca493c65b6f6e9c20b53d66fcac6e7 blob + 57a92c08b9d52ddc94273eb0e356af566991e9ea --- libexec/got-send-pack/got-send-pack.c +++ libexec/got-send-pack/got-send-pack.c @@ -537,6 +537,9 @@ send_pack(int fd, struct got_pathlist_head *refs, err = got_error_msg(GOT_ERR_BAD_PACKET, "unexpected message from server"); goto done; + } else if (n >= 4 && strncmp(buf, "ERR ", 4) == 0) { + err = send_error(&buf[4], n - 4); + goto done; } else if (strncmp(buf, "ok ", 3) == 0) { err = send_ref_status(ibuf, buf + 3, 1, refs, delete_refs); blob - 4ce5ebd37a22b2e351c4df6733eff8ab16ca3e27 blob + ed3e1ede4a34e7f539a89738ca6f2d6b40947627 --- regress/delta/Makefile +++ regress/delta/Makefile @@ -1,7 +1,7 @@ .PATH:${.CURDIR}/../../lib PROG = delta_test -SRCS = delta.c error.c opentemp.c path.c inflate.c sha1.c delta_test.c +SRCS = delta.c error.c opentemp.c path.c inflate.c sha1.c delta_test.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../include -I${.CURDIR}/../../lib LDADD = -lz blob - f52c80673381058fa605428772575e17f2276fd9 blob + 70bf4f896a17b76140e11c227248938fe1c0ac7a --- regress/fetch/Makefile +++ regress/fetch/Makefile @@ -6,7 +6,7 @@ SRCS = error.c privsep.c reference.c sha1.c object.c o deflate.c delta.c delta_cache.c object_idset.c object_create.c \ fetch.c gotconfig.c dial.c fetch_test.c bloom.c murmurhash2.c sigs.c \ buf.c date.c object_open_privsep.c read_gitconfig_privsep.c \ - read_gotconfig_privsep.c + read_gotconfig_privsep.c pollfd.c reference_parse.c CPPFLAGS = -I${.CURDIR}/../../include -I${.CURDIR}/../../lib LDADD = -lutil -lz -lm blob - 6dc6aa8d5dacb2eb47651672d28391cfb1ebd270 blob + f835e9f283dceca9d301438c5813505b06d0aa43 --- regress/idset/Makefile +++ regress/idset/Makefile @@ -2,7 +2,7 @@ PROG = idset_test SRCS = error.c sha1.c object_idset.c inflate.c path.c object_parse.c \ - idset_test.c + idset_test.c pollfd.c CPPFLAGS = -I${.CURDIR}/../../include -I${.CURDIR}/../../lib LDADD = -lutil -lz blob - 77026c9204de82a647b51112fe7f475d83c9e480 blob + 426283795a3cb7ae2b87d51af958f4315f3a5f30 --- tog/Makefile +++ tog/Makefile @@ -14,7 +14,7 @@ SRCS= tog.c blame.c commit_graph.c delta.c diff.c \ diff_output_unidiff.c diff_output_edscript.c \ diff_patience.c bloom.c murmurhash2.c sigs.c date.c \ object_open_privsep.c read_gitconfig_privsep.c \ - read_gotconfig_privsep.c + read_gotconfig_privsep.c pollfd.c reference_parse.c MAN = ${PROG}.1 CPPFLAGS = -I${.CURDIR}/../include -I${.CURDIR}/../lib