guix-forge
|
| 1 | Introduction | |||
| 1.1 | Philosophy | |||
| 2 | Tutorial | |||
| 3 | How To | |||
| 3.1 | How to set up cgit | |||
| 4 | Services | |||
| 4.1 | Git web viewers | |||
| 4.1.1 | cgit service | |||
| 4.2 | forge nginx service | |||
| 4.3 | ACME service | |||
| 4.4 | Specialized application deployment services | |||
| 4.4.1 | fcgiwrap service | |||
| 4.4.2 | gunicorn service | |||
| 5 | Reference | |||
guix-forge is a Guix service that lets you run a complete software forge in the manner of GitHub, GitLab, etc. Unlike other free software forges such as GitLab, Gitea, etc., guix-forge is not a monolith but is an assemblage of several pieces of server software wired up to function as one. In this sense, it is a meta-service. guix-forge does for software forges what Mail-in-a-Box does for email.
guix-forge integrates the following software components:
In the future, it will also provide:
A choice of different software components may be offered provided it does not complicate the interface too much.
guix-forge is provided on a best effort basis. Its design is unstable, and open to change. We will try our best to not break your system configuration often, but it might happen.
In order to empower ordinary users, software should not just be free (as in freedom), but also be simple and easy to deploy, especially for small-scale deployments. guix-forge is therefore minimalistic, and does not require running large database servers such as MariaDB and PostgreSQL.
While some state is inevitable, server software should strive to be as stateless as an old analog television set. You switch it on, and it works all the time. There are no pesky software updates, and complex hidden state. guix-forge tries to be as stateless as possible. Almost all of guix-forge's state can be version controlled, and the rest are simple files that can be backed up easily.
Git is already federated and decentralized with email. guix-forge acknowledges this and prefers to support git's email driven workflow with project discussion, bug reports and patches all happening over email.
guix-forge is opinionated and will not expose all features provided by the software components underlying it. Keeping configuration options to a minimum is necessary to help casual users deploy their own forge, and to reduce the likelihood of configuration bugs.
In this tutorial, you will learn how to set up guix-forge to host continuous integration for a project. For the purposes of this tutorial, we will set up continuous integration for the guile-json project.
First, we clone the upstream guile-json repository into a local bare clone at /srv/git/guile-json.
$ git clone --bare https://github.com/aconchillo/guile-json /srv/git/guile-json Cloning into bare repository '/srv/git/guile-json'... remote: Enumerating objects: 1216, done. remote: Counting objects: 100% (162/162), done. remote: Compressing objects: 100% (107/107), done. remote: Total 1216 (delta 96), reused 106 (delta 54), pack-reused 1054 Receiving objects: 100% (1216/1216), 276.10 KiB | 3.89 MiB/s, done. Resolving deltas: 100% (742/742), done.
Now that we have a git repository to work with, we start
writing our Guix system configuration. We begin with a bunch of use-modules statements importing all required modules.
(use-modules (gnu) (gnu packages autotools) (gnu packages gawk) (gnu packages guile) (gnu packages pkg-config) (gnu packages version-control) (gnu services ci) (forge forge) (forge laminar) (forge utils))
Then, we define the G-expression that will be run as a continuous integration job
on every commit. This G-expression uses invoke from (guix build utils). Hence, we make it available to the G-expression
using with-imported-modules. In addition, it needs a number
of packages which we make available using with-packages. And
finally, within the body of the G-expression, we have commands cloning
the git repository, building the source and running the tests.
The attentive reader may notice what looks like (guix
build utils) being referenced twice—once with with-imported-modules and again with use-modules. This is
not a mistake. G-expressions are serialized into Guile scripts. with-imported-modules ensures that code for (guix build
utils) is available and is in the load path. use-modules actually imports (guix build utils) when the script runs. with-imported-modules is like installing a library in your system,
and use-modules is like actually importing that library in a
script. Both are necessary.
(define guile-json-tests (with-imported-modules '((guix build utils)) (with-packages (list autoconf automake coreutils gawk git-minimal gnu-make grep guile-3.0 sed pkg-config) #~(begin (use-modules (guix build utils)) (invoke "git" "clone" "/srv/git/guile-json" ".") (invoke "autoreconf" "--verbose" "--install" "--force") (invoke "./configure") (invoke "make") (invoke "make" "check")))))
Now, we configure a <forge-project> record that
holds metadata about the project and wires up the G-expression we just
defined into a continuous integration job.
(define guile-json-project (forge-project (name "guile-json") (user "vetri") (repository "/srv/git/guile-json") (description "JSON module for Guile") (ci-jobs (list (forge-laminar-job (name "guile-json") (run guile-json-tests))))))
The name and description fields are
hopefully self-explanatory. The user field specifies the
user who will own the git repository at the path specified by repository. That user will therefore be able to push into the
repository through ssh or similar. git provides various server-side hooks that trigger on various events. Of these, the
post-receive hook triggers when pushed commits are
received. guix-forge sets up a post-receive hook script in
the repository to trigger a continuous integration run on every
git push.
And finally, we put everything together in an operating-system declaration. Notice the forge service configured
with guile-json-project and the laminar service configured
with a port for the web interface to listen on.
(operating-system (host-name "tutorial") (timezone "UTC") (bootloader (bootloader-configuration (bootloader grub-bootloader))) (file-systems %base-file-systems) (users (cons* (user-account (name "vetri") (group "users") (home-directory "/home/vetri")) %base-user-accounts)) (packages %base-packages) (services (cons* (service forge-service-type (forge-configuration (projects (list guile-json-project)))) (service laminar-service-type (laminar-configuration (bind-http "localhost:8080"))) %base-services)))
Now that we have a complete operating-system
definition, let's use the following command to build a
container. After a lot of building, a container script should pop
out.
$ guix system container --network --share=/srv/git/guile-json tutorial.scm /gnu/store/ilg7c2hpkxhwircxpz22qhjsqp3i9har-run-container
The --network flag specifies that the container
should share the network namespace of the host. To us, this means that
all ports opened by the container will be visible on the host without
any port forwarding or complicated configuration. The --share=/srv/git/guile-json option shares the git repository we
cloned earlier, with the container.
To start the container, simply run the container script as root.
# /gnu/store/ilg7c2hpkxhwircxpz22qhjsqp3i9har-run-container
Now, you can see the status of laminar and running jobs through its web interface listening on http://localhost:8080. You can list and queue jobs on the command-line like so:
$ laminarc show-jobs guile-json $ laminarc queue guile-json guile-json:1
That's it! You just set up your own continuous integration system and took the first steps to owning your code!
You could easily use the same configuration to configure a Guix system instead of a container. To do so, you will have to take care of defining the bootloader, file systems and other settings as per your needs. The overall configuration used in this tutorial is repeated below for your reference.
1: (use-modules (gnu) 2: (gnu packages autotools) 3: (gnu packages gawk) 4: (gnu packages guile) 5: (gnu packages pkg-config) 6: (gnu packages version-control) 7: (gnu services ci) 8: (forge forge) 9: (forge laminar) 10: (forge utils)) 11: 12: (define guile-json-tests 13: (with-imported-modules '((guix build utils)) 14: (with-packages (list autoconf automake coreutils 15: gawk git-minimal gnu-make grep 16: guile-3.0 sed pkg-config) 17: #~(begin 18: (use-modules (guix build utils)) 19: (invoke "git" "clone" "/srv/git/guile-json" ".") 20: (invoke "autoreconf" "--verbose" "--install" "--force") 21: (invoke "./configure") 22: (invoke "make") 23: (invoke "make" "check"))))) 24: 25: (define guile-json-project 26: (forge-project 27: (name "guile-json") 28: (user "vetri") 29: (repository "/srv/git/guile-json") 30: (description "JSON module for Guile") 31: (ci-jobs (list (forge-laminar-job 32: (name "guile-json") 33: (run guile-json-tests)))))) 34: 35: (operating-system 36: (host-name "tutorial") 37: (timezone "UTC") 38: (bootloader (bootloader-configuration 39: (bootloader grub-bootloader))) 40: (file-systems %base-file-systems) 41: (users (cons* (user-account 42: (name "vetri") 43: (group "users") 44: (home-directory "/home/vetri")) 45: %base-user-accounts)) 46: (packages %base-packages) 47: (services (cons* (service forge-service-type 48: (forge-configuration 49: (projects (list guile-json-project)))) 50: (service laminar-service-type 51: (laminar-configuration 52: (bind-http "localhost:8080"))) 53: %base-services)))
guix-forge comes with an end-to-end cgit solution that not only sets up cgit itself but also an nginx server complete with automatically renewed TLS certificates. cgit even runs in its own container for maximal security.
The cgit service uses the forge-nginx service as its web server. The forge-nginx service in turn uses the ACME service to fetch and renew TLS certificates. Here's a minimal working configuration.
(use-modules (gnu) ((gnu packages admin) #:select (shepherd)) (forge acme) (forge cgit) (forge nginx) (forge socket)) (operating-system (host-name "forge") (timezone "UTC") (locale "en_US.utf8") (bootloader (bootloader-configuration (bootloader grub-bootloader) (targets (list "/dev/sdX")))) (file-systems %base-file-systems) (sudoers-file (mixed-text-file "sudoers" "@include " %sudoers-specification ;; Permit the acme user to restart nginx. "\nacme ALL = NOPASSWD: " (file-append shepherd "/bin/herd") " restart nginx\n")) (services (cons* (service cgit-service-type (cgit-configuration (server-name "git.example.org") (repository-directory "/srv/git"))) (service forge-nginx-service-type (forge-nginx-configuration (http-listen (forge-ip-socket (ip "0.0.0.0") (port 8080))) (https-listen (forge-ip-socket (ip "0.0.0.0") (port 4443))))) (service acme-service-type (acme-configuration (email "foo@example.org"))) %base-services)))
The cgit service configuration specifies the domain git.example.org to serve cgit on and the /srv/git repository directory containing bare git repositories to publish. The forge nginx service configuration specifies the ports to serve HTTP and HTTPS on. The ACME service configuration specifies the email address to register an ACME account with. The sudoers file declaration is required to allow the acme user to restart the nginx server when a certificate is renewed. The configured machine will start out with self-signed certificates. Run /usr/bin/acme renew the first time to get CA-issued certificates. Thereafter, certificates will auto-renew via a cron job.
When testing your deployment, it might help to start with the Let's Encrypt staging server as shown below. This will give you dummy certificates, but will help you avoid running afoul of Let's Encrypt rate limits. Once you know everything works, delete the ACME state directory (/var/lib/acme by default) and run /usr/bin/acme renew again to get real certificates.
(service acme-service-type (acme-configuration (email "foo@example.org") (acme-url %letsencrypt-staging-url)))
If you are running guix-forge in a Guix system container, do remember to mount the ACME state directory (/var/lib/acme by default) into the container from persistent storage.
cgit is a web frontend to serve git repositories on the web. Our cgit service features
git-http-backend<cgit-configuration>cgit (Default: cgit)cgit package to usegit (Default: git-without-safe-directory-check)git package to use. git provides the
smart HTTP protocol backend.server-namerepository-directory (Default: "/srv/git")socket (Default: (forge-unix-socket (path "/var/run/fcgiwrap/cgit/socket")))readme (Default: %cgit-readme)snapshots (Default: (list "tar.gz"))"tar", "tar.gz", "tar.bz2", "tar.lz", "tar.xz", "tar.xst" and "zip".about-filter (Default: (program-file "about-filter" (about-filter-gexp this-cgit-configuration)))commit-filter (Default: #f)email-filter (Default: #f)source-filter (Default: (file-append (cgit-configuration-cgit this-cgit-configuration) "/lib/cgit/filters/syntax-highlighting.py"))mimetype-file (Default: (file-append mailcap "/etc/mime.types"))repository-sort (Default: 'age)'name (sorting by repository name)
and 'age (sorting most recently updated repository
first).plain-email?#true, full email addresses will be
shown. Else, they won't.extra-options (Default: '())The forge nginx service is a wrapper around the nginx web service in Guix upstream. It features
When using this service, you must allow the acme
user to restart nginx using sudo. This is so that newly
obtained certificates can be deployed to nginx. You may achieve this
with the following in the sudoers-file field of your operating-system definition.
(operating-system … (sudoers-file (mixed-text-file "sudoers" "@include " %sudoers-specification "\nacme ALL = NOPASSWD: " (file-append shepherd "/bin/herd") " restart nginx\n")) …)
<forge-nginx-configuration>http-listen (Default: (forge-ip-socket (ip "0.0.0.0") (port 80)))<forge-host-socket>, <forge-ip-socket>, or <forge-unix-socket> object.https-listen (Default: (forge-ip-socket (ip "0.0.0.0") (port 443)))<forge-host-socket>, <forge-ip-socket>, or <forge-unix-socket> object.acme-state-directory (Default: "/var/lib/acme")acme-challenge-directory (Default: "/var/run/acme/acme-challenge")server-blocks (Default: '())ACME (Automatic Certificate Management Environment) is a protocol popularized by the Let's Encrypt certificate authority for the automatic issue and renewal of TLS certificates. The guix-forge ACME service featuers
The first time the ACME service is set up or each time new certificates are configured, self-signed certificates are created so that processes (such as nginx) that depend on these certificates can start up successfully. You must replace these with certificate authority issued certificates by running /usr/bin/acme renew. /usr/bin/acme renew automatically registers an ACME account unless one already exists and renews all configured certificates. It uses parameters that were configured in the ACME service and does not need any additional command-line arguments.
The ACME service does not use certbot, the official Let's Encrypt client. It instead uses uacme. uacme is smaller, simpler, manages far less state, does no magic, and is better suited to automation. However, the choice of backend tool is an implementation detail. The ACME service is an abstract service that is largely independent of the backend tool that powers it.
By using the ACME service, you agree to the Terms of Service of your ACME server.
<acme-configuration>uacme (Default: uacme)uacme package to useemailacme-url (Default: %letsencrypt-production-url)%letsencrypt-staging-url when testing your deployment.state-directory (Default: "/var/lib/acme")http-01-challenge-directory (Default: "/var/run/acme/acme-challenge")http-01-authorization-hook (Default: (program-file "acme-http-01-authorization-hook" (acme-http-01-webroot-authorization-gexp this-acme-configuration)))http-01-cleanup-hook (Default: (program-file "acme-http-01-cleanup-hook" (acme-http-01-webroot-cleanup-gexp this-acme-configuration)))key (Default: (acme-ecdsa-key))<acme-rsa-key> or <acme-ecdsa-key> object describing the ACME account and TLS
certificate keys. Changing this field does not affect keys already
generated and stored on disk.certificates (Default: '())<acme-certificate> objects
describing certificates to configureThe http-01-authorization-hook and http-01-cleanup-hook scripts are invoked with the following three
command-line arguments.
identifiertokenauth<acme-certificate>domainsdeploy-hook<acme-rsa-key>length (Default: 2048)<acme-ecdsa-key>length (Default: 256)fcgiwrap is a specialized web server for CGI applications. It provides a FastCGI interface that web servers such as nginx can talk to. We run separate containerized instances of fcgiwrap for each application.
Note that this service is different from the fcgiwrap service of the same name in Guix upstream.
<fcgiwrap-configuration>package (Default: fcgiwrap)fcgiwrap package to useinstances (Default: '())<fcgiwrap-instance> objects
describing fcgiwrap instances to run<fcgiwrap-instance>namesocket (Default: (forge-unix-socket (path (string-append "/var/run/fcgiwrap/" (fcgiwrap-instance-name this-fcgiwrap-instance) "/socket"))))<forge-host-socket>, <forge-ip-socket>
or <forge-unix-socket> object.usergroupprocesses (Default: 1)environment-variables (Default: '())<environment-variable> objects
describing environment variables that should be set in the execution
environmentmappings (Default: '())<file-system-mapping> objects
describing additional directories that should be shared with the
container fcgiwrap is run ingunicorn is a specialized web server for Python WSGI applications. We run separate containerized instances of gunicorn for each application.
<gunicorn-configuration>package (Default: gunicorn)gunicorn package to useapps (Default: '())<gunicorn-app> objects describing
gunicorn apps to run<gunicorn-app>namepackagewsgi-app-modulesockets (Default: (list (forge-unix-socket (path (string-append "/var/run/gunicorn/" (gunicorn-app-name this-gunicorn-app) "/socket")))))<forge-host-socket>, <forge-ip-socket> or <forge-unix-socket> objects
describing sockets to listen onworkers (Default: 1)timeout (Default: 30)extra-cli-arguments (Default: '())environment-variables (Default: '())<environment-variable> objects
describing environment variables that should be set in the execution
environmentmappings (Default: '())<file-system-mapping> objects describing
additional directories that should be shared with the container
gunicorn is run in<environment-variable>namevalue<forge-configuration>projects (Default: '())<forge-project> objects describing
projects managed by guix-forge<forge-project>namerepositoryuser (Default: #f)description (Default: #f)website-directory (Default: #f)laminar
user. The idea is that the website is built by a Guix derivation as a
store item and a symbolic link to that store item is created in the
parent directory.ci-jobs (Default: '())<forge-laminar-job> objects
describing CI (continuous integration) jobs to
configureci-jobs-trigger (Default: 'post-receive-hook for local repositories
and 'cron for remote repositories)'post-receive-hook, 'webhook, or
'cron representing the type of trigger for continuous
integration jobs.
'post-receive-hook'post-receive-hook is specified, the post-receive hook of the repository is configured to trigger CI
jobs. This is possible only for local repositories. Note that any
pre-existing post-receive hook is overwritten.'webhook'webhook is specified, a webhook server is configured to trigger CI
jobs when a request is received on http://hostname:port/hooks/<name> .'cron'cron is
specified, a cron job triggers the CI jobs once a day.parallel-ci-job-runs (Default: 1)repository-branch (Default: "main")<forge-laminar-job>namerunafter (Default: #f)trigger? (Default: #t)#t, this job is run on every commit. Else, it
must be manually set up to run some other way.contexts (Default: '())<forge-host-socket>hostname (Default: "localhost")port<forge-ip-socket>ip (Default: "127.0.0.1")"127.0.0.1" and "::1" for IPv4 and IPv6
respectively. The any address is "0.0.0.0" and "::" for IPv4 and IPv6 respectively.port<forge-unix-socket>path<webhook-configuration>package (Default: webhook)webhook package to usesocket (Default: (forge-ip-socket (ip "127.0.0.1") (port 9000)))<forge-ip-socket> object, to listen
on.log-directory (Default: "/var/log/webhook")hooks (Default: '())<webhook-hook> objects describing
hooks to configure<webhook-hook>idrun(klaus-gunicorn-app repository-directory #:key (klaus python-klaus) (sockets (list (forge-unix-socket (path "/var/run/gunicorn/klaus/socket")))) site-name)Return a <gunicorn-app> object to deploy klaus.
repository-directory is the path to the directory containing git repositories to serve.
klaus is the klaus package to use.
sockets is a list of <forge-ip-socket> or <forge-unix-socket> objects describing sockets to listen on.
site-name is the name of the klaus site to be displayed in the banner.