FreeBSD 14 - Deploying Poudriere + CCache + NGINX on LX2160a

posted revised
October 7, 2024 11:22:15 October 7, 2024 11:22:15

Basic Setup

Configure Latest for FreeBSD Repos

  1. Create local repos dir
  2. Copy generic quarterly config file to latest w/ naming convention in place
  3. Disable generic quarterly from polling
  4. Programmatically set latest config to use the desired polling
  5. Update pkg repo index
  6. Upgrade existing packages to latest
mkdir /usr/local/etc/pkg/repos
cp /etc/pkg/FreeBSD.conf /usr/local/etc/pkg/FreeBSD-latest.conf
echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf
sed -i '' -e 's|quarterly|latest|g' /usr/local/etc/pkg/FreeBSD-latest.conf
pkg update -f
pkg upgrade -y

Dependent Packages

We’ll use these at various stages, either explicitly in commands or implicitly during system administration.

pkg install -y anacron bash bash-completion beadm cmdwatch doas git git-lfs node_exporter pciutils rsync sudo tmux

Optional Packages

The following packages are general purpose tooling and services which are installed on most systems in my infrastructure.

pkg install -y autossh bat eza fusefs-sshfs grc htop hwloc2 hwstat ipmitool
pkg install -y ipsumdump lsof micro nmap pv tree usbutils vim

Enable and Configure Sudo

We’ll enable the wheel group to obtain sudo privs without a password.

\cat<< EOF> /usr/local/etc/sudoers.d/wheel
%wheel ALL=(ALL:ALL) NOPASSWD: ALL
EOF

Poudriere ZFS Datasets

Encrypted Dataset Support

Enable ZFS encryption key loading at boot.

sysrc zfskeys_enable=YES

Required Datasets

The following are created as separate datasets for various reasons:

  1. Some datasets benefit from block level deduplication
  2. Some datasets benefit from smaller or larger ZFS record sizes
  3. Some datasets benefit from greater or lesser compression rates
  4. Responsible dataset security (GBAC,AAA) implies dataset segmentation
  5. Responsible dataset management requires block-level workload isolation
  6. Advanced observability + analytics require workload segmented mount points
zfs create -o mountpoint=/usr/local/poudriere \
    -o reservation=256M \
    -o atime=off \
    -o relatime=on \
    -o recordsize=128K \
    -o dedup=sha256,verify \
    -o compression=zstd-1 \
    zroot/poudriere || exit 11

zfs create -o mountpoint=/usr/local/poudriere/ports \
    -o reservation=256M \
    -o atime=off \
    -o relatime=on \
    -o recordsize=128K \
    -o dedup=sha256,verify \
    -o compression=zstd-1 \
    zroot/poudriere/ports || exit 12

zfs create -o mountpoint=/usr/local/poudriere/distfiles \
    -o reservation=256M \
    -o atime=off \
    -o relatime=on \
    -o recordsize=128K \
    -o dedup=sha256,verify \
    -o compression=zstd-1 \
    zroot/poudriere/distfiles || exit 13

zfs create -o mountpoint=/var/cache \
    -o reservation=256M \
    -o atime=off \
    -o relatime=on \
    -o recordsize=128K \
    -o dedup=sha256,verify \
    -o compression=zstd-1 \
    zroot/var/cache || exit 14

zfs create -o mountpoint=/var/cache/ccache \
    -o reservation=256M \
    -o atime=off \
    -o relatime=on \
    -o recordsize=64K \
    -o dedup=off \
    -o compression=zstd-1 \
    zroot/var/cache/ccache || exit 15

Network Time Improvements

Standard NTPd is mostly awful compared to Chrony, so we’ll install and configure that service here.

pkg inst chrony

cat<< EOF> /tmp/chrony.conf
# Chronyd config -- RFC1918.host v1.1.0 -- FreeBSD 14.0-RELEASE-p4 -- 2023-1219
#-----------------------------------------------------------------------------------------------------------------------
# This file is managed by Ansible
#-----------------------------------------------------------------------------------------------------------------------
## Use public servers from the pool.ntp.org project
pool 0.pool.ntp.org iburst minpoll 6 maxpoll 6
pool 1.pool.ntp.org iburst minpoll 6 maxpoll 6
pool 2.pool.ntp.org iburst minpoll 6 maxpoll 6
pool 3.pool.ntp.org iburst minpoll 6 maxpoll 6

## Record the rate at which the system clock gains/losses time
driftfile  /var/db/chrony/drift

## Location for NTS files
# ntsdumpdir /var/db/chrony
# keyfile /usr/local/etc/chrony.keys

## Allow system clock to be stepped in 3x updates if offset > 1s
makestep 1.0 3

## Enable kernel synchronization of the real-time clock (RTC)
rtconutc
rtcsync

## Increase the min sources required to adjust system clock
minsources 3

## Get TAI-UTC offset and leap seconds from the system tz database
leapsectz right/UTC

## Specify directory for log files
logdir /var/log/chrony
dumpdir /var/db/chrony

## Select which information is logged
logchange 0.5
log measurements statistics tracking rtc
EOF

mv /usr/local/etc/chrony.conf /usr/local/etc/chrony.conf.dist
mv /tmp/chrony.conf /usr/local/etc/chrony.conf

sysrc ntpd_enable=NO
sysrc ntpd_sync_on_start=NO
sysrc chronyd_enable=YES
service chronyd start

Poudriere Config

We’ll configure this here, but we’ll also have a Jinja2 template for it automation purposes. Note that automake is essential to Poudriere and without it there are many packages which will fail to build.

pkg install -y poudriere-devel automake

PARA_JOBS=$(expr $(sysctl -n hw.ncpu) - 4)
PREP_JOBS=$(expr ${PARA_JOBS} + 1)

mv /usr/local/etc/poudriere.conf /usr/local/etc/poudriere.conf.dist

cat <<\EOF >/usr/local/etc/poudriere.conf
ZPOOL=zroot
ZROOTFS=/poudriere
#
FREEBSD_HOST=https://download.freebsd.org
RESOLV_CONF=/etc/resolv.conf
#
# BASEFS=/usr/local/poudriere
BASEFS=/srv/jails/containers/poudriere
#
POUDRIERE_DATA=${BASEFS}/data
USE_PORTLINT=yes
USE_TMPFS=yes
TMPFS_BLACKLIST_TMPDIR=${BASEFS}/data/cache/tmp
#
# DISTFILES_CACHE=/usr/local/poudriere/ports/distfiles
DISTFILES_CACHE=${BASEFS}/ports/distfiles
#
GIT_BASEURL=git.freebsd.org/src.git
GIT_PORTSURL=git.freebsd.org/ports.git
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
BAD_PKGNAME_DEPS_ARE_FATAL=no
CCACHE_DIR=/var/cache/ccache
CCACHE_DIR_NON_ROOT_SAFE=yes
CCACHE_GROUP=ccache
CCACHE_GID=65531
PARALLEL_JOBS=${PARA_JOBS}
PREPARE_PARALLEL_JOBS=${PREP_JOBS}
SAVE_WRKDIR=yes
WRKDIR_ARCHIVE_FORMAT=txz
NOLINUX=yes
ALLOW_MAKE_JOBS=yes
ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py*"
TIMESTAMP_LOGS=yes
MAX_EXECUTION_TIME=86400
NOHANG_TIME=7200
ATOMIC_PACKAGE_REPOSITORY=yes
COMMIT_PACKAGES_ON_FAILURE=no
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=1
PORTTESTING_FATAL=yes
BUILDER_HOSTNAME=${HOSTNAME}.rfc1918.host
PRESERVE_TIMESTAMP=yes
BUILD_AS_NON_ROOT=yes
PORTBUILD_USER=nobody
PORTBUILD_GROUP=nobody
PORTBUILD_UID=65532
PORTBUILD_GID=65532
BUILDNAME_FORMAT="%FT%T%z"
DURATION_FORMAT="%H:%M:%S"
USE_COLORS=yes
TRIM_ORPHANED_BUILD_DEPS=yes
DELETE_UNKNOWN_FILES=yes
DELETE_UNQUEUED_PACKAGES=no
URL_BASE=http://${HOSTNAME}.rfc1918.host
HTML_TYPE="hosted"
HTML_TRACK_REMAINING=yes
DETERMINE_BUILD_FAILURE_REASON=yes
MAKEWORLDARGS="WITHOUT_LLVM_ASSERTIONS=yes WITH_MALLOC_PRODUCTION=yes -DMALLOC_PRODUCTION"
PACKAGE_FETCH_BRANCH=latest
PACKAGE_FETCH_URL=pkg+https://pkg.freebsd.org/\${ABI}
# END
EOF

Setup CCache Perms and Dirs

Create groups for Portbuild and CCache

pw groupadd portbuild -g 65532
pw useradd portbuild -u 65532 -g portbuild -d /nonexistent -s /usr/sbin/nologin
pw groupmod -n portbuild -m root

pw groupadd ccache -g 65531
pw groupmod -n ccache -m root,portbuild

Create dirs and set perms

# this tmpdir could be created as a tmpfs memory backed directory via fstab
#  perhaps later...
mkdir /var/cache/ccache/{debug,tmp}

find /var/cache/ccache/ -type d -exec chmod 2770 {} +
find /var/cache/ccache/ -type f -exec chmod 0660 {} +

chmod 1770 /var/cache/ccache/tmp
chmod -R u+s /var/cache/ccache
chmod -R g+s /var/cache/ccache
chown -R root:ccache /var/cache/ccache

Write CCache config files

echo "umask = 0002" >> /var/cache/ccache/ccache.conf

cat <<EOF >/usr/local/etc/ccache.conf
base_dir       = /var/cache/ccache
cache_dir      = /var/cache/ccache
debug_dir      = /var/cache/ccache/debug
temporary_dir  = /var/cache/ccache/tmp
EOF

Install and Configure NGINX for Web Access

Package additions

pkg inst nginx nginx-prometheus-exporter modsecurity3-nginx

Generating Crypto Requirements

The following OpenSSL command will take some time to complete, so you may want to run it in a different terminal pane from the others, and let it run until it’s finished.

openssl dhparam -out /usr/local/etc/nginx/dhparam.pem 4096

Service configurations

Enable separate group for TLS certs

pw group add -g 644 -M root,www -n pki

mkdir -p /usr/local/etc/pki
chown -R root:pki /usr/local/etc/pki
chmod 750 /usr/local/etc/pki

Nginx Base Config

mkdir /usr/local/etc/nginx/conf.d
chown -R www:www /usr/local/etc/nginx
chmod -R u+s /usr/local/etc/nginx
chmod -R g+s /usr/local/etc/nginx

# adding a log mime-type allows nginx to serve log files in the browser
sed -i '' -E 's|text/plain[\t\ ]*txt|text/plain txt log|g' /usr/local/etc/nginx/mime.types

cat <<EOF >/usr/local/etc/nginx/nginx.conf
## nginx base config for poudriere
user www;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
include /usr/local/etc/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';

  access_log          /var/log/nginx/access.log  main;
  sendfile            on;
  tcp_nopush          on;
  tcp_nodelay         on;
  keepalive_timeout   65;
  types_hash_max_size 2048;
  include             /usr/local/etc/nginx/mime.types;
  default_type        application/octet-stream;

  include /usr/local/etc/nginx/conf.d/*.conf;
}
## END
EOF

Nginx Site Config

The following SSL cert references are specific to this install, as it’s using a Comodo SSL Wildcard. You will want to adjust the file names accordingly to your own SSL cert files.

SRVC_DOMAIN="rfc1918.host"
SRVC_HOST="sys-rfc100-arm64-hclx2n2"
SRVC_FQDN="${SRVC_HOST}.${SRVC_DOMAIN}"

mkdir -p /usr/local/etc/pki/${SRVC_DOMAIN}

cat <<EOF >/usr/local/etc/nginx/conf.d/poudriere.conf
## nginx service config for poudriere
server {
  listen 80;
  server_name     ${SRVC_FQDN};
  access_log      /var/log/nginx/${SRVC_HOST}.access.log;
  error_log       /var/log/nginx/${SRVC_HOST}.error.log;
  location / {
    return 302 https://${SRVC_FQDN}/$request_uri;
  }
}

server {
  listen 443   ssl http2;
  server_name  ${SRVC_FQDN};
  set          $cache_uri $request_uri;
  set          $srvchost  "SRVC_HOST";

  # TLS
  resolver                   9.9.9.9 8.8.8.8 valid=300s;
  resolver_timeout           5s;
  ssl_ciphers                HIGH:!aNULL:!MD5:EECDH+AESGCM:EDH+AESGCM;
  ssl_dhparam                /usr/local/etc/nginx/dhparam.pem;
  ssl_ecdh_curve             secp384r1;
  ssl_prefer_server_ciphers  on;
  ssl_protocols              TLSv1.2 TLSv1.3;
  ssl_session_cache          shared:SSL:1m;
  ssl_session_tickets        off;
  ssl_session_timeout        10m;
  ssl_stapling               on;
  ssl_stapling_verify        on;
  ssl_certificate            /usr/local/etc/pki/${SRVC_DOMAIN}/${SRVC_DOMAIN}_ssl-comodo.ca-cert-chain.pem;
  ssl_certificate_key        /usr/local/etc/pki/${SRVC_DOMAIN}/${SRVC_DOMAIN}_ssl-comodo.key;

  # Allow gzipping js, css, log, svg and json files
  gzip on;
  gzip_buffers 16 8k;
  gzip_comp_level 6;
  gzip_http_version 1.0;
  gzip_min_length  1100;
  gzip_proxied any;
  gzip_vary on;
  gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript image/gif image/jpeg image/png application/json image/svg+xml;

  # Logging
  access_log           /var/log/nginx/${SRVC_HOST}.access.log;
  error_log            /var/log/nginx/${SRVC_HOST}.error.log;

  # Base html dir for poudriere site template
  root         /usr/local/share/poudriere/html;

  # Something probably useful
  location ~ /.well-known {
    allow all;
    }

  # Image file caching
  location ~* ^.+\.(jpg|jpeg|gif|png|ico|svg|woff|css|js|html)$ {
    add_header Cache-Control "public";
    expires 1d;
   }

  # Yep, packages
  location /pkg {
    autoindex on;
    alias /usr/local/poudriere/data/packages;
    }

  # WebDAV
  location /cache/ {
    # Where to store cache files
    # Enable needed HTTP methods
    # Allow creating subdirectories
    # Allow individual cache entries to be up to 100 MiB
    # Don't log 404 Not Found replies as errors
    # Enable WEBDAV reads
    dav_access user:rw group:rw all:r;
    alias /usr/local/poudriere/data/cache;
    log_not_found off;
    dav_methods PUT DELETE;
    create_full_put_path on;
    client_max_body_size 100M;
      }

    ## log data
    # Allow caching dynamic files but ensure they get rechecked
    location /data {
    alias /usr/local/poudriere/data/logs/bulk;
    location ~* ^.+\.(log|txz|tbz|bz2|gz)$ {
      add_header Cache-Control "public, must-revalidate, proxy-revalidate";
        }
      }

    # Don't log json requests as they come in frequently and ensure
    #   caching works as expected
    location ~* ^.+\.(json)$ {
      add_header Cache-Control "public, must-revalidate, proxy-revalidate";
      access_log off;
      log_not_found off;
      }

    # Allow indexing only in log dirs
    location ~ /data/?.*/(logs|latest-per-pkg)/ {
      autoindex on;
      }
    }

  ## Testing
  location /pkg/FreeBSD:14:arm64 {
    autoindex on;
    alias /usr/local/poudriere/data/packages/FBSD-140-ARM64j-FBSDp;
    }
}
## END

Enable and Start Nginx

sysrc nginx_enable=YES

Poudriere Operator Notes

Create two builder jails

Because reasons, we’ll create one FreeBSD 14 and one FreeBSD 15 jail… naming conventions subject to change…

poudriere jail -c -j FBSD-140-ARM64j -v 14.0-RELEASE -a arm64.aarch64
poudriere jail -c -j FBSD-150-ARM64j -v 15.0-CURRENT -a arm64.aarch64

Create port tree from git repo using main branch

poudriere ports -c -p FBSDp -m git+https -B main -U https://git.freebsd.org/ports.git

[1] Create port repo based pkglist

cd /usr/local/poudriere/ports/FBSDp
find ./ -type d -depth 2 \
  | awk -F/ '{print $2"/"$3}' \
  | sort \
  | grep -v Makefile \
  | egrep -v "Mk/"\
    |"Templates/Licenses"\
    |"Tools/scripts" \
  > /usr/local/etc/poudriere.d/FBSD-140-ports.repo-all.pkglist

[2] Create ‘installed’ query for pkglist (includes deps)

pkg query -e %a=0 %o | sort -h > /usr/local/etc/poudriere.d/FBSD-140-builder.query-all.pkglist

[2] Create ‘installed’ query for pkglist (without deps)

pkg prime-origins | sort -h > /usr/local/etc/poudriere.d/FBSD-140-builder.prime-origins.pkglist

Set options for a given port (if needed)

poudriere options -j JAIL-NAME -p FBSDp -n CATEGORY/PORTNAME

Set ‘make/config’ options for a package + no-alter of dependency options

poudriere options -p FBSDp -c -n CATEGORY/PORTNAME

Create build session with pkglist file

poudriere bulk -j JAIL-NAME -p FBSDp -f <pkglist file>

Create build session with all pkgs

poudriere bulk -j JAIL-NAME -p FBSDp -a

Poudriere Setup for HardenedBSD

HBSD Public Build Servers

Download Server for Installers

Git Repo Server for OS Sources

Git Repo Server for Ports

Jail Setup and Management

Create Jail via Git

poudriere jail -c -j HBSD-14-STABLE-GIT-AMD64j -v hardened/14-stable/master -a amd64 -m git+ssh -U git@git.hardenedbsd.org:hardenedbsd/HardenedBSD.git -K HARDENEDBSD

Create Jail via Tgz

Syntax for 14-STABLE

poudriere jail -c -j HBSD-14-STABLE-URL-AMD64j -v hardened/14-stable -a amd64 -m url=https://installers.hardenedbsd.org/pub/current/amd64/amd64/installer/LATEST/

Syntax for 13-STABLE

poudriere jail -c -j HBSD-13-STABLE-URL-AMD64j -v hardened/13-stable -a amd64 -m url=https://installers.hardenedbsd.org/pub/13-stable/amd64/amd64/installer/LATEST/

ARM64 Jails - Git only

Syntax for 14-STABLE

poudriere jail -c -j HBSD-14-STABLE-GIT-ARM64j -v hardened/14-stable/master -a arm64.aarch64 -m git+ssh -U git@git.hardenedbsd.org:hardenedbsd/HardenedBSD.git -K HARDENEDBSD

Ports Build Config

Create ports tree git+https (does not require ssh key on repo)

poudriere ports -c -p HBSDp -m git+https -B hardenedbsd/main -U https://git.hardenedbsd.org/hardenedbsd/ports.git

Create ports tree git+ssh (req ssh key added to repo)

poudriere ports -c -p HBSDp -m git+ssh -B hardenedbsd/main -U git@git.hardenedbsd.org:hardenedbsd/ports.git

Create build session with all pkgs from port repo

poudriere bulk -j HBSD-14-STABLE-URL-AMD64j -p HBSDp -a

(optional) Update the jail and build a selected kernel from src

poudriere jail -u -j HBSD-14-STABLE-GIT-AMD64j -p HBSDp -v hardened/14-stable/master -a amd64 -m git+ssh -U git@git.hardenedbsd.org:hardenedbsd/HardenedBSD.git -K HARDENEDBSD

References