pkgbase: what can go wrong?

Yesterday I wrote about how I ended up manually converting a FreeBSD 15 server and the eight jails it hosts to packaged base manually. I dashed that post off in a bit of a hurry because I knew that if I put it off until later I would probably never get around to it. 20 or so hours later, here’s a followup with some caveats and some tips on how to ensure the conversion goes well.

First, I should point out that the procedure outlined in yesterday’s post has no guardrails and should not be attempted unless you are very comfortable with FreeBSD system administration and willing and able to fix things if they go awry.

Second, perhaps I should elaborate on why I (mostly) skipped 15.0. We (the FreeBSD project collectively) have been working on packaged base for over a decade. At some point we thought 14.0 was when it would finally land, but that did not pan out. Then it was going to be 15.0, and there was a massive push to get everything ready, both in base and in pkg… and we still failed: packaged base in FreeBSD 15 is officially a technology preview, and FreeBSD 16 is when we will finally transition. As a result, the pre-release churn on the stable/15 branch was much higher than for a normal major release and a number of things fell through the cracks (including some that had nothing to do with packaged base). The bottom line is that I did not feel that 15.0 was production-ready when it came out, but I worked hard to get most of the issues I was aware of sorted out for 15.1, and I feel much better about 15.1 than I did about 15.0.

Third, I left out a couple of steps for brevity. Most importantly, the systems I converted were all on ZFS (eight jails and their host, as previously mentioned) which allowed me to take a snapshot of the root filesystem (which in a standard FreeBSD install includes /var and /usr/local) so if something had gone wrong I could simply have reverted to that, or rebooted to the boot environment that freebsd-update created when it upgraded my system to 15.1.

My case is also atypical because I have my own package repositories, including for packaged base. I converted the host to packaged base using the official FreeBSD-base repository first, then switched to my own before running pkg upgrade -fy; I then converted the jails directly to my own repository without passing through FreeBSD-base. This is somewhat risky because my base packages are built with non-standard options and do not match FreeBSD distribution sets perfectly, but I know exactly where they differ and how to deal with it.

Assuming your system meets the prerequisites (recently updated binary install of 15.0 or 15.1), the worst thing that can happen is that you forget to install a set and end up with unregistered files (e.g. you had tests.txz installed on your system but left out FreeBSD-set-tests from your pkg install --register-only command line). This is easily remedied, even after the fact, by registering that set and then removing it:

# pkg install --register-only -y FreeBSD-set-tests
# pkg remove -fy FreeBSD-set-tests
# pkg autoremove -y

But watch out! FreeBSD-set-tests depends on FreeBSD-set-base which depends on FreeBSD-set-minimal and FreeBSD-set-optional, so if you had already trimmed your system down to e.g. FreeBSD-set-minimal-jail you will have to remove those extra sets again:

# pkg remove -fy FreeBSD-set-{base,minimal,optional}
# pkg autoremove -y

Since we used --register-only, one thing that can’t happen is that pkg replaces a modified configuration file with its stock version, saving the modified version with a .pkgsave extension. On the other hand, it is possible that pkg upgrade -fy fails to install a file because a directory of the same name exists. I’ve had that happen exactly once due to a discrepancy in /usr/src (I’m not sure why but freebsd-update(8) has never been entirely reliable when it came to patching the source tree). So you may want to run the following after converting:

# find -xs / -name '*.pkgnew'

If you get any hits, there will usually be an empty directory of the same name (without the .pkgnew suffix) which you can remove before manually renaming the file.

Another useful post-conversion command is:

# find -xs / \( -name var -prune \) -or -not -type d -print0 |
xargs -0 pkg which 2>&1 | grep 'not found' | less

This will give you a list of files on the root filesystem which pkg does not know about. You should be able to identify most of them (configuration files, the trust store, miscellaneous generated files such as /etc/spwd.db).

Finally, look for empty directories on the root filesystem:

# find -xs / -type d -empty

There will usually be a few in /usr/lib/debug and /usr/tests which you can remove if you did not choose to install debugging symbols or tests.

And my final piece of advice, if you want minimum hassle at the cost of having the machine do some pointless work that immediately gets reverted, just register all the sets, then remove the ones you don’t have or don’t want. You’ll get errors from autoremove about missing files, but that (and the few dozen extra seconds the process will take) is about the only downside.

pkgbasify

How to easily convert your FreeBSD 15 system to packaged base

I just spent the day catching up on FreeBSD email and events after a week of marshaling, and upgrading various systems to FreeBSD 15.1. Most of my various systems (servers, VMs, and jails) are still running 14.3 or 14.4 because, in my opinion, 15.0 shipped with too many unresolved warts, but now that 15.1 is out (albeit not yet announced) it is high time to not only upgrade everything to 15.1 but also make the switch to packaged base (which I was so far only using on one VM and one jail).

After a brief look at the FreeBSD Foundation’s pkgbasify script I came to the conclusion that I do not trust it to work correctly, so I developed my own method for converting a system from distribution sets to packaged base. Assuming a full install (including debug symbols, source tree, and tests) of FreeBSD 15.0-RELEASE or 15.1-RELEASE with pkg 2.7.5 recently updated to the latest patch level (if any), you can convert to packaged base with a single command:

# pkg install -r FreeBSD-base --register-only FreeBSD-set-{base,lib32,kernels}{,-dbg} FreeBSD-set-{src,tests}

Note 1: I’m assuming that you are using bash or zsh or some other shell that supports curly brace notation; FreeBSD’s /bin/sh does not.

Note 2: the pkg-install(8) manual page claims that you can abbreviate --register-only to -X, but you can’t, although that will be fixed in the next version.

There may be minor discrepancies between what the installer and freebsd-update(8) produced and what the packages contain, which you can usually fix with a forced upgrade:

# pkg upgrade -r FreeBSD-base -fy

If you do not have a full install, you should trim down the pkg install command line to only the sets that correspond to what you have; otherwise, the subsequent pkg upgrade will install what was missing. Particularly, you should not install FreeBSD-set-src if /usr/src is already managed by Git.

Once you’re done, you can start trimming down your system by removing sets you don’t need, then autoremoving their contents. For instance, if you’ve decided that you no longer need tests or debugging symbols:

# pkg remove -fg FreeBSD-set-\*-dbg FreeBSD-set-tests
# pkg autoremove

Note that package sets are marked vital so you must use -f when removing them.

Now is the time to enable the FreeBSD-base repository, by the way. Manually specifying it with -r FreeBSD-base temporarily enables it for the current command, but you’ll want it permanently enabled:

# mkdir -p /usr/local/etc/pkg/repos
# echo 'FreeBSD-base: { enabled: yes }' >/usr/local/etc/pkg/repos/FreeBSD-base.conf

Finally, if on bare metal or a VM, create a boot environment of your converted system:

# bectl create $(pkg query %v FreeBSD-runtime)

The only issue I’ve noticed after converting one bare-metal server and the eight jails it hosts is that pkg leaf does not take shared library dependencies into account:

# sudo pkg -j myjail leaf
FreeBSD-bluetooth-lib-15.1
FreeBSD-set-base-jail-15.1
FreeBSD-ufs-lib-15.1
FreeBSD-zfs-lib-15.1
pkg-2.7.5

Those three -lib packages provide libraries which are used by other base packages, but the dependencies are all implicit, so they show up as leaves despite not actually being removable.

I did a thing

This afternoon, I graduated from “has not yet owned an Alfa Romeo” to “currently owns an Alfa Romeo”.

Anyone who knows me will tell you I’m a car guy. And any car guy (as opposed to a Porsche guy or a Corvette guy or a Miata guy) will tell you that every car guy ought to have owned an Alfa Romeo. The problem is that in order to have owned an Alfa Romeo, you must first own an Alfa Romeo; and then, presumably, either you get rid of it or it gets rid of you, or you both get rid of each other. And until today, I had not yet owned an Alfa Romeo.

This afternoon, I graduated from “has not yet owned an Alfa Romeo” to “currently owns an Alfa Romeo”. Specifically, a 2007 Alfa Romeo 147 1.6 TS. At some point in the future, I will move on from “currently owns an Alfa Romeo” to “has owned an Alfa Romeo”, thereby completing my car guy journey; but that will take a lot of work and a bit of money, although in theory I should be able to flip it for a decent profit.

Yes, it’s red, and yes, it has little Italian flags all over.

Currently planned work:

  • Install “24h Le Mans” sticker on trunk lid for immediate 5 bhp boost
  • Change all fluids and filters
  • Replace rear wiper blade
  • Polish headlights
  • Investigate lifter tick
  • Replace clutch pump and / or follower and / or hydraulic line
  • Replace passenger side window regulator
  • Replace driver side window and mirror controls
  • Replace trunk lid lock actuator
  • Fix driver’s seat release
  • Fix or replace various bits of interior trim
  • Replace gear shift lever knob, possibly with a shorter one
  • Replace whip antenna with either bee sting or shark fin

Maybe I’ll even remember to post updates.

Update: 5 bhp boost unlocked

Close-up of the trunk lid of a red Alfa Romeo 147 1.6 TS sporting a white decal showing the outline of the Circuit de la Sarthe with the text “24h Le Mans”.

Update: of course my Italian car has electrical gremlins.

Inside view of the raised trunk lid of my Alfa Romeo 147 with the lining removed. The trunk lock is disconnected and black and red multimeter probes are inserted into the plug. I am holding the multimeter in my left hand; it is set to 20 V and the display is showing12.04.

Update 2026-03-16: I had low washer nozzle pressure and assumed the pump might need changing. Only after ordering a replacement pump from a breaker did I realize that the real issue was that the washer hose was getting pinched when I closed the hood. Cutting the cable ties that tied it to the hood strut solved the issue.

View of the front of a red Alfa Romeo 147 from right above / next to the passenger window. The hood is open and there are traces of washer fluid on it. A multimeter is propped up against the hood so that it is visible from the driver's seat. A ratchet is lying on top of the engine, which is visible through the gap between the rear edge of the open hood and the body.

Update 2026-03-19: It passed inspection today, with a slew of minor defects but little I wasn’t already aware of and nothing I can’t fix.

What year is it, NUUG?

The Norwegian Unix User Group is celebrating its 40th anniversary on August 10. I was invited to speak at the event and initially accepted, but after learning of the schedule for the event, which was not communicated to me at the time, I have decided to withdraw.

The Norwegian Unix User Group was founded in 1984, and is celebrating its 40th anniversary on August 10. A short while ago, I was approached by a member of the organizing team (whom I shall not name) and asked to speak at the event as a representative of the Norwegian BSD community. I was given no information about the event beyond what was expected of me (a 30-minute presentation on the topic of my choice) and the names of some of the other speakers. After some hesitation, as I was having a hard time coming up with a topic that I felt would fit the occasion, I initially accepted.

One thing I found odd was that NUUG’s own event calendar (archive) has no information about the event beyond a time and a place. I get that the list of speakers is not yet finalized, even though there’s less than a month to go, but there is literally nothing beyond “more information to come”. All the links just point back to the same page.

Screenshot of NUUG’s event calendar as of 2024-07-12

This morning, someone pointed out to me that there is a registration page (archive) for the event (which, again, NUUG’s own website does not link to or even mention!) with extensive information about the event. And that information… is not good.

Screenshot of a portion of the registration page for NUUG's 40th anniversary celebration, which includes the finale of Miss Norway 2024 and an opportunity to take photos with the contestants.
Screenshot of a portion of the registration page as of 2024-07-12

tl;dr: the event is collocated with the Miss Norway 2024 finals and attendees are promised opportunities to have their pictures taken with the winners and runners-up.

WHAT THE EVERLOVING FUCK, NUUG?

It goes without saying that I will not be speaking at this event. I will try to turn my presentation (which I’d only outlined, not written yet) into a blog post, but no promises. It would have been a reflection on what “Unix” actually means (beyond the simple fact of the trademark) and how that has changed over the years.

If you’re a member (and perhaps even if you’re not), I encourage you to contact NUUG and let them know your thoughts on this matter.

EDIT: I have written to NUUG to ask them to either change their plans for the event, in which case I will be happy to attend and speak, or cancel my membership. I will update this post again when (if) they respond.

EDIT 2024-08-07: I have yet to receive an official response from NUUG. I’ve been told out-of-band that my email has been received and my membership canceled, but I appear to still be on the mailing list. In the meantime, Norwegian magazine kode24 has written a piece on the topic, citing my tweet and this blog post, which kicked up a bit of a fuss on Twitter. Knut Yrvin chose to respond directly to my month-old tweet with a strawman and insults while Malin Bruland rode to his defense with the same strawman. I am not impressed.

DNS over TLS in FreeBSD with Quad9

It has come to my attention that Quad9 have a blog post providing incorrect instructions for how to set up a FreeBSD system to use their service. I have attempted to get in touch with the author and get him to correct it but have received no response. So here, for the benefit of the Great Search Engine Gods, is the correct procedure; see my earlier post on the topic for more details on how it works.

# cat >/etc/rc.conf.d/local_unbound <<EOF
local_unbound_enable="yes"
local_unbound_tls="yes"
local_unbound_forwarders="9.9.9.9@853#dns.quad9.net 149.112.112.112@853#dns.quad9.net 2620:fe::fe@853#dns.quad9.net 2620:fe::9@853#dns.quad9.net"
EOF
# service local_unbound setup
# service local_unbound restart

No need to reboot.

Note that if you only have IPv4, you may experience slightly degraded performance unless you leave out the IPv6 addresses from the local_unbound_forwarders line (and vice versa in the unlikely scenario where you only have IPv6).