What started as a weekend project turned into a full-fledged port FreeBSD is an operating ... ## My Background ... ## Talos Secure Workstation In 2016, I became very concerned about the implications of the Intel ME for owner control of hardware. In April, I read [this message](https://lists.fsfe.org/pipermail/discussion/2016-April/010919.html "this message"), which resonated with me. I immediately started lurking in the [IRC channel](irc://chat.freenode.net/talos-workstation "IRC channel") and reading about the development of a fully-open hardware platform, the [Talos Secure Workstation](https://www.raptorengineering.com/TALOS/prerelease_specs.php "Talos Secure Workstation"). In October 2016, a [CrowdSupply](https://www.crowdsupply.com/raptor-computing-systems/talos-secure-workstation "CrowdSupply") campaign was launched to fund the production of the Talos mainboard. I immediately jumped onboard as one of the early backers. While this campaign was ultimately unsuccessful, I continued hanging out in IRC. ## Talos II Although the Talos Secure Workstation never saw the light of day, the project never truly died. On May 31, 2017, the channel topic was changed to mention "Talos II taking shape -- direct to market, no crowdfunding! Details to follow in coming months..." In July, a [teaser page](https://raptorcs.com/TALOSII/prerelease.php "teaser page") was added, and on August 8, 2017, preorders were officially opened for the Talos II. I didn't immediately notice that the preorders were open, but when I did notice on August 21, 2017, I immediately placed a preoder for a Talos II. Fast forward to April 2018, when my brand new Talos II motherboard arrived and I started playing with it. I had previously mentioned to the FreeBSD developers who hung out in #talos-workstation at the time that I was interested in testing and debugging FreeBSD. Which I did. In October 2018, Matthew Macy suggested I join the FreeBSD side of things over on EFNet IRC. So I did. And immediately met a bunch of familiar faces. At that point, I was fully focused on FreeBSD on POWER. ## Enter POWER9BSD In October 2018, an experimental FreeBSD fork called [POWER9BSD](https://github.com/POWER9BSD "POWER9BSD") was started by [Matthew Macy](https://github.com/mattmacy "Matthew Macy"). At the time, I was mainly assisting in testing and familiarizing myself with kernel internals, while also working on familiarizing myself with the ELFv2 ABI, among other things. POWER9BSD was focused on adding modern features to the FreeBSD kernel. Development ended up stalling after Matthew Macy went on to working on other projects instead, but the features added slowly trickled into the main FreeBSD repository as dependencies became unblocked. Due to its use of kernel IFUNCs, it required a modern toolchain, which meant that it wasn't easily mergable as-is. ## Joining FreeBSD On May 31, 2019, I officially became a [FreeBSD committer](https://www.freebsd.org/news/newsflash.html#event20190531:02 "FreeBSD committer"), working on the PowerPC architectures, after being mentored under Justin Hibbits. ## A new ABI The bulk of the effort that made a Little Endian port possible actually happened throughout 2019, as FreeBSD PowerPC64 migrated from the legacy ELFv1 ABI (Application Binary Interface) to the ELFv2 ABI, as part of a migration from the ancient GCC 4.2.1 compiler to the Clang compiler. This new ABI was mandated by the lack of support for ELFv1 in the lld linker, but there were other reasons to migrate as well. One of the big benefits to ELFv2 is the removal of function descriptors, a concept whereby C function pointers were actually pointers to 24-byte structures containing metadata about how to do a function call rather than the address of the function entry point. These descriptors were a common portability headache with software, as they made function pointers behave differently than all other architectures, including 32 bit powerpc. FreeBSD had some support for ELFv2 in the kernel already, but userland support was lacking due to ELFv1 working "good enough." I put a lot of work into championing and assisting with the ELFv2 transition. My Talos II was the first machine to be fully selfhosted on ELFv2 (in early 2019), and for most of 2019, I was one of the few people experiementing with ELFv2. The demand for ELFv2 became more immediate after continued existence of the powerpc platforms became [contingent on migration away from GCC 4.2.1.](https://lists.freebsd.org/pipermail/freebsd-arch/2019-August/019674.html "contingent on migration away from GCC 4.2.1.") While this deadline ended up being stretched a bit, we ended up successfully [transitioning](https://svnweb.freebsd.org/base/head/?view=revision&revision=356112 "transitioning") during the 2019 holidays. As a side benefit, landing ELFv2 support unblocked merging more of the POWER9BSD features to mainline FreeBSD (such as IFUNC support), and was the last thing blocking a Little Endian port. ## A weekend project While there were no technical issues blocking a Little Endian port, nobody had picked it up and worked on it. I had done some minor experiments on and off that had never gone anywhere regarding Little Endian, but other than knowing that it should be a relatively easy process, I hadn't put much effort into it. The largest effort I had made was [preparing the compiler](https://reviews.llvm.org/D73425 "preparing the compiler") for building for `powerpc64le-unknown-freebsd13.0` (which is the "target triple" describing FreeBSD for PowerPC64LE.) I had written this in January 2020 and then went on to working on other things. Then I got bored. On August 14 2020, I started working on a PowerPC64LE kernel port, mainly to see if I could get it to an interesting state over the weekend, using QEMU as an initial platform due to its debugging facilities. ```Aug 14 12:59:58 apropos of nothing: Aug 14 13:00:05 root@talos:/usr/src # file /usr/obj/usr/src/powerpc.powerpc64le/sys/GENERIC64LE/kernel.full Aug 14 13:00:05 /usr/obj/usr/src/powerpc.powerpc64le/sys/GENERIC64LE/kernel.full: ELF 64-bit LSB pie executable, 64-bit PowerPC or cisco 7500, OpenPOWER ELF V2 ABI, version 1 (FreeBSD), dynamically linked, with debug_info, not stripped Aug 14 13:18:37 Not that it boots or anything, but I have the build system set up Aug 14 13:23:45 Nice! Aug 14 13:30:24 Bdragon28: That's cool! ``` Progress went faster than expected. By August 15, I had fixed enough bugs to finally accomplish [console output](https://drop.rtk0.net/20200816.txt "console output"). This was [followed](https://drop.rtk0.net/20200816_1.txt "followed") in quick succession by getting as far as, by August 17, being [able to mount](https://drop.rtk0.net/20200817_2.txt "able to mount") an empty rootdisk I had prepared. Whoops! Time to build something for the kernel to actually execute! The weekend was over, but I had certainly solved my boredom problem! After quickly making a [minor kernel change](https://svnweb.freebsd.org/base/head/sys/powerpc/include/elf.h?r1=365693&r2=365692&pathrev=365693 "minor kernel change") that I knew ahead of time would be needed to for the kernel to be able to load Little Endian binaries, I started a cross buildworld (easier than it sounds, as FreeBSD has excellent for cross-building the base system!) In parallel to this, I had been testing on and off on physical hardware, and had gotten as far as [showing trash on the console](https://drop.rtk0.net/20200817_3.txt "showing trash on the console") before crashing. I gave up with hardware for the moment and loaded the cross-built base system onto my prepared QEMU rootdisk. To my extreme surprise, it [loaded init](http://drop.rtk0.net/20200817_5.txt "loaded init") the first try. Some slight tweaks to my rootdisk later and I had a [functioning boot to multiuser](https://drop.rtk0.net/20200817_6.txt "functioning boot to multiuser"). Whoops! Time to switch back to fixing boot on hardware! By the 20th, I had gotten the console issue [fixed ](https://drop.rtk0.net/20200821.txt "fixed ")and was well into fixing issues further in. Bugfixes continued on the [21st](https://drop.rtk0.net/20200821.txt "21st") and [22nd](https://drop.rtk0.net/20200822.txt "22nd"), until finally, late on the 22nd, I had a [functioning kernel](https://drop.rtk0.net/20200822_1.txt "functioning kernel") on hardware. At that point, I had a platform that was working well enough to self-host (i.e. I was able to build and install FreeBSD from within itself), and I had a machine that I could use to continue development. While it ended up taking more than a weekend, I had gone from zero to self-hosting in a matter of days. On August 26, I had most of the remaining bugs fixed, and had generated full installation media and provided it to the other people on the FreeBSD PowerPC team to experiment with. ## Getting serious ... (discuss poudriere and toolchain stuff here) ## Primetime Late on September 22, 2020, I committed my work to the FreeBSD repository, making PowerPC64LE a new FreeBSD architecture. ## The Future Given how in flux things are at this point, FreeBSD 13 will launch with PowerPC64LE being considered an "experimental" platform. ## Final thoughts ...