 |
| PSI Raining on the CPU Scheduler |
As I began familiarizing myself with various Yocto builds, I watched a few Yocto tutorials on optimizing compilation times, and came across a 2022 video "making bitbake Psi-chic" and other tutorials to enable PSI at GRUB.conf for Yocto builds (2022 Slides).
 |
| from p.7&8 of the 2022 Slides & /layers/bitbake/lib/bb/runqueue.py (it appears the BB_PRESSURE_MAX_{CPU|IO|MEMORY} was not found in local.conf) |
However, the feature was added only in kernel versions 4.2 through 5.15, where a proc/pressure folder exists. According to this presentation, a new process needs to be used to enable it for kernel versions after that (e.g. 5.20+). Some distros do not even compile it so you have to check which kernel and distro version you have.
(Edit: 7-12-26: Some 2023 forums confirm it was removed after kernel 5.15, and it needs to be recompiled with a distro if someone wants to use it. Also, it confirms Arch and Zen kernel use it (e.g. Garuda might still use it):
https://techpatterns.com/forums/about3057.html
This post continues from my previous interest in reducing compiling times, and around the same time of browsing the Yocto channel on Youtube, found another very interesting video when I stumbled upon a link to a website called Compiler Explorer, posted on a superuser/stackexchange forum comparing performance levels of GCC and LLVM compilations, provides a branch prediction 101 crash course on CPU processing, but also covers edge cases where sorted, unsorted Python, and low-level languages can reveal significant limits (or optimizations) of out-of-order processors (and hence why to use them-and me thinking about the relative performance gains my Acer netbook with a C-50 has over my Atom N450 netbook.) But more on that later - the Compiler Explorer, that is.
This morning I also noticed a pattern in my knowledge recollection of distros i have seen and heard of. Basically, it goes like this: I read about something that I find very interesting, and spend a few hours or days going down a rabbit hole exploring use cases on whether or not I would actually need or want to use something. So a story I read about in 2013 actually has relevance to this current effort. Since I had been an avid/habitual reader of Slashdot from the years of 2006-2013 or so, I recall my first introduction to a news story mentioning anything resembling personalized kernel scheduling- e.g. pre-empting kernel threads to run userspace more according to your actual syscalls, rather than whatever's running in the background, when Slashdot posted an article, circa 2013 or possibly earlier on DragonFlyBSD:
 |
How I imagined a 2004 FreeBSD Board meeting
|
"Why did the Linux developers use ANYTHING from the gnu project? Tecnical debt is what that is. They weaned themselves off the minix filesystem but not off the gnu compiler, and were crazy enough to let Ulrich Drepper switch them from their own libc5 to glibc because he'd sprayed it down with thread locking back when Java fundamentally could not run without threads because Sun had invented threads to overcome their terrible process switching performance and were SO PROUD OF IT that they never added poll/select bindings to Java... Sigh.)"
I didn't bother to install DragonflyBSD in 2013 because the performance improvement cases were not yet significant or clear enough, but there have been several stories of other distros implementing smarter kernel schedulers, such as gaming distros. A couple years ago, I was reading forums where Steam users were discussing which linux OS they'd like to migrate to after Windows 10 support ends, and one of them mentioned Garuda. So I realize that scheduling has a general purpose benefit, and not just edge cases for things like servers running IllumOS or Kubernetes.
Garuda uses 5.15 LTS "Zen," and while this is an older kernel, may have some features that are optimized for older systems.
"Garuda OS, commonly known as Garuda Linux, uses the linux-zen kernel as its default. This modified kernel is heavily optimized for desktop responsiveness and gaming, featuring lower latency, higher tick rates, and better frame pacing. [1, 2, 3] "Other kernel schedulers include BORE, which is in another Arch-based distro, CachyOS. Less than 2 months ago, Phoronix reported that Intel is also working on improving scheduling for old systems, improving minimal frames per second in games called "flattening the pick." Thus I recognize a pattern that I've overlooked for quite a while. There are CPU performance improvements that I could get from my CPU/system even without overclocking, but the current configuration, even though not a "one-size-fits-all," is still an overall well-rounded choice without the time investment to actually root out the cause of the stalling (in my case 99% utilization on 8 threads may or may not have been optimum, and running new compilations for 5 hours might be optimized with a few tweaks-maybe I can cut 1-2 hrs from a build if I'm really lucky).
Before I began this rabbit hole a few days ago, I thought, I am just learning to compile a few distros like core-minimal, Sato and a few other embedded distros. Why spend all this time optimizing for PSI when I am not even familiar with customizing a distro very much? Well, I already waited quite a few hours on several builds, some which were successful, but stock images. Why not learn how to optimize Yocto for PSI? So that has become my new goal. This could be in the format of compiling a linux distro like Ubuntu or Rocky Linux with PSI enabled in GRUB, and then Yocto preinstalled so it can check for that and adjust its builds based on expected tasks.
On a much more basic level, however, my first goal is trying to figure out if it's even enabled in Yocto 5 & 6.0. It's possible Wrynose already implements it, but the process isn't visible to the user. I see mentions of it from 2023 but some users experienced issues like OOM, so it's unclear if they removed it.It's not clear if I am unable to check the info even if Bitbake has direct access to it and adjusts its builds accordingly.
I recall reading one of the server blogs that a later kernel or software added a feature where applications can communicate with the PSI dynamically. Here are some of the blogs I found that did benchmarks using Prometheus and other pages mentioning implementation decisions:
https://github.com/bottlerocket-os/bottlerocket/issues/3517#issuecomment-1772448856
https://kubernetes.io/blog/2026/05/12/kubernetes-v1-36-psi-metrics-ga/
https://wrightrocket.blogspot.com/2019/08/linux-performance-linux-kernel-42.html
https://unixism.net/2019/08/linux-pressure-stall-information-psi-by-example/
https://www.lacy.ie/technology/2023/10/22/pressure-stall-information.html <--developed load shedding using PSI as a way to prevent outages like the one that happened at Amazon in 2018.
https://facebookmicrosites.github.io/psi/docs/overview
"PSI provides for the first time a canonical way to see resource pressure increases as they develop, with new pressure metrics for three major resources—memory, CPU, and IO.
These pressure metrics, in conjunction with cgroup2 and other kernel and userspace tools described below, allow you to detect resource shortages while they’re developing, and respond intelligently—by pausing or killing non-essentials, reallocating memory in the system, load shedding, or other actions.
PSI stats are like barometers that provide fair warning of impending resource shortages, enabling you to take more proactive, granular, and nuanced steps when resources start becoming scarce."
https://zhuanlan.zhihu.com/p/523716850 (I used Google Translate)
https://docs.rockylinux.org/9/gemstones/core/view_kernel_conf/ (not PSI related but info for my Rocky build)
^As you can see, a lot of the use cases of PSI have to do with large servers at places like Amazon, Meta, and personal hosting on a budget. Seeing more specialized optimizations for Yocto would be interesting, such as a musl recipe, and the benchmarks all seem to point to improved gains.
Other notes: I can see the potential use of systemd. I just prefer to learn the hard way and look for a reason to use it or possibly see if it can help debug an issue, kind of like Compiler Explorer. Otherwise, it could also get in the way, like SELinux could. That said, I am testing out that in Rocky Linux by compiling another SELinux focused OS, called Sulka. I probably could test out a different Yocto build, especially since I've wanted to test my Galileo board, but since I ran into so many issues with the bootloader when trying to flash Debian, I may as well compile a distro for my Raspberry Pi. I could also just run VirtualBox or Qemu to test it (technically Sato already satisfied that thirst, but I'm not too fond of using Qemu unless I really need to- the latency seems higher than other VMs, and I don't feel the need to set up RDP on my own PC/LAN just to test a distro, plus I prefer VirtualBox, which I used in the days before it was acquired by Oracle, so I know it was*pure* in*heart* and* may* still* be*-ok*I*know*that*might*not*sound*funny*).
 |
Today is 7-11 |
No comments:
Post a Comment