Using the new bridges of FreeBSD 15
blog.feld.me102 points by vermaden 16 hours ago
102 points by vermaden 16 hours ago
When this settles down, I look forward to all of jail/iojail, Sylve, Bastille, Bhyve documenting this in a mutually consistent manner. As it stands, I have managed to completely knot my brain over the abstractions, what is happening. It's me, not the systems, but I think there is a little bit of "meh, I understand it, so it must be obvious to anyone smart" going on, and alas, I am not smart, and I get confused easily.
I'm in bastille atm, but have been in all of them and TrueNAS core. and libvirt over on the other unix.
Honestly, best thing I did was ditch all that and just read the handbook, specificially chapter 17 [1]. All of my jails are now set up manually, initially using /etc/jail.conf, and now individual jail configs in /etc/jail.conf.d/.
I still use vm-bhyve [2] for my Bhyve virtual machines, but that's been rock solid for me for years.
From TFA:
They have also soft-deprecated the ability to have any layer 3 addresses on member interfaces which makes it behave like a real hardware switch. The net.link.bridge.member_ifaddrs sysctl controls this behavior and it will be removed in FreeBSD 16.0-RELEASE, same as if set to zero.
If that's right, how does that interact with things like Linux's veth pairs? [0] Can the half of the pair that's not a member of the bridge have an IP address?
[0] I assume something like that exists in FreeBSD-land.
Bhyve bridges are inefficient: every packet traverses NIC → CPU → bridge → VM, adding unnecessary copies that kill throughput. Switching to SR-IOV eliminated that overhead and I saturated the 10 GbE link.
I do not see how that follows. Memory bandwidth is measured in the hundreds of Gb/s. You can issue tens of unnecessary full memory copies before you bottleneck at a paltry 10 Gb/s.
It is much more likely there is something else terribly wrong in a network stack if it can not even drive a measly 10 Gb/s.
That assumes memory bandwidth is the issue, and not latency and/or CPU.
My stupid Zen 3 Frankenrouter absolutely saturates both directions of a 10Gbit symmetric link, and it's using Linux software bridges, software firewalling, and software routing. At ~400usec at idle, latency is low, but twice that of a host system that has no software bridges. [0]
Some tiny, underpowered ARM box wouldn't have the power to do all that in software, but you're not going to be running VMs on a tiny, underpowered ARM box.
[0] However, the fully-loaded latency is far better than the system with no software bridges; ~1200usec vs ~7200usec. One might conclude that factors other than the software bridges, firewalls, and routing are the significant components of the latency figures.
I for one welcome and applaud any progress on the bsd front,and this seems to be huge.
> -tso4 -tso6 -vlanhwfilter -vlanmtu -vlanhwtso -vlanhwtag -vlanhwcsum -lro
Whys the author disabling tso and lro? Whats the motivation?
I'm not familiar with the other flags.
People found this worked in the past and it gets copied around. There is no reason to disable some of this. Bridge will automatically disable LRO and find the common set of other offloads. TSO is not useful for a bridged guest.
LRO because the bridge has to forward the real frames. TSO because it’s fairly useless now.
Why sudden surge of FreeBSD-related posts?
Did anything special or new happened on FreeBSD land?
I think people are looking for new alternatives to tinker with. Linux is becoming new Windows and BSDs new Linux. I dunno what is Windows becoming, but it ain't good.
15.0 was released a couple months ago, hence the title.
We have three (including this) FreeBSD posts in the past two days.
Back to FreeBSD: Part 1 (hypha.pub) https://news.ycombinator.com/item?id=47108989
Linuxulator on FreeBSD Feels Like Magic (hayzam.com) https://news.ycombinator.com/item?id=47113527
Generally people get more excited any time a major release of anything comes out. But FWIW HN has always had favorable front paging for anything related to FreeBSD and OpenBSD.
Not disagreeing, but if the release was a few months ago, then the poster is quite correct - there is a recent "surge" of FreeBSD related posts. And these are not quite ... how shall I word it somewhat nicely ... not quite as fascinating to, say, a wider linux community as such. With that I don't mean "because we use linux, we are snobs", but that what the FreeBSD guys talk about, seems a little bit ... outdated. The heavy use of shell scripts for instance in this post here - I never understood that focus on shell scripts in general, including on Linux. I transitioned into using ruby (or python, but mostly ruby) to replace all shell script needs a long time ago. Every time I am assumed to have to write a shell script I wonder why I would want to cripple myself when I can use a better programming language instead. Many of these shown "innovations" are also not really groundbreaking. To me it seems as if there is a distinct lack of FreeBSD users out there compared to Linux users. As a consequence Linux simply has a lot more information and news (a lot of which is also low quality of course; I am not saying it is all pancakes and sunflowers in the Linux ecosystem either).
> The heavy use of shell scripts for instance in this post here...
There's exactly one in the post. It's ten non-blank/non-comment lines, and the author says of it
This is not well designed but it gets the job done.
When my sysadmin programs get more complicated, I reach for something more suited (like Erlang), but if the shell script is simple and only has deps on other standalone programs, then I write a shell program.
No conspiracy, I think it just happens. One person posts something, maybe someone else reads it and gets into a rabbit hole on a topic, or maybe someone sees an opportunity to throw more conversation pieces at something hot.
Perhaps the initial posts spurred reader interest in FreeBSD which then spurred further posts?
FreeBSD is great - good to see it get positive "airtime."