【Community Conversation May 2024】Part 1 Immutable - ZimaOS Technical Framework
Last week, our CEO Lauren and CTO Tiger had an engaging fireside chat with a community developer Axel. Throughout the conversation, we uncovered the innovative thinking that drives the ZimaOS team, from their visionary approach to hardware and software design to their meticulous selection of foundational technologies, and discussed the progress of AI development within our products. We explored a wide range of topics including immutable architecture, AI integration, containerization, virtualization, the app store ecosystem, and gained valuable insights from Axel’s firsthand experience with Zima hardware.
We divided this conversation into four distinct themes. In this episode, we’ll delve into the captivating topic of immutable architecture.
- Technical considerations about immutability
- Acceleration by Home Assistant
- Rebasing in ZimaOS
- Fallback and rollback strategies
Video version is at the end of this post
Axel
So my first question would be, it was a surprise to me that after 25 years I switched actually back to Linux after using MacOS, and I’m using also immutable operating systems like for instance Blue Fin or I used to do to work on Vanilla OS before, and you told me on discord that actually ZimaOS is immutable as well. And I thought like maybe you can give me a little bit more context on why ZimaOS is immutable and what you’re using for that instance.
Tiger
That’s a great question to start off this conversation. So ZimaOS is a Builtroot based embedded Linux to be super accerated. The nature of Bulitroot is it pre compiles whatever dependent packages into a Squash FS image. Squash FS, to certain extent, is basically a format where everything is packed into a file and then just simply mounted to the Linux and it’s not writable. It’s read only. So this came from the very beginning, a background about update mechanism of CasaOS. It was not pleasant for people to upgrade CasaOS because packages needs to be downloaded, needs to be extracted somewhere, both on needs to be cleaned up, you know, here and there, all kinds of issues.
And then we started to explore solutions and we first look at LibreELEC. Basically, I think that’s the operating system for Retro Arch or, well, for Kodi, I forgot what exactly was the platform with that, but it was Bilutroot based. And we find out it’s awesome to swap up entire system by just a file, where you don’t have to worry about where to expect files. And things just work, like whatever file system structure you put into the image, it gets, you know, restored exactly the same. And we find that much easier in terms of OTA over the year update, we find that super interesting. So the initial idea was to turn CasaOS into a Bilutroot based because of what I just said.
And then the opportunity came. We feel we need a operating system specific for ZimaCube. And so we took advantage that we werelooking at the new OTA mechanism, and then just changed the CasaOS, which is Bilutroot based over to ZimaOS. And of course it’s squash FS, it’s immutable, and it allows a much simpler installation. And user files simply needs to be overlaid on top of squash FS so that it’s writable. So it has a super clear boundary between what’s immutable, what’s mutable. That’s great because, you know, we know that the user is constrained from making modification to the base system. So whatever issue that comes up, it’s easier for us to trace down, you know, where the issue comes. It’s much easier for us to reproduce problem. Simply took the user file from a testing environment, like anything assuming there’s nothing confidential in the user partition, we just, you know, take that over and then it’s easier for us to reproduce the problem. Those are just some benefits I could think of. There’re many other benefits. Like you use immutable OS a lot Like you know, I don’t think I need to go further, but that’s the very technical wise brief of ZimaOS.
Axel
I’m always surprised because when we talk about immutable Oasis, I remember it might have been even 8,9 years ago, around Core OS when it was still independent, and that was before I think even rated of the Fedora Foundation actually jumped on it. But it was such a new concept. And then I didn’t hear about it for quite a couple of years. So yeah, is it that you use SquashFS because of the embedded architecture that you had before with ZimaBlade or what actually make you thought about that? Because it has to be I think like two years already that you took that decision to for an immutable OS or at least two years.
Tiger
Yes, I like to clarify that immutability was not the main purpose we went after. It was a side benefit and not become the most important benefit. The initial thought was simply, you know, something that’s easy to be upgraded or over the year OTA. We actually looked at, other than Billroot, we looked at round to snap. I’m not sure if you’re familiar with that. The Ubantu, Linux OS, they have an version of want to cut open to core. We looked at that as well.
But due to the nature that it is super commercial and there’s a key and then we, our nature is open source, so it doesn’t really work together. We also looked at… there’s a few roots like starting with letter y, I just all of a sudden I forgot the name of it. But it’s very similar to Bear Root. It’s claimed to be much more flexible than Builtroot. And, but the thing is, we find the learning curve is kind of steep, but the both team is a young team…I’m old, but the Russian team is very young. But no one heard OS before. So we figured it’s gotta be something easier to hands on, to start with.
Luckily, there was a great opportunity. It’s a project called Home Assistant OS. Yeah, if you go to GitHubu on Home Assistant, you’ll see they have a project called operating system, right? So we studied that project. The good thing is the project is under an open source license. I forgot it’s Apache 2 or MIT, but it’s compatible with, you know, our needs. So we took that as our starting point. We studied deeply, we took months study how it was built, constructed. It’s a Builtroot based operating system. The repository of the ZimaOS is private, but there’s a line at the very bottom of the read me. It says, ‘this is a lot of credis goes to the home assistant as we took’. It was a great acceleration to our building process.
Axel
I didn’t expect Home Assistant to be the influence there, but it’s very interesting to hear.
Tiger
Yeah, that’s a bit of history over there.
Axel
And we talked a little bit. You said like overview air updates were one of the key reasons why you have chosen that. And of course, I suppose rollbacks are therefore not an issue we talked a little bit about. On the other hand side, about the principles of rebasing. Because as I stumbled across Blue Fin or Bazzite for gaming, I mean, you basically have like Fedora Silverblue at the bottom, but then you can rebase actually towards different configurations. On top of that, is that also something that you monitor as well for maybe a future roadmap or what’s your opinion on that?
Tiger
So I looked at the term rebasing from the project you mentioned, and we actually have already implemented under, we use a a framework called RAUC. It’s a upgrade mechanism for Builtroot, maybe for other projects. But I think it integrates super well with Bilutroot . With RAUC, it is able to divide entire system into partition A and partition B. And basically you mount your squash FS to one of it. And then we, if you have it, in version 1.0. And on partition A, if you have 1.1, it will be installed on partition B. And then, you know, keep switching back and forth. There’s a data partition that sitting somewhere along, you know, the same desk and it’s getting minded to the right location after the partition is booted, is loaded. So I think that’s very similar to the rebasing. Basically, all user files there are stored in that data location. And it has nothing to do with the base petition. And every upgrade, it will depend on the current partition, it will dump the new image to the other partition.
And it’s interesting you talk about rollback. Our mechanic, we have a mechanism. Actually, it’s not our mechanism. We’re just using the feature of RAUC. If there’s anything around with the initialization process or the upgrade process, you can mark the current boot failed, which is unfortunate. But the fortunate thing is it will rollback to the other partition, and then as a rollback. So basically if you install ZimaOS as behind the scene there simply just two partitions it as a fail safe mechanism. I think that should be very similar to the rollback and rebasing mechanism you’re talking about.
Axel
Since you told me that in comparison to Zima Blade, like ZimaOS s and Zima Cube is more focused for normal users. I would say not for hackers and so on. Yeah, so it’s said that fallback or rollback in that case, is that more or less happening automatically or does the user have to do anything? Is there some sort of detection? Because it sounds really interesting how you do that. Hum.
Tiger
We build UIs around all these. We have a UI which, well, the user won’t be able to know. Like, what’s the mechanism behind this thing is simply just like an upgrade. It’s like an upgraded UI in your router, I don’t know, your phone? People just hit on update and it just happens it takes care automatically and it just happens. But if you’re a tinker, you can, of course, go to the command line. We actually provide a nightly built at the current stage. By the way, thankfully there is a great feedback from our community to, you know, asking about nightly built and people can actually download the nightly built in RAUC format and then run the command line.
It’s like few seconds and then automatically goes to the later version. And then on the book loader screen, it can actually select which petition wanna go to. Like if I don’t like this nightly built , I can always go to the previous nightly built and then vote for the new nightly built, and then apply it again. You can always do that. Yeah, but to, you know, simply answer your question, yes, we build UIs around the entire OTA mechanism. So the consumer user, average Joe users, they don’t have to worry about the command line stuff.
