> Yes, but we caution users to evaluate if you really need kubernetes. Chances are you don't and you will experience severe performance and security problems if you choose to run under k8s. If you still find you must here are instructions for running Nanos under k8s.
I can speak to this. Containers, and by extension k8s, break a well known security boundary that has existed for a very long time - whether you are using a real (hardware) server or a virtual machine on the cloud if you pop that instance/server generally speaking you only have access to that server. Yeh, you might find a db config with connection details if you landed on say a web app host but in general you still have to work to start popping the next N servers.
> Yes, but we caution users to evaluate if you really need kubernetes. Chances are you don't
That's an odd, perhaps presumptuous, claim to make considering this isn't even in the same orchestration/scheduling space as Kubernetes. Especially without mentioning alternatives.
From later in the FAQ:
> The complexity that comes with kubernetes is that it requires you to re-invent all the layers of a cloud platform that already exists. If you run a vanilla linux instance on AWS you get out of the box: networking, storage, security, routing, etc all for free.
They might not like the abstractions, but to say you have to "re-invent" all those layers is reaching.
I believe kubernetes is a little misunderstood- kubernetes is not software, and is not an abstraction on top of cloud. It’s an attempt at creating a standard-ish API for cloud.
It’s currently being implemented as an abstraction on top of existing systems, because cloud providers have to leverage what they have and because its API still lacks all the functionality. But gradually it became a managed service that can provision databases, etc, and in another 5-10 years most cloud interactions will happen through it
I guess the main issue is that with Kubernetes, and chiseled containers we are quite close to the same idea, even if a full kernel might be underneath, and thus it is quite hard to gain adoption.
I joke that in the end Andrew Tanenbaum got the upper hand over Linus, because the supposed performance benefit of a monolithic kernel over microkernels is meaningless in a Kubernetes cluster, or distributions like SUSE Linux Enterprise Micro and Fedora CoreOS.
I certainly don't speak for the nanos project, but in my mind this was a key reason for even doing this work. That we could write an adaption layer that would hoist the world as we know it off the sandpile and put it someplace else where things made more sense.
its a shipping container, an adaption jig, an invitation to rethink the world underneath without insisting that anyone rewrite anything.
This seems designed to run inside virtual machines so there's a similar flavour. But I guess if you are running containers inside VMs you could substract one layer by reimplementing your application into a OS component.
A unikernel application is a VM. It's like a container, but without needing docker + linux to run on.
I was running a web application written in ruby, distributed in a container, running in docker on linux in a VM. That could become a unikernel running directly instead of the VM. Saves quite some layers i'd say :)
What do you mean by "directly"? I'm not familiar with Nanos internals, but after skimming their FAQ[0] it seems that Nanos is kind of VM that can't run on bare metal and still requires hypervisor (presumably Linux, unless your CTO plays a lot of golf with MS salespeople):
Nanos is a single process operating system designed to run as a virtual machine and has no support to run on hardware.
It seems you are right. The OPS documentation mentions that you can deploy it on bare metal, but this is a recent comment [0] that contradicts that .
right now we don't have any plans to support bare metal
installs like this as that would imply a bunch of other
mgmt related tooling that would not be present
(eg: start/stop the server, configure networking,
deploy a new one, access rights, etc.) it also breaks
the assumptions we have that it is only being deployed
as a vm which means having to support a ton of random
hardware drivers, nanos is intended to always be ran on
top of a hypervisor of some kind - whether it's public
cloud or something under your own control
(eg: proxmox/vsphere/etc.)
It seems like they make some distinction between true bare metal and somewhat bare metal, which is highly confusing.
It's bare metal in the sense that it's self-bootstrapping but the "metal" it supports is only a paravirtualized system. This is what they mean when they say that they don't want to support tons of random hardware drivers: they've written support for KVM paravirt devices (which are nearly universally available on VMs), and that allows the kernel to run on most hosting providers.
Unikernels and virtualization are orthogonal, you can run eg on-prem appliances with unikernels on bare iron. Eg sounds like Netapp ONTAP is/was like this at some point.
I like that I can get more performance, but I don't necessarily want an anemic environment for when things go wrong. Production machines are often pretty barebones to begin with - making investigating issues often feel like working through a key-hole.
I'd be curious how much performance this might bring vs, say, a Linux kernel with an application loaded direct from initramfs. It's not terribly complicated to build a single-purpose initramfs.
There's a lot that an operating system brings to the table, that casual users may not be aware of. They tend to have a bunch in them not because "why not" but because it's providing value.
It's strange to me that Alpine Linux won over Unikernels. I guess it makes sense, once you add logging, observability, printf-debugging (just kidding) to your unikernel, you basically have a full OS anyway.
It's just familiarity and path-dependence. People expect everything to work like it does on their MacBook and if there's a bit of overhead to doing that, oh well.
> logging, observability, printf-debugging (just kidding) to your unikernel, you basically have a full OS anyway.
I mean -- isn't that all the good stuff?
Like -- I boot this thing and I don't really have a filesystem in the way we know it. First of all, that's confusing (I say this as a ZFS devotee). Second, I have no idea why this is acting crazy in production, or in testing, compared to the Linux version? Etc., etc.
> I don't really have a filesystem in the way we know it.
This is a rather common misconception. Nanos (and many other unikernel projects) have filesystems. Most of the applications we target are webapp servers, databases, etc. They all want to, at the very least, write to a tmp file and more commonly want to load lots of files. What they don't have is an interactive userland.
Logging and observability are valuable. But running a full multiuser OS with kernel and userland for your one process, adding extra context switches and what have you to everything you do, just for the rare occasions where you log in and run a few commands, seems crazy to me. As long as it can output logs/traces/etc. to a collector (which is what I'd want to do anyway, no-one wants to have to SSH to each separate instance to look at log files on the local filesystem) and there's a way to attach a debugger (e.g. the Java style where your debugger connects to it on a given port), I don't see any advantage from having e.g. a filesystem per se. Likewise being able to run it locally the same way as production is important, but that doesn't have to mean running it as a Linux process - people are happy running Docker images for local dev, running a unikernel in a VM isn't a lot different from that.
It's not in the same category, Alpine is just a regular Linux distribution on the userspace side, and you have a traditional big generic kernel with your app running in userspace.
Well docker based stuff seems to have a lot more of an ecosystem on the orchestration side of things.
If I want to have a bunch of instances scaled up and wired together network wise and defined in a nice yaml file, kubernetes makes this easy and replicable and I just have to push a container image somewhere like GHCR and have it all rolled out with CI.
You can deploy almost the same system on top of the managed kubernetes offerings from any of the big cloud providers, so other than working out a few differences generally your config will work on anything.
If I want to do this with unikernels I'm lost, would I do it with AWS services like EC2 and auto scaling groups or something? How do I get it all wired together and configured? Perhaps terraform but I'm not sure.
It's all about the maturity of the glue and the architectures, not to mention training and familiarity. Does (mystery unikernels solution) have the equivalent of custom controllers somewhere in that stack?
K8s provides an abstraction layer over resources and lets you define your system holistically.
If this exists in a mature and flexible state for unikernels I'm all ears.
What real advantage is there to using unikernels over alpine linux + k8s (or whatever) orchestration?
> If I want to do this with unikernels I'm lost, would I do it with AWS services like EC2 and auto scaling groups or something?
This is a key point that many many people struggle with until they actually push their unikernels out to the cloud. It is one thing to run ops locally but try doing a 'ops image create -t aws' followed by 'ops instance create -t aws' and then you'll understand how we manage to push a lot of the scheduling work back onto the cloud provider itself.
Yeah I've popped my image onto AWS and got it running in an instance, I figure the next port of call is the terraform part of the docs to see how you'd put something out in production.
It would be cool to write some sort of book like, ops for k8s people, where it takes a lot of the common concepts in k8s and explains how you'd do them with this system.
ops seems like decent a CLI to get things done but as far as I can see not really an orchestration system in the same way as k8s? Perhaps I misunderstood.
You are correct that ops does not provide any sort of orchestration framework today, but it's also not necessarily needed in a lot of cases because the end artifacts are vms not containers.
There is a terraform provider that we ended up writing for one company but only because they were a (large) hashicorp shop and nothing went to prod unless it was in tf: https://docs.ops.city/ops/terraform This doesn't really do much though. It's only there to check a box and fit into an existing paradigm.
A lot of the orchestration bits that k8s provides is because when it was initially adopted there wasn't a great way to attach networks/storage/etc. to various containers like people had for on the clouds with vms. OPS just re-uses the same primitives you'l find on every single cloud out there. If you need more advanced functionality like auto-scaling that also can utilize the clouds ASGs.
Yeah I guess a lot of the orchestration can be effectively handled by terraform, I get what you mean regarding cloud primitives. This is my next step to seeing how this can work in a production environment. Instead of an ingress you can use ELB, for instance.
I guess this means your environment is very "native" to a specific cloud however and not as easily transferrable.
One thing I did like about k8s is that it's easy to self-host with microk8s but tbh, I think in production you'd be using a cloud provider's managed k8s (e.g. AKS) anyway.
Looking at how to attach/configure a ElastiCache, RDS, or some EFS to a project also.
What's the best way to runtime configure an image when you create it as an instance? I couldn't see any options on ops instance create for environment variables or anything like that but this also seems like, weird - IE you wouldn't give a VM environment variables, but with a normal EC2 instance there's an agent that handles this sort of thing. Would you have to bake the configuration into the image? Also how does one output the ELF for the image? Is it just the file of [image name] in ~/.ops/images ?
Or is it the case that terraform should build a correctly pre-configured image as part of the apply process?w
Always good to have other options, of course! I do appreciate the minimalism. This is exciting :)
The terraform docs example uses GCP, perhaps if I figure out how to do it on AWS I could PR the docs with an example.
At that stage where my questions lead to more questions.
Ian has been at this for years. He's incredibly smart and persistent. It's not gained a lot of traction from my take on it, but I keep a close eye on it because ultimately I think he has the right idea. Time will tell.
Forgot to mention this but https://nanos.org is also related with https://nanovms.com (to deploy unikernels) and ops.city (which handles the package distributions), so it's like a whole ecosystem.
Some things in [1] might provide answers. To be fair, from a cursory look at [2], many of the concerns from [1] appear to be addressed--or at least works in progress--in [2]. I also enjoyed [3] on this topic.
After some brief research, I haven't found any notable emphasis on formal verification in any well-known unikernel (imperfectly defined by online searching, LLM exploration, and focusing on languages like Haskell and Rust that tend to more interested in correctness). Did I miss anything? Even the Wikipedia pages lacks any mention of verification.
The closest I've found was a Reddit article titled "Stardust Oxide: I wrote a unikernel in Rust for my bachelors dissertation" with this comment "My impression, after investigating Singularity for my own undergrad: No. They did bytecode varification against a manifest after codegen, in their own intermediate language that all installed programs had to use. For the Rust compiler I imagine you're thinking of doing certification against the module sources? That's not going to provide the same level of verification."
"As part of our mission to build robust and secure systems, we strive to support technology that roots its design in the field of formal methods"
has a TCP stack manually derived from a formal model
https://tarides.com/blog/2024-01-24-mirageos-designing-a-mor...
Thanks. / My interest is centered around AI safety. It seems clear that any learning system* that can improve its own code must be run in a provably secure virtual environment (at the very least).
* by learning system I include any system that learns from experience
And I am not sure what "thing" I am using. Is there some disambiguation? I know is OPS is the orchestration CLI, but I am confused at the difference between Nanos and NanoVMs. What should I call the section of my README that deals with this tech? Currently gone with Nanos/OPS but I am confused.
I experimented a little with NanoVMs. I wanted to deploy them in AWS ARM64 instances to test if it was possible. It took a bit of effort, but I think it's worth it.
There is something nice about them as your mental map is reduced to the minimum amount of components, your app - amalgamated with the unikernel - and the virtualization platform of your choice.
I need to test support for volumes, and if there is interest I might do a write-up about it.
Say I have a binary “helloctl” that I want to package, where do I read about how to do this?
Took a look at the docs and it is neat with details about nanos but no info about how to use my own binaries, except install ops and run the sample nodejs app.
I could of course install ops and try to look around for other pkg commands but a basic “get started” should show me how to package my own stuff. Or may be I am just too lazy :)
Would unikernels be feasible for online judge platforms (e.g. codeforces, leetcode)? I was thinking something along the lines of: spawning a unikernel for each submission -> running a single file program -> streaming the output back to a server, all in <3s and potentially with multiple submissions running in parallel. I'm fairly new to this, so I'm not sure if that would be overkill.
Online judge platform do a very small subset of problem.
You can sandbox it to no network or filesystem accesses, and no syscalls except the few like read/write/select.
Yes, if your goal is to use unikernels. But if your goal is to pick the best solution for this application, there isn't an obvious advantage for unikernels.
The last time I tried, it was pretty slow to fire up, especially when I tried to run multiple submissions at once. It's also even more overkill for my use case; I currently use isolate (https://github.com/ioi/isolate) which is just a wrapper around cgroups/namespaces, and it's been a lot faster.
Sidenote: I'm not really looking to replace it, I was just asking out of curiosity since this is my first time hearing of unikernels
"real-time" when it comes to kernels doesn't necessarily imply "fast" and Nanos only targets virtualized environments as well. RT is more for situations like "If I hit the brakes I want the car to stop". Nanos' sweet spot is more for cloud workloads.
Having said that with enough interest I think we could look at project such as https://projectacrn.org/ but it's not really a focus at the moment and would probably become a 'flavor'.
nanos's small size and linux binary compatibility make it potentially a great candidate for
- pedagogy
- systems language research
- CPU architecture research
- embedded systems development where a linux process model is useful
- OS research (assuming anyone still does that)
- systems with high performance or low variance needs
- HPC
I thought I was going to do is research. There was a consensus at the time in the community and at least one famous white paper that I can’t find now (pike? rashid?) that explained it was dead. I kind of agree and kind of don’t. I think we have more exploring to do on what the right structure is, but I’m not sure I’d call it research. I think we really missed an opportunity to lay down a new foundation with cloud services. Distribution and security are still two pretty massive holes, and people are looking at them.
An 'official' community space is a blessing, I think Zig took the wrong direction here,
I can understand why project maintainers don't want to carry the burden of gardening a space, but Zig is not doing itself any favors here. Instead, I would suggest maintainers to open a "vacancy" for an accessible and privacy friendly space, and if it fits the criteria, bless it. Or just start one yourself and hand it over.
Why? From a quick glance, most of the Zig communities are on discord. That means all folks that value privacy would not participate. And even if you are ok to lose out on that demographic, you still have a highly fractured information space. Although in the case of discord, "information space" is too generous. "Information black hole" is a more apt term, because none of it gets indexed by search engines. One cannot do without a bit of leadership to make a project successful.
No, they aren't. They are silos, with no proper libre/open source clients with propietary protocols. Also, the Discord company can send any of your data to /dev/null anytime.
This looks interesting! One thing I like about docker is being able to inherit from other images, so for instance in their webserver example, I could inherit from a simple webserver Dockerfile and then my local code would be only the static files I wish to serve.
More like being able to extend existing "recipes" for a unikernel, like you can with Dockerfiles. So you can have a Dockerfile that creates a general solution (e.g. nginx) and extend it to provide your exact implementation (nginx + your public files copied over into the right place).
The way this works is that you take a package (like nginx or node or what-have-you) and then you use any of the configuration vars like 'File' or 'Dirs' or 'MapDirs' and the like to create the filesystem with what you want on it: https://docs.ops.city/ops/configuration#files .
You can then deploy this as-is with a simple config.json or you can create your own new package (for instance if it's something like node and you want to share it with your team).
Well chuffed. I couldn't get ops pkg push to work at all, and instead had to manually upload a tar.gz as no matter what I did, it wouldn't find my local image (though it would find it if I did ops pkg load -l nano-web_0.0.1).
Is there some sort of Discord or IRC to ask dumb questions instead of spamming up HN?
Reminds me of the famous Bill Gates quote: “Measuring software development progress by lines of code is like measuring aircraft building progress by weight”
At the end of the day you'd still be running linux. So what are some key differences between Linux and Nanos (excluding the millions of lines of code, userland arguments)?
There is still quite a lot of code in your average linux kernel that is not easy to ifdef out. For instance the capability of running many different programs by many different users means you have a scheduler that has to support that. You have to protect those different programs from being screwed with by other 'users' or other programs (eg: just cause I'm able to own one program doesn't give me access immediately to someone else's on the same machine if owned by a different user) even though a lot of companies have largely walked away from that concept.
It goes a lot deeper than that though. Shared memory, IPC amongst processes, semaphores all touch that. Then we start talking about users and their ptys and managing that entire environment. Then you have to start looking at /sys and /proc and /dev and all the other places that get stuffed with all sorts of things.
It really truly is very different because it started with a completely different architecture and deployment model in design.
The unikernel still has vastly reduced overall system surface area— for both good and ill. Less security concerns, random package manager nonsense, things that need to be monitored in case they randomly explode, and all the rest of it… but also harder to get a shell or connect a debugger.
There are a ton of different download options https://ops.city/downloads (versus just compiling from source yourself if you're super paranoid), including signed/notarized/stapled builds from companies such as Apple.
Tangent: I have an earnest interest in learning about methods for installing software obtained from the internet which would be any more secure than this.
This is true. There is quite a lot of applications that you can just run out of the box but then I'll give you two cases where that won't be the case and do require patches:
1) In many interpreted languages it is common to have convenience commands that shell out to call a script which shells out to call another script and 8 layers later you get to the actual real command. When I'm creating a package for an application that does this I have to usually figure out what env vars are being set, what paths are being changed and so forth. This is probably a super easy thing for whoever made the software to begin with but not so easy for someone that wants to just use it. So the solution is to make the original author aware that it might go into an unikernel environment or, far less probable, convince them that a better method would be to not do this practice to begin with as it.
2) In older software (specifically I'm looking at mid-to-late 90s), in a time before threads and commodity SMP machines and the cloud it was pretty common to write software that used multiple processes to do many things. Postgres is the most common example I use here (keep in mind postgres is descended from Ingres from the early 80s and Dr. Stonebraker is now on his tenth? twentieth? database venture DBOS (https://www.dbos.dev/) - which definitely has ideas that we are very keen about.
Anyways, that's not really the case today with go, rust, java, etc. For apps like this we will, from time to time port them. That's exactly what we did with postgres too to make it multi-threaded instead: https://repo.ops.city/v2/packages/francescolavra/postgres/16... .
I think there is a lot of opportunity out there for individuals to come in and create newer versions of software like this and get some really awesome benefits while maintaining more or less the same parity.
Partly because it's an ecosystem that I'd like to see grow. More people using it means that the other Rust-based projects I use will be better too.
Partly because it is the "carbon fiber" of programming languages. Just its use is something you can use to market to other engineers.
Partly because it's very ergonomic and I enjoy working with it. I'd much rather dive into esoteric Rust code than esoteric C code.
Partly because code written in of Rust will have less errors than code written inside of C. It isn't just memory safety - it's all of the type theory/best practices we figured out in the last 50 years.
I understand why it was written in C. It's an easier language to work with and they probably needed something out quickly. But coding in Rust is a sign (not proof, but a sign) that a program is of quality make.
For this type of unikernel project C makes sense. I’m a fan of both C and Rust. I like that Rust prevents typical code safety problems, but I like that C hardly changes over time whereas younger languages like Rust are constantly changing. It’s plenty possible to write correct, clean, memory-safe, and understandable code in C, especially if verified by extensive fuzz testing and code scanning.
It’s really small. I know c doesn’t translate to rust, but it would be a good template for someone doing that. You need the state in an os, and using threads as memory containers is a bad idea here. But you could certainly do the array thing or arc everything steal the interrupt notion from the embedded rust community, write a different async runtime and that would get you most of the way
> Yes, but we caution users to evaluate if you really need kubernetes. Chances are you don't and you will experience severe performance and security problems if you choose to run under k8s. If you still find you must here are instructions for running Nanos under k8s.
https://nanos.org/faq
Pretty interesting FAQ! I hope to see more HN discussion about this page!