Much of the value from Chaos testing can be gotten much more simply with good rolling CI. Many of the problems that Chaos engineering solved are now considered table stakes, directly implemented into our frameworks and tested well by saidsame CI.
A significant problem with early 'Web Scale' deployments was out of date or stale configuration values. You would specify that your application connects to backend1.example.com for payments and backend2.example.com for search. A common bug in early libraries was that the connection was established once at startup, and then never again. When the backend1 service was long lived, this just worked for months or years at a time - TCP is very reliable, especially if you have sane values on keepalives and retries. Chaos Monkey helped find this class of bug. A more advanced but quite similar class of bug: You configured a DNS name, which was evaluated once at startup, and again didn't update, Your server for backend1 had a stable address for years at a time, but suddenly you needed to failover to your backup or move it to new hardware. At the time of chaos monkey, I had people fight me on this - They believed that doing a DNS lookup every five minutes for your important backends was unacceptable overhead.
The other part is - Modern deployment strategies make these old problems untenable to begin with. If you're deploying on kubernetes, you don't have an option here - Your pods are getting rebuilt with new IP addresses regularly. If you're connecting to a service IP, then that IP is explicitly a LB - It is defined as stable. These concepts are not complex, but they are edge boundaries, and we have better and more explicit contracts because we've realized the need and you "just do" deploy this way now.
Those are just Chaos Monkey problems, though - Latency Monkey is huge, but solves a much less common problem. Conformity Monkey is mostly solved by compliance tools; You don't build, you buy it. Doctor Monkey is just healthchecks - K8s (and other deployment frameworks) has those built in.
In short, Chaos Monkey isn't necessary because we've injected the chaos and learned to control most of what that was doing, and people have adopted the other tools - They're just not standalone, they're built in.
I buy vinyl for the album cover. If somebody were to sell me a digital download which also ships me an empty slipcase, I would buy it, for almost the same price that I pay for vinyl. I do have the record player, but I don't think I've used it more than a dozen days in my life.
There's a lot of value to the physical artifact, but the precise nature of the physical artifact is up for playing with.
A friend was selling a parent's vinyl collection on eBay, and a couple of times buyers asked to be sent only the slipcase to save weight and therefore postage costs.
AI driven research tells you everything you need to know about your conclusions; there's a hint of truth that's hiding an incredible web of misconceptions.
Mesh VPNs as a security mechanism replacing having secure server to server communication is just replacing one soft-center security mechanism with another. Mesh VPNs as the gateway to services that are themselves well secured is well over doubly secure over just having publicly accessible services; now you need the security holes to line up.
I am a Site Reliability Engineer (SRE), Google Style, with experience at both large and small organizations. I can help you build a Platform Engineering practice from the very beginning. I'm looking to help small dev teams increase their velocity by implementing best-practices of Devops: CI/CD, Kubernetes Deployments, and effective Monitoring frameworks.
I am a Site Reliability Engineer (SRE), Google Style, with experience at both large and small organizations. I can help you build a Platform Engineering practice from the very beginning. I'm looking to help small dev teams increase their velocity by implementing best-practices of Devops: CI/CD, Kubernetes Deployments, and effective Monitoring frameworks.
The reason to target k8s on cloud vms is that cloud VMs don't subdivide as easily or as cleanly. Managing them is a pain. K8s is an abstraction layer for that - Rather than building whole machine images for each product, you create lighter weight docker images (how light weight is a point of some contention), and you only have to install your logging, monitoring, and etc once.
Your advice about bigger machines is spot on - K8s biggest problem is how relatively heavyweight the kublet is, with memory requirements of roughly half a gig. On a modern 128g server node that's a reasonable overhead, for small companies running a few workloads on 16g nodes it's a cost of doing business, but if you're running 8 or 4g nodes, it looks pretty grim for your utilization.
You can run pods, with podman and avoid the entire k8s stack or even use minikube on a machine if you wanted to. Now that rootless is the default in k8s[0] the workflow is even more convenient and you can even use systemd with isolated users on the VM to provide more modularity and seporation.
It really just depends on if you feel that you get value from the orchestration that full k8s offers.
Note that on k8s or podman, you can get rid of most of the 'cost' of that virtualization for single placement and or long lived pods by simply sharing a emptyDir or volume shared between pod members.
There is enough there for you to test to see that the performance is so close to native sharing unix sockets that way, that there is very little performance cost and a lot of security and workflow benefits to gain.
As podman is daemonless, easily rootless, and on mac even allows you to ssh into the local linux vm with `podman machine ssh` you aren't stuck with the hidden abstractions of docker-desktop which hides that from you it has lots of value.
Plus you can dump a k8s like yaml to use for the above with:
podman kube generate pgdemo-pod
So you can gain the advantages of k8s without the overhead of the cluster, and there are ways to launch those pods from systemd even from a local user that has zero sudo abilities etc...
I am using it to validate that upstream containers don't have dial home by producing pcap files, and I would also typically run the above with no network on the pgsql host, so it doesn't have internet access.
IMHO the confusion of k8s pods, being the minimal unit of deployment, with the fact that they are just a collection of containers with specific shared namespaces in the general form is missed.
As Redhat gave podman to CNCF in 2024, I have shifted to it, so haven't seen if rancher can do the same.
The point being is that you don't even need the complexity of minikube on VM's, you can use most of the workflow even for the traditional model.
The NSA has railroaded bad crypto before [1]. The correct answer is to just ignore it, to say "okay, this is the NSA's preferred backdoored crypto standard, and none of our actual implementations will support it."
It is not acceptable for the government to be forcing bad crypto down our throats, it is not acceptable for the NSA to be poisoning the well this way, but for all I respect DJB, they are "playing the game" and 20 to 7 is consensus.
I found the opposite problem. I tried to hang out with non-tech people, I spent a lot of time hanging out with non-tech people. The kinds of people who I ended up hanging out with, and I recognized that this might just be a me problem, though I certainly tried to avoid pigeonholeding myself, were not great people. The great non-tech people I met didn't stay very long.
reply