This is completely wrong. First, disabling overcommit is wasteful because of fork and because of the way thread stacks are allocated. Sorry, you don't get exact memory accounting with C, not even Windows will do exact accounting of thread stacks.
Secondly, memory is a global resource so you don't get local failures when it's exhausted, whoever allocates first after memory has been exhausted will get an error they might be the application responsible for the exhaustion or they might not be. They might crash on the error or they might "handle it", keep going and render the system completely unusable.
No, exact accounting is not a solution. Ulimits and configuring the OOM killer are solutions.
Without going into a discussion about whether this is right or wrong, how is fork(2) wasteful?
fork(2) has been copy-on-write for decades and does not copy the entire process address space. Thread stacks are a non-sequitur either as stack uses the data pages, the thread stacks are rather small in size in most scanarios, hence the thread stacks are also subject to copy-on-write.
The only overhead that the use of fork(2) incurs is an extra copy of process' memory descriptior pages, which is a drop in the ocean for modern systems for large amounts of RAM.
Unless you reserve on fork you're still over committing because after the fork writes to basically any page in either process will trigger memory commitment.
Thread stacks come up because reserving them completely ahead of time would incur large amounts of memory usage. Typically they start small and grow when you touch the guards. This is a form of overcommit. Even windows dynamically grows stacks like this
> Thread stacks come up because reserving them completely ahead of time would incur large amounts of memory usage. Typically they start small and grow when you touch the guards. This is a form of overcommit.
Ahead of the time memory reservation entails a page entry being allocated in the process’s page catalogue («logical» allocation), and the page «sits» dormant until it is accessed and causes a memory access fault – that is the moment when the physical allocation takes place. So copying reserved but not accessed yet pages has zero effect on the physical memory consumption of the process.
What actually happens to the thread stacks depends on the actual number of active threads. In modern designs, threads are consumed from thread pools that implement some sort of a run queue where the threads sit idle until they get assigned a unit of work. So if a thread is idle, it does not use its own stack thread and, consequently, there is no side effect on the child's COW address space.
Granted, if the child was copied with a large number of active threads, the impact will be very different.
> Even windows dynamically grows stacks like this
Windows employs a distinct process/thread design, making the UNIX concept of a process foreign. Threads are the primary construct in Windows and the kernel is highly optimised for thread management rather than processes. Cygwin has outlined significant challenges in supporting fork(2) semantics on Windows and has extensively documented the associated difficulties. However, I am veering off-topic.
I am aware reserving excess memory doesn't commit said memory. But it does reserve memory, which is what we were talking about. The point was that because you can have a lot of threads and restricting reserved stacks to some small value is annoying all systems overcommit stack. Windows initially commits some memory (reserving space in the page file/ram) for each but will dynamically commit more when you touch the guard. This is overcommit. Linux does similarly.
Idle threads do increase the amount of committed stack. Once their stack grows it stays grown, it's not common to unmap the end and shrink the stacks. In a system without overcommit these stacks will contribute to total reserved phys/swap in the child, though ofc the pages will be cow.
> Windows employs a distinct process/thread design, making the UNIX concept of a process foreign. Threads are the primary construct in Windows and the kernel is highly optimised for thread management rather than processes. Cygwin has outlined significant challenges in supporting fork(2) semantics on Windows and has extensively documented the associated difficulties. However, I am veering off-topic.
The nt kernel actually works similarly to Linux w.r.t. processes and threads. Internally they are the same thing. The userspace is what makes process creation slow. Actually thread creation is also much slower than on Linux, but it's better than processes. Defender also contributes to the problems here.
Windows can do cow mappings, fork might even be implementable with undocumented APIs. Exec is essentially impossible though. You can't change the identity of a process like that without changing the PID and handles.
Fun fact: the clone syscall will let you create a new task that both shares VM and keeps the same stack as the parent. Chaos results, but it is fun. You used to be able to share your PID with the parent too, which also caused much destruction.
> Idle threads do increase the amount of committed stack.
I am not clear on why the stack of an idlying thread would continue to grow. If a previously processed unit of work resulted in large amounts of memory pages backing the thread stack getting committed, then yes, it is not common to unmap the no longer required pages. It is a deliberate trade-off: automatic stack shrink is difficult to do safely and cheaply.
Idle does not actually make stacks grow, put simply.
> The nt kernel actually works similarly to Linux w.r.t. processes and threads.
Respectfully, this is slightly more that entirely incorrect.
Since Linux uses a single kernel abstraction («task_struct») for both processes and threads, it has one schedulable kernel object – «task_struct» – for both what user space calls a process and what user space calls a thread. «Process» is essentially a thread group leader plus a bundle of shared resources. Linux underwent the consolidation of abstractions in a quest to support POSIX threads at the kernel level decades ago.
Since fork(2) is, in fact, clone(2) with a bunch of flags, what you get depends on clone flags: sharing VM, files, FS context, signal handlers, and whether you are in the same thread group (CLONE_THREAD) and that creates a new thread group with its own memory management (but populated using copy-on-write), separate signal disposition context, etc.
Windows has two different kernel objects: a process (EPROCESS) and a thread (ETHREAD/KTHREAD). Threads are the schedulable entities; a process is the container for address space, handle table, security token, job membership, accounting, etc. They are tightly coupled, but not «the same thing».
On Windows, «CreateProcess» is heavier than Linux fork for structural reasons: it builds a new process object, maps an image section, creates the initial thread, sets up the PEB/TEB, initialises the loader path, environment, mitigations, etc. A chunk of that work is kernel-side and a chunk is user-mode (notably the loader and, for Win32, subsystem involvement). Blaming only «userspace» is wrong.
Defender (and third-party AV/EDR) can measurably slow process creation because it tends to inspect images, scripts, and memory patterns around process start, not because of deficiences of the kernel and system calls design.
> […] because after the fork writes to basically any page in either process will trigger memory commitment.
This is largely not true for most processes. For a child process to start writing into its own data pages en masse, there has to exist a specific code path that causes such behaviour. Processes do not randomly modify their own data space – it is either a bug or a peculiar workload that causes it.
You would have a stronger case if you mentioned, e.g., the JVM, which has a high complexity garbage collector (rather, multiple types of garbage collectors – each with its own behaviour), but the JVM ameliorates the problem by attempting to lock in the entire heap size at startup or bailing if it fails to do so.
In most scenarios, forking a process has a negligible effect on the overall memory consumption in the system.
> In most scenarios, forking a process has a negligible effect on the overall memory consumption in the system.
Yes, that’s what they’re getting at. It’s good overcommitment. It’s still overcommitment, because the OS has no way of knowing whether the process has the kind of rare path you’re talking about for the purposes of memory accounting. They said that disabling overcommit is wasteful, not that fork is wasteful.
Yep. If you aren't overcommitting on fork it's quite wasteful, and if you are overcommitting on fork then you've already given up on not having to handle oom conditions after malloc has returned.
This is a crucial distinction and I agree when the problem is framed this way.
The original statement by another GP, however, was that fork(2) is wasteful (it is not).
In fact, I have mentioned it in a sister thread that the OS does not have a way to know of the kind of behaviour the parent or the child will exhibit after forking[0].
Generally speaking, this is in line with the foundational ethos of the UNIX philosophy where UNIX gives its users a wide array of tools tantamount to shotguns that shoot both forward and backward simultaneously and the responsibility for with the number of deaths and permanent maimings ultimately lies with its users. In comparison, memory management in operating systems that run mainframes is substantially more complex and sophisticated.
[0] In a separate thread, somebody else has mentioned a valid reverse scenario where the child idles by after forking and it is the parent that makes its data pages dirty causing the physical memory consumption to baloon.
> because of fork and because of the way thread stacks are allocated
For modern (post-x86_64) memory allocators a common strategy is to allocate hundreds of gigabytes of virtual memory and let the kernel handle deal with actually swapping in physical memory pages upon use.
This way you can partition the virtual memory space into arenas as you like. This works really well.
Which is a major way turning off overcommit can cause problems. The expectation for disabling it is that if you request memory you're going to use it, which is frequently not true. So if you turn it off, your memory requirements go from, say, 64GB to 512GB.
Obviously you don't want to have to octuple your physical memory for pages that will never be used, especially these days, so the typical way around that is to allocate a lot of swap. Then the allocations that aren't actually used can be backed by swap instead of RAM.
Except then you've essentially reimplemented overcommit. Allocations report success because you have plenty of swap but if you try to really use that much the system grinds to a halt.
> your memory requirements go from, say, 64GB to 512GB.
Then your memory requirements always were potentially 512GB. It may just happen to be even with that amount of allocation you may only need 64GB of actual physical storage; however, there is clearly a path for your application to suddenly require 512GB of storage. Perhaps when it's under an attack or facing misconfigured clients.
If your failure strategy is "just let the server fall over under pressure" then this might be fine for you.
> Then your memory requirements always were potentially 512GB. It may just happen to be even with that amount of allocation you may only need 64GB of actual physical storage; however, there is clearly a path for your application to suddenly require 512GB of storage.
If an allocator unconditionally maps in 512GB at once to minimize expensive reallocations, that doesn't inherently have any relationship to the maximum that could actually be used in the program.
Or suppose a generic library uses buffers that are ten times bigger than the maximum message supported by your application. Your program would deterministically never access 90% of the memory pages the library allocated.
> If your failure strategy is "just let the server fall over under pressure" then this might be fine for you.
The question is, what do you intend to happen when there is memory pressure?
If you start denying allocations, even if your program is designed to deal with that, so many others aren't that your system is likely to crash, or worse, take a trip down rarely-exercised code paths into the land of eldritch bugs.
Yeah but these only request address space. The memory is neither readable nor writable until a subsequent mprotect call. Ideally reserving address space only shouldn’t be counted as overcommit.
What they should do, actually, is sue the EU for harassment in the US, like 4chan and the kiwifarms did with the UK. And then the EU can start firewalling X. Firewall everything bad, age gate everything, throw up the great firewall of the EU. I need Brussels to protect my freedoms.
It isn't but then everyone does it and then everyone does it recursively and 70kb become 300MB and then it matters. Not to mention that "well constructed, well tested, well distributed" are often actually overengineered and poorly maintained.
you must understand that your country is basically a uniparty that allows vigorous debate in a very zone of ideas (to paraphrase Chomsky), e.g Biden allowed the genocide to continue for as long as he was in office. Your country is basically subservient to Capital and the Israeli lobby.
The lesson to draw from Gaza is that if you become inconvenient to "the people in power" tomorrow, you would meet the exact same fate.
Exactly, it's laughable that this is coming from the same people who cheered on auto de-monetization for even mentioning the word "Covid" in a YT-video or the countless de-platforming and de-banking of individuals. Is this still gaslighting or something else?
Google didn't censor covid-related conspiracies or whatever at the behest of the government. YouTube can censor whoever it wants but the US government cannot.
Also, do you have any actual evidence of political debanking in the US? I can't find any references to it other than the propaganda of the current administration.
Google’s employees didn’t mention government pressure earlier and the company only started talking about that in conjunction with their PR plan to get favorable treatment from the current administration:
Version 1 was never licensed to anyone. Version 2 was only licensed to universities for an undiscolsed price. Version 3 was sold as a book, I think this is the version you are referring to. However note that this version contained a license that only allowed non commercial uses of the source code. It also came with no support, no community and no planned updates (the project was shelved half a year later in favor of inferno)
More than the price tag the problem is that plan 9 wasn't really released until 2004.
Why would it? The key difference is businesses / people making choices and the government coercing businesses / people to make choices the government wants. One is a first amendment violation, one is exercising the first amendment.
A company with significant coercive power (through sticky market effects) is much better thought of as a governmentesque censoring entity rather than some mere group of individuals exercising their individual speech. What you're describing is not a virtue of the first amendment, but rather a shortcoming of its implementation and a subsequent failure to properly regulate corporations/LLCs/etc. OP is right - people's ready embrace of corpo-authoritarianism when it lined up with their social mores set the stage for where we are at now. That comic has always been a low point of Munroe's.
This is a long winded way of ignoring the fact that this is government coercion, something the 1st amendment was designed to prevent. What you're talking about is _not_ something the 1st amendment was designed to prevent.
> What I'm taking issue with is your whitewashing corporate censorship as "exercising the first amendment" in your first comment.
It read more as equivocating completely different things, one of which is much more serious. The fact is "corporate censorship" is not a 1st amendment issue.
Ask yourself who is more legitimized to take such enforcement actions: a handful of privately owned corporations acting in concert or a democratically elected government?
No, the "first amendment only applies to government" is cope of the highest degree, it was always a cope.
You accepted that an oligopoly could dictate what you could do with devices you owned because it suited your preferences and now that you are on the other side of the sword you squirm.
XKCD's point stands as long as nobody has a monopoly on an important medium. Apple and Google effectively do have a duopoly on mobile app distribution, and mobile apps are an important medium for speech in 2025.
From the first part of your comment, it directly follows that neither Apple nor Google has a required monopoly on the important medium.
IMO, we need to stop thinking in this broken paradigm of "-opolies" (with its loaded requirement to define what constitutes a given "market") and look at the actual coercive power they wield through market stickiness. Apple and Google both wield much coercive power with regards to software running on mobile devices.
I am all for letting low effort labor exploiters create low effort comics. Same as I am for my having no obligation to support them directly by reading their comics.
Entertainers are a kind of contemporary secular faith healer and tribal shaman, imo.
To mix metaphors, stopped clocks who can be right and worth listening to only in very specific situations.
> 2800 day streak here, primarily in Finnish. I haven't been a fan of the app for a long time, but the problem I've always had switching is the question: What else?
I think you are reading that wrong, go.sum isn't a list of dependencies it's a list of checksums for modules that were, at some point, used by this module. All those different versions of the same module listed there, they aren't all dependencies, at most one of them is.
Assuming 'go mod tidy' is periodically run go.mod should contain all dependencies (which in this case seems to be shy of 300, still a lot).
