We just released 4.8 million real user–ChatGPT conversations collected from our public chatbot.
- Covers a wide range of topics and languages, all from actual users in the wild.
- Includes 122K conversations from reasoning models (o1-preview and o1-mini) which are long, often involving complex problem solving, and very costly to collect.
Thanks for the feedback! Yes, the demo is definitely limited. The reason I built NeuralOS is that I'm excited about a future where boundaries between software categories fade away. Imagine converting a movie directly into an interactive video game, customizing app interfaces by talking to it, or sharing the same underlying physics/world model between movies and games. Perhaps someday, movies or even interactive games could just be detailed text prompts describing scenes and characters, with the OS "hallucinating" everything on the fly (maybe movies adapt to user preferences as well so different users watch different "versions" of the same underlying movie plot). This minimizes storage and download times, but also provides maximal flexibility.
Unlike other ML-based OS projects (such as Gemini OS, which generates code and renders traditional UIs), NeuralOS directly generates every pixel. While this makes it susceptible to hallucination, in my opinion the other side of hallucination is full flexibility. In the future, I imagine operating systems running entirely (or mostly) on GPUs, adapting to user intent on the fly rather than relying on pre-designed menus and options.
I definitely have many of the same dreams as you. I've always been captivated by the holodeck. But I'm not convinced things need to be neural from top to bottom. There are many things I do not want my machine to hallucinate about. There are things I want to be static and uncompromising. You're also talking about compression, which, to be fair, is what current ML systems do best. Though I think we need some serious innovation to get to the point of generating world models.
That isn't to say that I don't think there shouldn't be neural OS's. But I do imagine them being something radically different. Do we really want them to mimic what we have now? Or is that not, in some vague way, more like a mind?
Regardless, I think this is really neat. I'm a big fan of doing things "just because" and "I wonder what would happen if". So I'm not trying to knock you down. I mean, I'm wrong about a lot of things haha
This essentially is the idea of Star Trek computers, where there were "neural gel packs" being programmed/primed for different purposes on the starship's systems.
Damn, I have to think about this more. Essentially you are building a holodeck computer, where the users interacting with it just describe roughly what they want and the computer just generates it - in human language being the primary interface.
When we say "powered by a neural network," we mean something fundamentally different from a traditional OS (or even gemini os). NeuralOS is essentially a video generation model that "hallucinates" every pixel on the screen in response to user inputs (mouse movements, clicks, keyboard inputs).
There is no underlying kernel, no function calls, no program execution, and no networking. Everything is purely visual and imagined by the neural model. You can think of it as a safe, isolated container where nothing can actually run or cause harm, since no real code executes. It's essentially an interactive video simulation, conditioned entirely on user inputs.
Note: The Space is intended as a template, so please duplicate it and run with your own GPU for a better experience. (The default Space has only one worker.)
Recommended GPU: At least an L40, ideally an A100-large. (The original demo at neural-os.com used H100s.)
All code and models are self-contained in the huggingface space.
Actually NeuralOS works very differently from Gemini OS. NeuralOS directly generates each screen at the pixel level entirely from neural networks, while Gemini OS generates code that's then rendered into a traditional UI. This difference is why NeuralOS is much slower and currently runs at a lower frame rate.
Thanks everyone for trying out NeuralOS, and apologies for the frustrating user experience!
I coded up the demo myself and didn't anticipate how disruptive the intermittent warning messages about waiting users would become. The demo is quite resource-intensive: each session currently requires its own H100 GPU, and I'm already using a dispatcher-worker setup with 8 parallel workers. Unfortunately, demand exceeded my setup, causing significant lag and I had to limit sessions to 60 more seconds when others are waiting. Additionally, the underlying diffusion model itself is slow to run, resulting in a frame rate typically below 2 fps, further compounded by network bottlenecks.
As for model capabilities, NeuralOS is indeed quite limited at this point (as acknowledged in my paper abstract). That's why the demo interactions shown in my tweet were minimal (opening Firefox, typing a URL).
Overall, this is meant as a proof-of-concept demonstrating the potential of generative, neural-network-powered GUIs. It's fully open-source, and I hope others can help improve it going forward!
This is a shockingly fresh idea. I get that this generates every pixel from scratch, unlike Gemini approaches. But, i wonder how do you think this type of neural OS would be able to communicate with the internet or other similar neural os. It would have to at least send and get http responses?
Nǐ hăo, xìe xìe Yuntian!
I read the readme and paper but haven’t played around much yet. I find this fascinating and I don’t care much about poor “experience” because intuitively I feel this idea couldn’t produce something as reliable and flexible as a real OS anyway. I see you talked about inability to install new software and my reaction was “well obviously”, because surely it will be at least as limited as the training data, while a real OS provides lots of software of great complexity which is seldom used.
Could you talk about your hopes for the future on this project? What are your thoughts on having a more simplified interface which could combine inputs in a more abstract way, or are you only interested in simulating a traditional OS?
Thanks again.
PS the waiting time while firefox “loads” made me laugh. I presume this is also simulated.
Thanks for your comment! I completely agree that currently NeuralOS is far from being as reliable as a real OS. The Firefox loading time is indeed a funny artifact of the neural model simulating delay in real OS.
However, my real dream behind this project is to blur the boundaries across applications, not just simulate traditional OS interactions. For example, imagine converting a movie we're watching directly into an interactive video game, or instantly changing the interface of an app (like Signal) to something we prefer (like Facebook Messenger) on the fly.
Of course, the current training data severely limits what's achievable today. But looking forward, I envision combining techniques from controllable text generation (such as Zhiting Hu's "Toward Controlled Generation of Text" paper) or synthesizing new interaction data to achieve greater and customization. I believe this is a promising path toward creating truly generative and personalized interfaces.
Hey OP, I understand the load you are seeing on the servers. Given that, can you describe the functionality and how the interface is supposed to work? Specifically how NNs and LLMs provide this functionality?
A generative operating system that directly predicts screen images based on mouse and keyboard inputs, powered by an RNN for state modeling and a diffusion model for image generation.
- Covers a wide range of topics and languages, all from actual users in the wild.
- Includes 122K conversations from reasoning models (o1-preview and o1-mini) which are long, often involving complex problem solving, and very costly to collect.
- 2.5M conversations from GPT-4o.
Links:
- Non-toxic version: https://hf.co/datasets/allenai/WildChat-4.8M
- Full version (gated): https://hf.co/datasets/allenai/WildChat-4.8M-Full
- Exploration tool: https://wildvisualizer.com