"and investigate how much of the usual machinery of reinforcement learning algorithms can be replaced with the tools..."
These sounds as if the authors have confidently figured out that the current reinforcement learning formulation is not good enough.
On the other hand, I think the recent large language models have showed us that much of the world knowledge is indeed predictive. That, if you can predict accurately (next words), you can understand higher more abstract things. The hypothesis that much of world knowledge is predictive, is very important in the framework of reinforcement learning because that means that with enough General Value functions learned off-policy, one can predict almost anything about the world that is useful to the agent in achieving its goals. (cf the Horde paper).
By "understand" I mean the knowledge that is missing to the agent in order to control the environment toward achieving its goal. Reinforcement learning is concerned with this sort of interaction, between an agent (a decision maker) and an (unknown) environment. The (only) goal of the agent is to maximize its cumulative sum of reward (cf the Reward hypothesis(.
> And why are you calling "knowledge" that which is predictive?
No, I do not think I am saying that, but if it comes across like that, let me be more precise.
I mean that most (but not all) of the world knowledge is predictive. An example of not-predictive knowledge is factual knowledge, like mathematics. Knowledge being predictive is important for an autonomous decision maker because the knowledge can be verified solely by the agent (not a teacher, as it is in the framework of supervised learning). One crucial thing to understand about the framework of reinforcement learning is that it makes the agent solely responsible for its way of behaving. As it should be. Then, an effective way to do that in a scalable way (to not rely on some oracle teacher or anything else) is to be able to verify any knowledge that the agent wants to acquire, in order to achieve its goals.
This kind of tool destroys ones ability to program long sustainable production code. For a novice programmer this has tremendous negative effect on the learning curve. For an experienced programmer this tool is useless, because an experienced programmer will NEVER rely on "popularity" of some code-snippet out there in the wild. Programming is a very intense and deep practice and it is certainly not crafted using this kind of tools. This tool helps people write poor quality code for customers. Makes me wonder, what Knuth would say on this?
What about experts who are transitioning new languages or technologies? I'd say this tool has tremendous value for them. Also, as long as novice programmers understand the fundamentals (DS, algorithms) something like this won't have a 'tremendous negative effect' on their learning curve.
IntelliJ already has most of what this tool offers (but for java as opposed to python). It has auto-complete, quick access to documentation, quick fix suggestions (such as missing imports, etc.) and many many others not available in Kite at the moment. You wouldn't say IntelliJ "destroys ones ability to program long sustainable production code" would you? Because if you would, you'd be absolutely wrong.
IntelliJ certainly doesn't do this. IntelliJ and Kite have functionalities in common, which are fine. The other parts of Kite that IntelliJ hasn't, are problematic.
Is it even good practice to have a dozen of tabs open?
Maybe it depends on the type of websites open? Travel related with 10 SO questions and 3 Youtube instances? That's not focused work right?
The point is I am very confused why people have dozens of tabs open at once. Please share with me the reason behind this habbit
I use tree style tabs[treeee] in Firefox and often open 5-10, process them (which may involve opening several more each). Eventually I find the information I need, but this makes it easy to group the depth of my search. This branch is from the API docs, that branch is from StackOverflow (and usually has a subbranch of questions that catch my attention from the more fun StackExchange sites, whoops), and then another branch with language specific questions. I try to clear everything at the end of the week, but it's really not a huge burden because I can collapse a tree and just mentally ignore it until then. Average day with lots of coding? 100+ tabs easily.
1. Before I finish studying a subject, I'm unable to identify the best sources of information.
2. Before I identify the best sources of information, I'm unable to adequately determine which information I should act on and which sources I should make note of in my personal wiki.
3. Before I make note of the best sources of information, I shouldn't close any tabs.
So when I study a subject, I keep all tabs open except for those resources that are clearly of low quality. After I finish studying the subject, I process all tabs, often hundreds, and I make note of the best sources of information in my personal wiki.
One reason is trying to follow a HN discussion. I do not see a better way to do this than keeping several tabs open with topics I would like to recheck a few hours later.
There could be other ways doing this and I have some ideas, but right now multiple tabs seem to be most practical solution for this.
I don't drink coffee at all. I think it has the placebo effect and we actually don't work harder. In my case, when I drink coffee, I get nervous and can't think straight anymore because of the energy rush
These sounds as if the authors have confidently figured out that the current reinforcement learning formulation is not good enough.
On the other hand, I think the recent large language models have showed us that much of the world knowledge is indeed predictive. That, if you can predict accurately (next words), you can understand higher more abstract things. The hypothesis that much of world knowledge is predictive, is very important in the framework of reinforcement learning because that means that with enough General Value functions learned off-policy, one can predict almost anything about the world that is useful to the agent in achieving its goals. (cf the Horde paper).