Catchall for 25 years :) (on domainfactory - df.eu) each company/service gets their own email prefix, so I can determine spam and also filter unsolicited emails.
My general point on GraphRAG is that it extracts and compresses the horizontal topic-clustering across many documents and makes that available for retrieval.
And that by creating the semantic network of entities, you can use patterns in the graph structure to answer questions that rely on information coming together from different documents. Think the detectives board connecting facts with strings from many different sources.
Feel free to ping me for a deeper discussion: michael at neo4j
RenTec was covered in much depth at the Acquired podcast.
Basically algorithms from signal processing applied to huge volumes of historical and current data to determine buy and sell signals. Originally developed for national defense.
Very secretive all external partners were bought out. Only hundred or so people benefited in the billions per person. Including Robert Mercer of Trump campaign financing and Cambridge Analytica fame.
Very interesting but also disheartening episode about smart people only caring about getting richer.
Sometimes I think - what if all the smartest people that work on purely commercial things, had rather spent their time on solving problems in medicine, etc.
Of course, some will argue that
A) These people wouldn't have been equally motivated to work on such problem, compared to the ones that make them wealthy.
B) Some of the investment folks are contributing to the actual sciences, by fronting them with money.
But, still, I can't help but to think what a brain drain the finance industry is. You take some of the smartest and most motivated people out there, and make them spend all their energy on vacuuming pennies off the market, or identifying commercially successful companies.
It's been an utter cultural disaster. We've missed out on so much original science and progress because these smart idiots threw away their talents on building machines to game the casino, when they could have made game-changing contributions to fundamental original research.
Imagine if Shannon, Turing, Von Neumann, Einstein, and Dirac had done this.
Yes, they put some money back, but not nearly enough to compensate for the damage.
The real disaster has been normalising this kind of "success" as the best of all possible achievements, when in fact it's spectacularly cheap and unambitious compared to the goals of previous generations.
If anyone thinks I'm overstating the heresy here, remember - a financialised economy is optimised for short-term gain, not long-term development.
The flip side of "investment" is an economy where hundreds of millions are bankrupted by health insurance, where rents are unaffordable (never mind property), where workers are treated like spreadsheet assets and not like people, where fraud is endemic, where many people are putting off having kids because they literally can't afford them, where planes fall out of the sky, and where the entire machine regularly demands government bailouts because it's stuck in a manic depressive cycle of overconfidence and opportunism followed by collapse.
That's not even looking at the incredibly toxic political effects.
>We've missed out on so much original science and progress because these smart idiots threw away their talents on building machines to game the casino, when they could have made game-changing contributions to fundamental original research.
The vast majority of people who succeed in finance are very ambitious; if finance wasn't an option they'd just have found some other way to make money, not suffer as a peon doing fundamental research for mediocre pay in a lab somewhere. Just be glad they didn't go into politics where their ambition could have done even more damage.
Eh, it's hard to call it much of a "brain drain" when getting a research job is so difficult. There's no shortage of smart people interested in working in research for less than they'd make at any professional role, much less in the highest-paid industries.
PhD programs pay basically nothing, are selective, require candidates to jump through all sorts of hoops and still have no trouble filling out. Later on, becoming a professor—or some other sort of researcher with similar scope, autonomy and funding—is basically impossible, harder than making a bunch of money in quantitative finance. And yet each opening has hundreds of realistically qualified applicants. (Realistically qualified in the sense that they'd be able to do good research, anyway.)
Ya, "brain drain" is probably not the right metaphor. But you get the gist: Smart people have more opportunities in finance.
As you know, funding for academics, scholastics, and basic research has been on a decades long decline. The West's investment in knowledge production peaked in response to Sputnik. As the Cold War wound down, neoliberalism and the "peace dividend" replaced that commitment.
Too bad.
Maybe climate crisis, our new existential threat, will be another Sputnik moment.
The are various models from textbooks now that seem (or are presented as) too naive to be applied to financial markets, and too slow (eg gradient descent/expectation maximisation) on 1980s computers with "big data".
And then, the academic perspective is that prices should be modelled as random walks, though you may talk/learn about things such as "trend" and volatility. Suggesting that hidden variables/states/transitions can be learned from historical data is usually considered pseudo-scientific.
Meanwhile it so obviously worked for RenTec, with relatively miniscule computing capacity, for decades.
Repeating the academic perspective just seems disgenuine. If prices are not random walks, then financial markets are actually games.
I think this is one of those cases where conflating "unpredictable" with "random" breaks down. As a matter of simplification, we treat many unpredictable processes as random if making the process at least somewhat predictable is sufficiently intractable. While this can change very quickly e.g. breaking encryption algorithms, for many data models the gains in making a process less unpredictable are more incremental and are largely dependent on having both better math and more efficient compute.
Unpredictability is as much a computational intractability frontier as it is a math problem. We know how to do approximately optimal prediction, but if you have to throw a supercomputer at the calculation and wait until the heat death of the universe to get an answer (which is the essential reality) then it has no value. But if you can grind out small improvements at the prediction frontier on a tractable amount of computing hardware due to algorithm advances and mathematical improvements in more narrow cases, then you have an almost unbounded greenfield to work with and these improvements will generalize well across diverse markets.
Predictability is defined in terms of probability distributions, and a price X is typically defined to be drawn from something like Xₜ₊₁ ~ Xₜ + N(μ,σ). The purpose might be to quantify some property of μ or σ or something like that. It means "X₀.ₜ does not hold any information about Xₜ₊₁". The assumption is that any price move reflects new information. It might be a reasonable model for various purposes.
But if this model was "true", RenTec would not work, and the "efficient market hypothesis" is invalidated. Which seems plainly obvious.
Ok, so if the market is not efficient, then it's actually a game (poker-like?) and zero sum. Academically, unthinkable thoughts.
> And then, the academic perspective is that prices should be modelled as random walks, though you may talk/learn about things such as "trend" and volatility.
The math involved in finance and economics always seems way behind that of other fields. The problem is that the other fields with more advanced math are so deep in theory that the people working in those areas are often either unaware of the potential real world applications of their work, or they are simply not interested in it (I’ve noticed there seems to be little overlap between the type of personality inclined to explore abstract theories as its own reward and the type of personality that prefers to apply existing knowledge to a real world problem).
> Suggesting that hidden variables/states/transitions can be learned from historical data is usually considered pseudo-scientific.
I mean, there’s a definitive answer to the question of stock market predictability. Unfortunately, it’s also uncomputable: if the conditional Kolmogorov complexity of a stock price time series given relevant auxiliary data is less than the size of the time series data (roughly speaking), then the stock price is predictable to some degree. Otherwise, it’s not.
I would be extremely skeptical if anyone claimed that stock price is truly Kolmogorov-random. However, I also think no single trading group’s algorithms (and data) are sufficiently more advanced than any other group’s to the point where algorithmic arbitrage is obvious to the market (or maintainable over a sufficiently long time period). I would not be surprised though if a sudden ML breakthrough destabilizes the entire market at some point in the near future when one group does in fact realize a step function improvement in their algorithms.
To educate a man in mind and not in morals is to educate a menace to society. -- Teddy Roosevelt (I'm speaking generally and not saying that Jim Simon was bad)
IIRC, their ELI5 testimony to the Senate (not the one you linked elsethread) is that RenTec's long-term strategy is to make a small profit over many many transactions. Versus buy and hold, or value invest, or whatever.
I don't understand anything about finance. That said, it sounds to me that RenTec is (or portraying themselves as) a classic volatility based hedge fund.
(A good friend is a hedge fund trader. He has tried to explain the maths to me a few times. Something something about Brownian motion, NPV, predicting herd migration. Alas, I am but a simple bear.)
But all their data collection gives me pause. I do think they they're better at spotting market signals. Like using FourSquare check-in location data to predict retail performance. Like using a VPN to spy on users to spot emerging competing startups.
My pet theory is that RenTec's play is restraint, to be patient slow capital. Even though they (probably) have data for bonanza predictions, like your NVDA example, they some how have the discipline to eek out modest profits, preferring consistency over riding the tiger.
And yeah, it’s not remotely surprising they’ve made trades like selling NVDA at $700. Judging a fish by its ability to fly etc. RenTech doesn’t work by picking stocks based on industrial trends or anything — as far as we know that sort of stuff is literally not even an input.
“Predict the market” is an insufficiently defined phrase to argue over. Strictly speaking, of course they cannot “predict the market.” You’re describing a time machine or a crystal ball, and no, they don’t have either.
What they can do, as demonstrated consistently since approximately their founding, is eek out tiny, repeatable edges on the market and exploit them at rather large scale in a variety of market conditions for dramatically longer periods than anyone else.
That is in practice the most consistently-slightly-correct market prediction anyone has ever achieved.
Maybe then it's real magic. Everybody knowns The Goetic Circle of Solomon cannot have more than 72 different demons. :-)
Are you familiar with "Fooled by Randomness" by Taleb? Here is one of the simple tricks discussed there. The details are of my own writing, the mechanics of it are as described.
How you can easily implement a Hedge Fund to beat the market and become a rich investment manager, that will show up every day on CNBC.
Step 1: Choose randomly 20 stocks from the hundreds in the NASDAQ and the SP500. Do this hundred times and create 100 funds.
Step 2: Let the funds run for a while and keep closing the worst performing
Step 3: At the end you will end with one or two that beat all market indexes
Step 4: After 3/5 years publish a full announcement page on the FT and Wall Street Journal, explaining how your Hedge Fund has consistently beat the market.
Step 5: Invite people to give you more money to manage, due to your amazing expertise.
Make sure to charge a 5% management fee. Show up on CNBC once in a while
for free, for increased exposure.
Step 6: Setup a Foundation to make sure you enjoy your billions tax free...
Another way to do, if you morals are let's say, more flexible, is setup several funds, trade your main ( profitable fund) again the other funds ;-)
Okay, so is that what you’re claiming they’re doing?
If so, please share your evidence.
If not, then I’m not sure what is the point of this conversation.
There are plenty of hypothetical ways to beat the market and they make for the same quality of conversation as a drunk uncle’s “brilliant” day trading strategy that he just needs a few bucks to execute. Carry on all day if you wish, no one will be better off for it.
:) If I crack the Renaissance Technologies fund algorithm I don't think you will be the first one I will notify...:-)
However, I know where I would start to investigate. The employees of the fund, do some of biggest political donations in the USA. Astute investigative journalists could start to look there. Are they doing to avoid scrutiny of their activities? If you cracked the market why such a high level of donations? Philanthropy? Then open your fund...
...this is another misintepretation. You're looking at a list of donations by employees, who presumably don't fall under the same political persuasion. We know this for a fact given that Jim Simons and Robert Mercer are themselves on opposite ends of the spectrum (and each was very politically active).
Anyway yeah, I'm sure RenTech is just lacking scrutiny. Surely no astute investigative journalist has thought to look their way.
> this is another misintepretation. You're looking at a list of donations by employees.
They have only 300 employees, and they are all partners apparently.
Financial investigative journalism is dead in the US. Journalists are paid badly, and work for the multinationals they should scrutinize.
In Europe the FT tried to pretend they were still a respectable financial publication, but shit their pants when the German financial regulator threaten them. They learned to be quiet since...
If you're accusing RenTech of something, be direct about it.
This is a typical conspiracy theorist style. Always pointing out "look there..." without ever saying what they're talking about, because there is actually nothing there.
> If you're accusing RenTech of something, be direct about it.
Oh now trying to explain the algos, of a fund of 300 employees, who never has losses, and makes some of the biggest political donations in the US...Is accusing?
What happened to intellectual curiosity? Don't look up?
And 13f's only show their long positions on that day, delayed by up to 45 days. And from my understanding intraday was primarily what medallion did, 1 day to 2 weeks. No high frequency, no long term.
> David: This may be the craziest part about the whole TSMC founding story. I'm 99.9% sure, Ben, you do not know this. Do you know what the pre-money valuation was on TSMC?
> Ben: No, I couldn't find that anywhere.
> David: It was $0. Morris Chang got no equity. Zero.
> Ben: So 100% of the company was owned by the investors?
> David: Fifty percent by the government and the other 50% were owned by the investors. Morris got nothing.
> Ben: And just got to keep his salary.
> David: He was a government employee.
> Ben: Wow.
> David: There by the grace of the government.
> Ben: Oh my God.
> David: Isn't that unbelievable? This is so the opposite of Silicon Valley.
I wonder if there are few things going on here. If TSMC was a success then he would be well enough rewarded and not everyone wants to be a billionaire. Then there is the reduced influence from having no equity. I guess if Chang was dependent on government support and confident in his ability to influence government then having some equity makes little difference?
Quick 5 minute video on downloading and running Hugging Face language models in GGUF format (quantized by TheBloke) with Ollama on your local machine and checking GPU consumption with asitop (Apple Silicon Mac Top).
I only clicked on the comments to find out if anyone had posted this. It is also a favorite of mine and I don't even really write in Lisp! Though it has actually inspired me to start using emacs and fiddle with elisp and CL a bit.
The article is factually incorrect on many fronts. There were only 3 reactors running at last. Nuclear was never profitable, the country shouldered most of its costs (esp for radioactive waste storage)
The renewable energy transition was blocked and delayed by the same Merkel govt. Germany was once leading in wind and PV energy and all those industries were destroyed and almost 100k jobs lost. 60bn subsidies per year go to fossil fuel companies including coal mining and burning. Often under the disguise of job retention. All through industrial lobbying and strong unions in the fossil fuel industries.
Germany's renewable industry also was never profitable, if you look at the whole country. It was always propped up by huge subsidies, and when those subsidies where scaled back (as planned and announced beforehand), the whole rent-seeking industry folded and business went to China.
It is hard to argue about energy cost in Germany, because all energy sources are a mixture of privately financed and public/state-subsidized, but always in different, complicated and frankly weird ways. Things are easier to see in countries where either practically everything is done by the state (e.g. France) or everything is more free-market (e.g. the US).
It would be more profitable than burning coal, were externalities priced in to fossil fuel energy generation as they are with nuclear energy.
With nuclear, every kWh generated incorporates the cost of decommissioning, storage, truly baroque safety compliance, and all the rest.
With coal, every kWh incorporates the cost of… coal. What’s the plan for the radioactive ash? Dump it. Somewhere. Wherever. What’s the plan for the atmosphere? Screw it. What’s the plan for decommissioning? File for bankruptcy and let the state deal with it.
How much CO2 german fossil fuel energy production is putting the air compared to the others?
Maybe Germany could go energy martial law, that to build and deploy wind/hydro/solar/etc production plants to shut down their fossil fuel energy production plants. In the meantime, if France is ramping up on nuclear energy, Germany will get some from the EU electric grid.
Huh, maybe I missed something, my understanding was, that they have no new plants ready to go anytime soon as they just left their old ones, which now have the issues they have. Do you have a link?
You aren't considering the phase-out costs, particularly mortality and environmental costs from the fossil fuels that Germany has now doubled down on.
Green Party activists often demonize nuclear while calling for renewables and their inevitable companion of gas peaking plants. Activists pretend "renewables + gas" is green and clean using insane cognitive dissonance, while claiming that renewables can power 100% of a country's energy needs -- a farcical claim not supported in any country.
Have fun
https://gist.github.com/jexp/8d991d1e543c5a576a3f1ee70132ce7...