It sounds like the op is trying to provision and load 150GB in a reasonably fast manner. Once loaded, presumably any of the usual suspects will be fast enough. It’s the up front loading costs which are the problem.
Anyway, I’m curious what kind of data the op is trying to process.
I am trying to load and serve the Microsoft academic graph to produce author profile pages for all academic authors! Microsoft and google already do this but IMO they leave a lot to be desired.
But this means there are a hundred million entities, publishing 3x number of papers and a bunch of metadata associated. On redshift I can get all of this loaded in minutes and takes like 100G but Postgres loads are pathetic comparatively.
And I have no intention of spending more than 30 bucks a month! So hard problem for sure! Suggestions welcome!
Anyway, I’m curious what kind of data the op is trying to process.