"The cost of training the model from scratch using our code is about $50k."
Still a substantially steep curve for a bootstrapping startup. It's something I continually run into myself. I have somewhat of a weekend project trying to build a search engine but man ... the cost of just the SSDs and GPUs is daunting on a regular salary. As the complexity of these models grows, so does the barrier to entry for a regular joe like me; which is a shame I think. I know in the US it's fairly normal for a data scientist to pull 100k+ / year, but in the Netherlands salaries pretty much stall at 40k (and angel investment in IT/AI is at an all time low). More generally I fear this will become a bit of a sociotechnical issue if complex AI models will be out of reach for entire economies (especially for cases like language because not everyone speaks English and "minor" languages like those in EU countries are a massive market to explore, yet hard to get into).
It may be high time to discuss what AI policy has actually done so far. From what I can tell, not much other than letting social scientists get in on the deep learning gravy train.
Meanwhile, misuses of ML are proliferating without limits, and 'AI policy' is apparently mostly used as a fig-leaf to collect good-will, marketing, and buy a seat at the table for future regulations. As usual, regulations will protect incumbents, so my as-usual cynical read is that OpenAI's policy interests are about protecting its own future interests. From that perspective, the entire GPT-2 stunt was highly effective.
Now depending on your outlook, that may be an argument that we need more people in policy, or fewer. Or different ones.
It's worth noting that, like other attempted replications, the perplexities of this model mostly aren't as good as GPT-2. Given that the title of the GPT-2 paper was "Language Models are Unsupervised Multitask Learners," I'd be interested in a lot more metrics before I'd believe GPT-2 has actually been replicated. Especially because every other time someone says this, metrics show otherwise. Until then, this is just a really big model.
What's it take to actually run a model like this, hardware-wise? I've been toying around with a gpt2 discord bot (https://github.com/ScottPeterJohnson/gpt2-discord) using just a CPU calculation, and already it takes up 2 GB RAM (and is slow obviously) on the 345M model. I might be able to get the 774M model running, but there's no way I can afford the full model, assuming linear RAM use. And that's just for CPU compute, I can't even begin to imagine how expensive GPU would be.
Twitter thread by a Research Scientist at OpenAI addressing OpenAI's policies in response to this discussion here: https://twitter.com/Miles_Brundage/status/116495932263331840...
Without context, this article reads like something generated by machine learning.
They spent $500k replicating it. But sure, you can do it too /s
It's fun to feed it the first paragraph of the Sokal affair paper:
Prompt:
There are many natural scientists, and especially physicists, who continue to reject the notion that the disciplines concerned with social and cultural criticism can have anything to contribute, except perhaps peripherally, to their research. Still less are they receptive to the idea that the very foundations of their worldview must be revised or rebuilt in the light of such criticism. Rather, they cling to the dogma imposed by the long post-Enlightenment hegemony over the Western intellectual outlook, which can be summarized briefly as follows: that there exists an external world, whose properties are independent of any individual human being and indeed of humanity as a whole; that these properties are encoded in ``eternal'' physical laws; and that human beings can obtain reliable, albeit imperfect and tentative, knowledge of these laws by hewing to the ``objective'' procedures and epistemological strictures prescribed by the (so-called) scientific method.
Completion:
Before one can explore these ``objective'' principles and procedures, one must do something important, namely, invent the objectivity -- or, more precisely, the objectivity-transcendent properties -- of the ``subjective'' functions of ``man" (as a species of ''objective'', or something-in-itself-independent, human, being). Since the 19th century, this has been done on a literal or phenomenological level, the metanarrative of psychoanalysis. However, the ultimate test of what constitutes an objective fact or structure is the statistical discovery of reliable facts or structures residing in the human mind or body. That is what the discipline of psychology is for. The distinction between the subjective and the objective is therefore necessary for the discipline of psychological science, which is by no means a waste of time, which is today a flourishing field of scientific inquiry, and which might contribute a great deal toward furnishing a genuinely complete theoretical understanding of human behavior and psychical experience.
In short, the philosophical implications of the post-Enlightenment philosophy of the sciences and of psychical psychology are -- when considered in the context of both human existence and human thought -- deeply unsatisfactory. Once we abandon the most basic postulate of scientific rationality -- that subjective knowledge and belief and behavior are not to be said to be existent but to be, not to be, knowable -- there is nothing left to suggest that the content of human knowledge and experience is dependent upon our conscious perceptions or personalities. We might describe that content as ineffable (it is irreducible to language, outside the experience of individual thinkers), or inconceivable (that which is not one's own consciousness, but is beyond the realm of conscious experience). In either case, there is no genuine need for an objective world, or, at least, no need for an objective foundation. In that sense, it is erroneous to assert that the scientific standpoint is a paradigm of objective reality; rather, it is the antidote to the paradigm of irreducible subjectivity and self-consciousness.
Unsupported by this analysis of the problem of scientific knowledge, the field of psychical science is often rejected by younger scholars, for whom the study of the human condition and its ineffable psychology is regarded as a particularly enticing but very unapproachable form of mysticism. They regard this section of the theory and practice of modern scientific inquiry as a kind of metaphysical theory-cracking rather than as a vehicle for the discovery of objective facts. Such a view can only be sustained on the basis of the strictest adherence to the doctrines of empiricism and fundamental materialism, respectively, and it is by no means fruitful for the real progress of science. Let me explain.
The residual emphasis on irreducible subjectivity in modern Western thought (and only by the residual emphasis) is a legacy of the metaphysical concept of objective reality which, to my mind, is still embraced by the cognitivist movement of the 20th century. It is an unfortunate legacy. Irreducible subjectivity and subjectivity-dependent, subjective perceptions were at the root of the metaphysics of Buddhism, and these beliefs now constitute the dominant conception of reality in contemporary Western philosophy, particularly Kantian, Descartesian, and Hegelian thought, which are each deeply indebted to Aristotelian theories of subjectivity.
While both these views of reality are patently absurd, and since the scientific revolutionary of the middle 19th century was able to disregard or repudiate them, the followers of Kant and Descartes were able to maintain that there is no need for a foundation for the science of knowledge. They could do this because they held to a primitive, problematic conception of objectivity, based on the notion of an objective, external world, in which human consciousness, thus independent of any particular body, mind, or culture, was inchoate, mutable, and subject to change or speculation. There was therefore no need to search for a theory of experience. Science and experience were simply different approaches, of which each was as good as the other, and they both...
Prompt:
The jig is up. And what now? Where do we go from here?
Completion:
Where Do We Go From Here: In the aftermath of the fall of the German Republican party, we now have a significant degree of instability across the earth’s systems of government and finance. The almost complete collapse of systemic forces in the Eurozone and limited success at stabilizing the system means the question is not if but when, what do we do next?
The answer is simple. We must move beyond the localized, bubble-like, and short-termist “get involved,” tactic of getting into the scene and trying to control it in some way. We have to come up with a way of shifting the socio-political power in the world, the prime place for transformation is worldwide at the supra-system level and not just the economy and finance. We must cast out the old dominated system, of which we have been just a part and recognize that we need a new dominant system that serves human interests, and the meta-level global system must serve human interests.
The fact that the global status quo is collapsing of its own weight shows us that the system is structured in such a way that the group of big players who have dominated and still dominate, are in ever-decreasing danger of losing both power and integrity.
The question now is, how? How do we avoid degeneration into chaos and conflict when the anarchic nature of the system leads inevitably to greater and greater competition among and frustration and anger in the younger generations?
This is a society, this is a planet and we live in the first global century of human history, which the young will pass from generation to generation in the next twenty years, or perhaps not. When we see the events of the last weeks and days, you can just imagine what will happen to this planet, to this planet and human society in the century ahead, and you can just imagine what the future will bring to this and subsequent generations.
When history describes the past, it sees the collapse of an old political establishment, of the traditional hierarchies of power, of society, of economics and finance. It sees a collapse in the old order of power and in the equilibrium it has created, which was grounded in constant growing jobs and the prosperity it produced. We are in the middle of a permanent expansion of capitalism, which also creates ever-growing wealth and prosperity for a small population of wealthy earners, while social polarization and inequality increase and older people depend on each other more and more desperately.
The forward and downward momentum of all these forces has created a situation in which there is almost no limit to the volume of the day to day, or minute to minute production and consumption, and in which there is no single concern about the future of the planet Earth. We have become so insatiable, in need, addicted to this ever increasing appetite for consumer goods, that we destroy the planet with it.
You see this just from what we feed our children, the choices we make, and the products we consume. You see it in our greedy attempts to buy as much as we can, even if it leads to ecological ruin. You see it in our drive to consume new and ever more lavish luxury products, materials, tools, devices, insatiable lifestyles, modern-day imperialism, racism, cynicism, competition, greed, consumerism, hubris, and endless pursuit of personal ambitions and leisure.
See how when push comes to shove, the social and economic growth created by the continued expansion of capitalism is now a life or death matter. See how the political establishment has failed us, all of us, and how we turned in desperation to another self-serving-self-protective-petty, self-interested-philistine mass-mediator, in the form of Mr. Romney, in order to maintain the old sources of power, to make the old social structures fit to serve human needs and the system could be kept going.
And now he has packed his bags and wants to leave, so there we are, stuck here with those of us who have found a way to provide for ourselves and live peacefully and prosperously, without the brutality and violence visited on us by politicians and corrupted systems. That is, unless we fix these broken systems and deliver an alternative based on human needs and human compassion.
How do we do it? How do we get there? Stay tuned and we’ll let you know.<|endoftext|>This exciting book is an overview of a phenomenon that started in the 1970’s, and became the most spectacular of all the urban myths. It combines all things the paranormal in this feature length book, from scientists to aliens to experimental reports.
William Kean is an astronomer working for NASA. One evening he is on his way to a remote overlook on a Martian hill. Suddenly, he is teleported to the top of a fifty story building, two thousand feet in the air. The building
I skimmed the article and stated reading the last paragraph to get an idea of what it was about. I was v confused..
It sounds like all the drama OpenAI made about not releasing the model was all just marketing. $50,000 is nothing for a nation-state or even just a motivated third party. I had always assumed OpenAI had spent well into the 6 or even 7 figures to train the full model.
MSFT has sort of invested $1 billion into OpenAI so I guess it worked!