The Importance of (Exponentially More) Computing Power

It is undeniable that the worldwide computer industry's center is the US, specifically in Silicon Valley. Much of the reason for the success of Silicon Valley had to do with Moore's Law: the observation by Intel co-founder Gordon Moore that the number of transistors on a microchip doubled at a rate of approximately every two years. According to the International Technology Roadmap for Semiconductors, Moore's Law will end in 2021. How can we rethink computing technology to res...

The question how complex systems become more organized and efficient with time is open. Examples are, the formation of elementary particles from pure energy, the formation of atoms from particles, the formation of stars and galaxies, the formation of molecules from atoms, of organisms, and of the society. In this sequence, order appears inside complex systems and randomness (entropy) is expelled to their surroundings. Key features of self-organizing systems are that they are ...

Training advanced AI models requires large investments in computational resources, or compute. Yet, as hardware innovation reduces the price of compute and algorithmic advances make its use more efficient, the cost of training an AI model to a given performance falls over time. To analyze this phenomenon, we introduce compute (investment) efficiency, which relates training compute investment to the resulting AI model performance. We then present a conceptual model of increase...

This paper introduces a method for linking technological improvement rates (i.e. Moore's Law) and technology adoption curves (i.e. S-Curves). There has been considerable research surrounding Moore's Law and the generalized versions applied to the time dependence of performance for other technologies. The prior work has culminated with methodology for quantitative estimation of technological improvement rates for nearly any technology. This paper examines the implications of s...

We show that, in the context of Moore's Law, overall productivity can be increased for large enough computations by `slacking' or waiting for some period of time before purchasing a computer and beginning the calculation.

Are the sciences not advancing at an ever increasing speed? We contrast this popular perspective with the view that science funding may actually see diminishing returns, at least regarding established fields. In order to stimulate a larger discussion, we investigate two exemplary cases, the linear increase in human life expectancy over the last 170 years and the advances in the reliability of numerical short and medium term weather predictions during the last 50 years. We arg...

Empirical research has shown performance improvement of many different technological domains occurs exponentially but with widely varying improvement rates. What causes some technologies to improve faster than others do? Previous quantitative modeling research has identified artifact interactions, where a design change in one component influences others, as an important determinant of improvement rates. The models predict that improvement rate for a domain is proportional to ...

Three factors drive the advance of AI: algorithmic innovation, data, and the amount of compute available for training. Algorithmic progress has traditionally been more difficult to quantify than compute and data. In this work, we argue that algorithmic progress has an aspect that is both straightforward to measure and interesting: reductions over time in the compute needed to reach past capabilities. We show that the number of floating-point operations required to train a cla...

An indispensable part of our lives, computing has also become essential to industries and governments. Steady improvements in computer hardware have been supported by periodic doubling of transistor densities in integrated circuits over the last fifty years. Such Moore scaling now requires increasingly heroic efforts, stimulating research in alternative hardware and stirring controversy. To help evaluate emerging technologies and enrich our understanding of integrated-circuit...

The demand for computing is continuing to grow exponentially. This growth will translate to exponential growth in computing's energy consumption unless improvements in its energy-efficiency can outpace increases in its demand. Yet, after decades of research, further improving energy-efficiency is becoming increasingly challenging, as it is already highly optimized. As a result, at some point, increases in computing demand are likely to outpace increases in its energy-efficien...