Moore's law, Wright's law and the Countdown to Exponential Space

As large-scale AI models expand, training becomes costlier and sustaining progress grows harder. Classical scaling laws (e.g., Kaplan et al. (2020), Hoffmann et al. (2022)) predict training loss from a static compute budget yet neglect time and efficiency, prompting the question: how can we balance ballooning GPU fleets with rapidly improving hardware and algorithms? We introduce the relative-loss equation, a time- and efficiency-aware framework that extends classical AI scal...

Are the sciences not advancing at an ever increasing speed? We contrast this popular perspective with the view that science funding may actually see diminishing returns, at least regarding established fields. In order to stimulate a larger discussion, we investigate two exemplary cases, the linear increase in human life expectancy over the last 170 years and the advances in the reliability of numerical short and medium term weather predictions during the last 50 years. We arg...

In this work we present the concept of guided self-replicating factory (GSFR). This factory would be established as a colony on the Moon, Mercury, Mars, or Asteroid Belt. GSFR would grow by using in situ materials in order to manufacture machines. The time it takes for GSFR to double it's mass and electric power output is called a doubling period. After about 50 doubling periods, GSFR would become a Dyson Sphere civilization, which can house $10^{16}$ people, $10^{19}\ tons$ ...

Recent research across mathematical problem solving, proof assistant programming and multimodal jailbreaking documents a striking finding: when (multimodal) language model tackle a suite of tasks with multiple attempts per task -- succeeding if any attempt is correct -- then the negative log of the average success rate scales a power law in the number of attempts. In this work, we identify an apparent puzzle: a simple mathematical calculation predicts that on each problem, th...

The paper "Critical Truths About Power Laws" (Science, 335, pp665-666) by MPH Stumpf MPH and MA Porter is commented

We study long-run progress in artificial intelligence in a quantitative way. Many measures, including traditional ones such as patents and publications, machine learning benchmarks, and a new Aggregate State of the Art in ML (or ASOTA) Index we have constructed from these, show exponential growth at roughly constant rates over long periods. Production of patents and publications doubles every ten years, by contrast with the growth of computing resources driven by Moore's Law,...

We discuss several models in order to shed light on the origin of power-law distributions and power-law correlations in financial time series. From an empirical point of view, the exponents describing the tails of the price increments distribution and the decay of the volatility correlations are rather robust and suggest universality. However, many of the models that appear naturally (for example, to account for the distribution of wealth) contain some multiplicative noise, w...

In the practical work of websites popularization, analysis of their efficiency and downloading it is of key importance to take into account web-ratings data. The main indicators of website traffic include the number of unique hosts from which the analyzed website was addressed and the number of granted web pages (hits) per unit time (for example, day, month or year). Of certain interest is the ratio between the number of hits (S) and hosts (H). In practice there is even used ...

We present a smoothly broken power law functional form (that we refer to as a Broken Neural Scaling Law (BNSL)) that accurately models & extrapolates the scaling behaviors of deep neural networks (i.e. how the evaluation metric of interest varies as amount of compute used for training (or inference), number of model parameters, training dataset size, model input size, number of training steps, or upstream performance varies) for various architectures & for each of various tas...

Considering a broad family of technologies where a measure of performance (MoP) is difficult or impossible to formulate, we seek an alternative measure that exhibits a regular pattern of evolution over time, similar to how a MoP may follow a Moore's law. In an empirical case study, we explore an approach to identifying such a composite measure called a Figure of Regularity (FoR). We use the proposed approach to identify a novel FoR for diverse classes of small arms - bows, cr...