Storage Capacity of the Tilinglike Learning Algorithm

Upper and lower bounds for the typical storage capacity of a constructive algorithm, the Tilinglike Learning Algorithm for the Parity Machine [M. Biehl and M. Opper, Phys. Rev. A {\bf 44} 6888 (1991)], are determined in the asymptotic limit of large training set sizes. The properties of a perceptron with threshold, learning a training set of patterns having a biased distribution of targets, needed as an intermediate step in the capacity calculation, are determined analyticall...

We demonstrate that the fraction of pattern sets that can be stored in single- and hidden-layer perceptrons exhibits finite size scaling. This feature allows to estimate the critical storage capacity \alpha_c from simulations of relatively small systems. We illustrate this approach by determining \alpha_c, together with the finite size scaling exponent \nu, for storing Gaussian patterns in committee and parity machines with binary couplings and up to K=5 hidden units.

The performance of large neural networks can be judged not only by their storage capacity but also by the time required for learning. A polynomial learning algorithm with learning time $\sim N^2$ in a network with $N$ units might be practical whereas a learning time $\sim e^N$ would allow rather small networks only. The question of absolute storage capacity $\alpha_c$ and capacity for polynomial learning rules $\alpha_p$ is discussed for several feed-forward architectures, th...

The aim of this thesis is to compare the capacity of different models of neural networks. We start by analysing the problem solving capacity of a single perceptron using a simple combinatorial argument. After some observations on the storage capacity of a basic network, known as an associative memory, we introduce a powerful statistical mechanical approach to calculate its capacity in the training rule-dependent Hopfield model. With the aim of finding a more general definitio...

It has been known for a long time that the classical spherical perceptrons can be used as storage memories. Seminal work of Gardner, \cite{Gar88}, started an analytical study of perceptrons storage abilities. Many of the Gardner's predictions obtained through statistical mechanics tools have been rigorously justified. Among the most important ones are of course the storage capacities. The first rigorous confirmations were obtained in \cite{SchTir02,SchTir03} for the storage c...

We derive the Gardner storage capacity for associative networks of threshold linear units, and show that with Hebbian learning they can operate closer to such Gardner bound than binary networks, and even surpass it. This is largely achieved through a sparsification of the retrieved patterns, which we analyze for theoretical and empirical distributions of activity. As reaching the optimal capacity via non-local learning rules like backpropagation requires slow and neurally imp...

We investigate a quantum perceptron implemented on a quantum circuit using a repeat until method. We evaluate this from the perspective of capacity, one of the performance evaluation measures for perceptions. We assess a Gardner volume, defined as a volume of coefficients of the perceptron that can correctly classify given training examples using the replica method. The model is defined on the quantum circuit. Nevertheless, it is straightforward to assess the capacity using t...

In this paper we revisit one of the classical perceptron problems from the neural networks and statistical physics. In \cite{Gar88} Gardner presented a neat statistical physics type of approach for analyzing what is now typically referred to as the Gardner problem. The problem amounts to discovering a statistical behavior of a spherical perceptron. Among various quantities \cite{Gar88} determined the so-called storage capacity of the corresponding neural network and analyzed ...

In a recent paper ``The capacity of the Hopfield model, J. Feng and B. Tirozzi claim to prove rigorous results on the storage capacity that are in conflict with the predictions of the replica approach. We show that their results are in error and that their approach, even when the worst mistakes are corrected, is not giving any mathematically rigorous results.

This paper describes a family of probabilistic architectures designed for online learning under the logarithmic loss. Rather than relying on non-linear transfer functions, our method gains representational power by the use of data conditioning. We state under general conditions a learnable capacity theorem that shows this approach can in principle learn any bounded Borel-measurable function on a compact subset of euclidean space; the result is stronger than many universality ...