ID: 2112.09420

A random energy approach to deep learning

December 17, 2021

View on ArXiv
Rongrong Xie, Matteo Marsili
Condensed Matter
Computer Science
Statistics
Disordered Systems and Neura...
Machine Learning
Machine Learning

We study a generic ensemble of deep belief networks which is parametrized by the distribution of energy levels of the hidden states of each layer. We show that, within a random energy approach, statistical dependence can propagate from the visible to deep layers only if each layer is tuned close to the critical point during learning. As a consequence, efficiently trained learning machines are characterised by a broad distribution of energy levels. The analysis of Deep Belief Networks and Restricted Boltzmann Machines on different datasets confirms these conclusions.

Similar papers 1

EnergyNet: Energy-based Adaptive Structural Learning of Artificial Neural Network Architectures

November 8, 2017

91% Match
Gus Kristiansen, Xavi Gonzalvo
Machine Learning

We present E NERGY N ET , a new framework for analyzing and building artificial neural network architectures. Our approach adaptively learns the structure of the networks in an unsupervised manner. The methodology is based upon the theoretical guarantees of the energy function of restricted Boltzmann machines (RBM) of infinite number of nodes. We present experimental results to show that the final network adapts to the complexity of a given problem.

Find SimilarView on arXiv

Energy-based Dropout in Restricted Boltzmann Machines: Why not go random

January 17, 2021

90% Match
Mateus Roder, Rosa Gustavo H. de, Albuquerque Victor Hugo C. de, ... , Papa João P.
Machine Learning
Artificial Intelligence

Deep learning architectures have been widely fostered throughout the last years, being used in a wide range of applications, such as object recognition, image reconstruction, and signal processing. Nevertheless, such models suffer from a common problem known as overfitting, which limits the network from predicting unseen data effectively. Regularization approaches arise in an attempt to address such a shortcoming. Among them, one can refer to the well-known Dropout, which tac...

Find SimilarView on arXiv

A Probabilistic Representation of Deep Learning

August 26, 2019

89% Match
Xinjie Lan, Kenneth E. Barner
Machine Learning
Machine Learning

In this work, we introduce a novel probabilistic representation of deep learning, which provides an explicit explanation for the Deep Neural Networks (DNNs) in three aspects: (i) neurons define the energy of a Gibbs distribution; (ii) the hidden layers of DNNs formulate Gibbs distributions; and (iii) the whole architecture of DNNs can be interpreted as a Bayesian neural network. Based on the proposed probabilistic representation, we investigate two fundamental properties of d...

Find SimilarView on arXiv

Deep Learning the Ising Model Near Criticality

August 15, 2017

89% Match
Alan Morningstar, Roger G. Melko
Disordered Systems and Neura...
Machine Learning
Machine Learning

It is well established that neural networks with deep architectures perform better than shallow networks for many tasks in machine learning. In statistical physics, while there has been recent interest in representing physical data with generative modelling, the focus has been on shallow neural networks. A natural question to ask is whether deep neural networks hold any advantage over shallow networks in representing such data. We investigate this question by using unsupervis...

Find SimilarView on arXiv

Geometry of energy landscapes and the optimizability of deep neural networks

August 1, 2018

89% Match
Simon Becker, Yao Zhang, Alpha A. Lee
Disordered Systems and Neura...
Machine Learning
Machine Learning

Deep neural networks are workhorse models in machine learning with multiple layers of non-linear functions composed in series. Their loss function is highly non-convex, yet empirically even gradient descent minimisation is sufficient to arrive at accurate and predictive models. It is hitherto unknown why are deep neural networks easily optimizable. We analyze the energy landscape of a spin glass model of deep neural networks using random matrix theory and algebraic geometry. ...

Find SimilarView on arXiv

Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey

July 27, 2021

89% Match
Benyamin Ghojogh, Ali Ghodsi, ... , Crowley Mark
Machine Learning
Neural and Evolutionary Comp...
Data Analysis, Statistics an...
Machine Learning

This is a tutorial and survey paper on Boltzmann Machine (BM), Restricted Boltzmann Machine (RBM), and Deep Belief Network (DBN). We start with the required background on probabilistic graphical models, Markov random field, Gibbs sampling, statistical physics, Ising model, and the Hopfield network. Then, we introduce the structures of BM and RBM. The conditional distributions of visible and hidden variables, Gibbs sampling in RBM for generating variables, training BM and RBM ...

Find SimilarView on arXiv

Ising models of deep neural networks

September 19, 2022

88% Match
Dusan Stosic, Darko Stosic, Borko Stosic
Statistical Mechanics

This work maps deep neural networks to classical Ising spin models, allowing them to be described using statistical thermodynamics. The density of states shows that structures emerge in the weights after they have been trained -- well-trained networks span a much wider range of realizable energies compared to poorly trained ones. These structures propagate throughout the entire network and are not observed in individual layers. The energy values correlate to performance on ta...

Find SimilarView on arXiv

Deep learning systems as complex networks

September 28, 2018

88% Match
Alberto Testolin, Michele Piccolini, Samir Suweis
Disordered Systems and Neura...
Machine Learning
Machine Learning

Thanks to the availability of large scale digital datasets and massive amounts of computational power, deep learning algorithms can learn representations of data by exploiting multiple levels of abstraction. These machine learning methods have greatly improved the state-of-the-art in many challenging cognitive tasks, such as visual object recognition, speech processing, natural language understanding and automatic translation. In particular, one class of deep learning models,...

Find SimilarView on arXiv

Learnergy: Energy-based Machine Learners

March 16, 2020

88% Match
Mateus Roder, Rosa Gustavo Henrique de, João Paulo Papa
Machine Learning
Computer Vision and Pattern ...
Machine Learning

Throughout the last years, machine learning techniques have been broadly encouraged in the context of deep learning architectures. An exciting algorithm denoted as Restricted Boltzmann Machine relies on energy- and probabilistic-based nature to tackle the most diverse applications, such as classification, reconstruction, and generation of images and signals. Nevertheless, one can see they are not adequately renowned compared to other well-known deep learning techniques, e.g.,...

Find SimilarView on arXiv

Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization

December 7, 2020

88% Match
Qianyi Li, Haim Sompolinsky
Machine Learning
Applied Physics

The success of deep learning in many real-world tasks has triggered an intense effort to understand the power and limitations of deep learning in the training and generalization of complex tasks, so far with limited progress. In this work, we study the statistical mechanics of learning in Deep Linear Neural Networks (DLNNs) in which the input-output function of an individual unit is linear. Despite the linearity of the units, learning in DLNNs is nonlinear, hence studying its...

Find SimilarView on arXiv