Training a 2 layer Neural Network using SVD-generated weights

Created in January 31, 2018

2018   ·   machine_learning   presentation   ·   neural_network   poseter   machine_learning

Abstract The objective of this study is to figure out a good initial weight for training a 2 layered neural network aimed at MNIST handwritten digit recognition. Conventionally, initial weights are set up with random values near zero. This made hard to understand where the weights converge. This study is going to introduce a deterministic weight initialization method based on the training dataset. By applying singular value decomposition on the training set, principal components are obtained. These principal components are going to be used for the first layer’s weight. Then we find the relevant weight for the second layer. At the end, the performance of 2 neural networks constructed by random weight initialization and the new pc weight initialization is going to be compared.

View poster



Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • Wasserstein Proximals Stabilize Training of Generative Models and Learn Manifolds
  • Lipschitz-Regularized Gradient Flows and Latent Generative Particles
  • Lipschitz Regularized Gradient Flows and Latent Generative Particles
  • Sample generation from unknown distributions - Particle Descent Algorithm induced by (f, Γ)-gradient flow
  • Python ODE solving tutorial