Initialization of weights

 

• The weights 𝑊[𝑙]${\mathrm{<br>}}^{}$ should be initialized randomly to break symmetry.
• However, it's okay to initialize the biases 𝑏[𝑙]${\mathrm{<br>}}^{<br>}$ to zeros. Symmetry is still broken so long as 𝑊[𝑙]${\mathrm{<br>}}^{}$ is initialized randomly.
• Initializing weights to very large random values doesn't work well.
• Initializing with small random values should do better.