The boundary of neural network trainability is fractal (AI summary)

Key Points

1. The paper investigates the fractal nature of the boundary between neural network hyperparameters that lead to stable and divergent training. The boundary is found to be fractal over more than ten decades of scale in all tested configurations.

2. Various experimental conditions were examined, including full batch training with tanh and ReLU nonlinearities, training a deep linear network, minibatch training, training on a dataset of size 1, and visualizing training success for different subsets of hyperparameters.

3. The experiments involved training a one hidden layer network with inputs x ∈ Rn and parameters W0 ∈ Rn×n, W1 ∈ R1×n on a mean squared error loss, and exploring different hyperparameter configurations.

4. Visualizations of the bifurcation boundary between hyperparameters that lead to successful and unsuccessful training of neural networks show fractal behavior in all experimental conditions.

5. Fractals were observed in the boundary between hyperparameters that result in successful or failed neural network training, suggesting relevance for meta-learning in optimizing hyperparameters.

6. The paper explores the generation of fractals in high-dimensional spaces and their potential dependence on properties of the generating function in neural network training.

7. Fractals were also observed in stochastic neural network training, indicating that the fractal nature of the boundary is not corrupted by minibatch noise.

8. The research suggests that fractals defined by neural network hyperparameters naturally exist in three or more dimensions and may be more organic and less symmetric compared to traditional Mandelbrot and Julia sets.

9. The paper concludes by noting the enjoyable nature of the project and potential implications for understanding chaotic meta-loss landscapes in neural network training.

Summary

The paper explores the similarities between the iterative function associated with fractals and the boundary between stable and divergent neural network training hyperparameters. The author experimentally examines the fractal nature of this boundary in various neural network configurations and finds that the boundary is fractal over more than ten decades of scale in all tested configurations. The paper visualizes the bifurcation boundary between hyperparameters leading to successful and unsuccessful training of neural networks and demonstrates that this boundary is fractal in various experimental conditions, including different network nonlinearities, training methodologies, and hyperparameter subsets.

The experiments involve training a one-hidden-layer network with inputs and parameters on mean squared error loss, and the visualizations show the fractal nature of the boundary between trainable and untrainable neural network hyperparameters. The paper also discusses the comparison between popular fractals and neural network training, explores fractal behaviors in high-dimensional spaces, and discusses the implications of fractal structures for meta-learning. The author notes the surprising presence of fractals in stochastic iterated functions and discusses the challenge of extending fractals to higher dimensions.

Additionally, the experiments suggest that meta-loss landscapes associated with neural network hyperparameters are chaotic and extremely sensitive to small changes, as they exhibit fractal properties. The exploration of the bifurcation boundary and the fractal nature of neural network hyperparameters in different experimental conditions contributes to understanding the complex dynamics of neural network training and its resemblance to fractal boundaries in other computational processes.

Reference: https://arxiv.org/abs/2402.06184