The success of convolutional neural networks is often attributed to locality and weight sharing. However, how these two properties separately shape learning, and how their effects depend on the spatial correlations of natural images, remain poorly understood. The authors address these questions in a teacher-student model with a convolutional teacher acting on P input positions with correlation c between positions.
They compare convolutional and locally connected students trained by online stochastic gradient descent, and derive ordinary differential equations that describe their dynamics exactly in the limit of large feature dimension D. For uncorrelated positions (c = 0), the two students follow the same learning trajectory, with the convolutional student learning exactly P times faster. The paper also finds that locality affects the sample complexity: a student that matches the kernel size of the teacher recovers it from O_D(D) samples, whereas O_D(D log D) samples are needed with a larger kernel.
For c > 0, spatial correlations can reduce the advantage of weight sharing. The asymptotic error of the convolutional student grows with c, whereas that of the locally connected student is independent of c and overall lower when evaluated at the same learning rate. When the learning rate is instead tuned to the training horizon, however, the locally connected student wins at early times and the convolutional student at late times, with a transition that the authors predict analytically.
The analysis assumes a teacher-student model with specific correlation structure and may not directly translate to all real-world image distributions. The regime where learning rate is tuned to training horizon shows a time-dependent transition between which architecture is preferable, which depends on the specific task and data correlations.
Read the paper on arXiv