ArXiv · 2026
The statistical accuracy of neural networks depends on both their approximation power and the complexity of the class fitted from data. While increasing network size is a natural way to improve approximation, parameter magnitude provides another resource whose role must be quantified in both respects. We establish a sharp width–magnitude tradeoff at fixed depth using one elementary bounded 1-Lipschitz Dyadic–Triangular Activation. For the unit β-Hölder ball on [0,1]ᵈ with 0<β≤1, the optimal Lᵖ approximation error for 0<p><∞ is of order [N²log(eNT)]^(-β/d) when the network width satisfies N≥2d+3 and the parameter magnitudes are bounded by T≥1. Matching lower bounds hold for every fixed globally Hölder activation; its Hölder exponent affects the constants but not the rate. Under bounded design densities and independent centered sub-Gaussian noise, approximate least squares over the full clipped class at depth 23 attains the classical Hölder minimax risk O(M^(-2β/(2β+d))) without logarithmic loss whenever N²log(eNT)asymp M^(d/(2β+d)), where M is the sample size. This yields a continuum of statistically optimal choices, ranging from unit parameter radius to fixed network size. At fixed size, four hidden layers with at most 8d+7 nonzero parameters give a near-optimal radius, while six layers with at most 8d+27 attain the optimal order log T=O(η^(-d/β)) at approximation error η. The same decoding method also yields fixed-size Transformer approximation.
Try inveni