Article
Computer Science, Artificial Intelligence
Yunfei Yang, Zhen Li, Yang Wang
Summary: We study the expressive power of deep ReLU neural networks for approximating functions in dilated shift-invariant spaces and estimate the approximation error bounds based on the network's width and depth. Our results have applications in classical function spaces and show that our neural network construction is asymptotically optimal.
Article
Mathematics, Applied
Ahmed Abdeljawad, Philipp Grohs
Summary: This paper presents a framework that demonstrates the ability of deep neural networks to approximate Sobolev-regular functions in Bochner-Sobolev spaces. By using the Rectified Cubic Unit (ReCU) as an activation function, the issues caused by the nonregularity of the commonly used Rectified Linear Unit (ReLU) activation function are avoided.
ANALYSIS AND APPLICATIONS
(2022)
Article
Computer Science, Artificial Intelligence
Ingo Guehring, Mones Raslan
Summary: This study investigates the necessary and sufficient complexity of neural networks to approximate functions from different smoothness spaces under the restriction of encodable network weights. By providing a unifying framework for constructing approximate partitions of unity with fairly general activation functions, almost optimal upper bounds in higher-order Sobolev norms were derived. This work advances the theory of approximating solutions of partial differential equations by neural networks.
Article
Computer Science, Artificial Intelligence
Sean Hon, Haizhao Yang
Summary: This study establishes approximation results of deep neural networks for smooth functions measured in Sobolev norms, providing nonasymptotic error bounds explicitly characterized in terms of both network width and depth. The results hold for different activation functions and demonstrate the relationship between network complexity and approximation rates.
Article
Computer Science, Theory & Methods
Xia Liu
Summary: Constructing deep neural networks with three hidden layers using a sigmoidal activation function is the main focus of this paper, aiming to approximate smooth and sparse functions. The paper proves that by controlling the number of free parameters, these constructed deep networks can achieve the optimal approximation rate for both smooth and sparse functions. Furthermore, it is shown that neural networks with three hidden layers can overcome the saturation phenomenon observed in some architectures.
JOURNAL OF COMPLEXITY
(2023)
Article
Computer Science, Theory & Methods
Dinh Dung, Van Kien Nguyen, Duong Thanh Pham
Summary: This paper investigates non-adaptive methods for approximating functions in Bochner spaces L2(U∞, X, μ) using deep ReLU neural networks. The domain U∞ can be either R∞ equipped with the standard Gaussian probability measure, or [-1, 1]∞ equipped with the Jacobi probability measure. The approximated functions are assumed to satisfy a certain weighted ⠂2-summability condition of the generalized chaos polynomial expansion coefficients with respect to the measure μ. The convergence rate of this approximation in terms of the size of approximating deep ReLU neural networks is proven. These results are then applied to approximate the solutions of parametric elliptic PDEs with random inputs, including lognormal and affine cases. ©2023 Elsevier Inc. All rights reserved.
JOURNAL OF COMPLEXITY
(2023)
Article
Computer Science, Artificial Intelligence
Scott Mahan, Emily J. King, Alex Cloninger
Summary: In this study, the closedness of sets of realized neural networks of a fixed architecture in Sobolev spaces was examined. It was found that under certain conditions, these sets of neural networks are not closed, and can approximate non-network target functions with unbounded parameter growth. Experimental results demonstrated that neural networks can closely approximate non-network target functions through training processes.
Article
Computer Science, Artificial Intelligence
Alexander Cloninger, Timo Klock
Summary: The study shows that using deep networks to approximate two-layer composite functions can achieve near-optimal approximation rates even with high-dimensional reducing map complexity. This suggests that deep networks are faithful to the intrinsic dimension governed by the function, rather than the complexity of the function's domain.
Article
Engineering, Multidisciplinary
Thomas O'Leary-Roseberry, Umberto Villa, Peng Chen, Omar Ghattas
Summary: Many-query problems in various fields require evaluations of high-dimensional parametric maps governed by PDEs, which can be prohibitive. The proposed approach of derivative-informed projected neural networks captures the geometry and intrinsic low-dimensionality of these maps by training weights in low-dimensional layers. This method achieves greater generalization accuracy and independence of weight dimension on parametric and output dimensions.
COMPUTER METHODS IN APPLIED MECHANICS AND ENGINEERING
(2022)
Article
Engineering, Multidisciplinary
Jamie M. Taylor, David Pardo, Ignacio Muga
Summary: When using Neural Networks to solve PDEs, the choice of the loss function is crucial. This work proposes the use of a Discrete Sine/Cosine Transform to accurately compute the H-1 norm, leading to the development of the Deep Fourier-based Residual (DFR) method. The DFR method efficiently approximates solutions to PDEs, particularly in cases where H2 regularity is lacking.
COMPUTER METHODS IN APPLIED MECHANICS AND ENGINEERING
(2023)
Article
Mathematics, Applied
Carlo Marcati, Christoph Schwab
Summary: In this paper, we prove the exponential expression rates of deep operator networks (deep ONets) between infinite-dimensional spaces, which emulate the coefficient-to-solution map of linear, elliptic, second-order, divergence-form partial differential equations (PDEs). We construct deep ONets within the branch and trunk ONet architecture, which can accurately emulate the coefficient-to-solution map in the H1 norm for various PDE coefficient sets. The size of these ONets is O(| log(E)|\kappa), where E > 0 is the approximation accuracy and \kappa > 0 depends on the physical space dimension.
SIAM JOURNAL ON NUMERICAL ANALYSIS
(2023)
Article
Mathematics
Gitta Kutyniok, Philipp Petersen, Mones Raslan, Reinhold Schneider
Summary: By utilizing the low dimensionality of the solution manifold, we have constructed neural networks for approximating the solution maps of parametric partial differential equations with significantly superior approximation rates compared to classical results.
CONSTRUCTIVE APPROXIMATION
(2022)
Article
Mathematics, Applied
Woochul Jung, A. Rojas
Summary: In this paper, we provide sufficient conditions for a family of continuous maps on a metric space to approximate every continuous map. This is achieved through a new extension property on metric spaces inspired by the absolute extensor [5]. An application to continuous-time deep residual networks is included.
JOURNAL OF MATHEMATICAL ANALYSIS AND APPLICATIONS
(2023)
Article
Mathematics, Applied
Maximilien Germain, Huyen Pham, Xavier Warin
Summary: Recently proposed numerical algorithms based on neural networks have shown remarkable performance in solving high-dimensional nonlinear partial differential equations (PDEs). These algorithms rely on probabilistic representation of PDEs and utilize backward stochastic differential equations and iterated time discretization. The MDBDP scheme, a machine learning version of the least square multistep-forward dynamic programming scheme, estimates the solution and its gradient simultaneously using neural networks and stochastic gradient descent. The main theoretical contribution is the approximation error analysis of the MDBDP scheme and the deep splitting scheme for semilinear PDEs, which provide convergence rates for a class of deep Lipschitz continuous GroupSort neural networks.
SIAM JOURNAL ON SCIENTIFIC COMPUTING
(2022)
Article
Mathematics, Applied
Fabian Laakmann, Philipp Petersen
Summary: Deep neural networks using ReLU activation function can efficiently approximate solutions for various types of parametric linear transport equations, even for high-dimensional and non-smooth cases. Through their inherent compositionality, these networks can resolve the characteristic flow underlying the transport equations and achieve approximation rates independent of the parameter dimension.
ADVANCES IN COMPUTATIONAL MATHEMATICS
(2021)
Article
Computer Science, Theory & Methods
Philipp Petersen, Mones Raslan, Felix Voigtlaender
Summary: The set of functions that can be implemented by neural networks of a fixed size has many undesirable properties, such as being highly non-convex and not closed with respect to Lp-norms. This may lead to issues in the training procedure of deep learning, such as no guaranteed convergence, explosion of parameters, and slow convergence.
FOUNDATIONS OF COMPUTATIONAL MATHEMATICS
(2021)
Editorial Material
Mathematics, Applied
Nair Abreu, Christoph Helmberg, Gitta Kutyniok, Vilmar Trevisan
LINEAR ALGEBRA AND ITS APPLICATIONS
(2021)
Article
Mathematics, Applied
Fabian Laakmann, Philipp Petersen
Summary: Deep neural networks using ReLU activation function can efficiently approximate solutions for various types of parametric linear transport equations, even for high-dimensional and non-smooth cases. Through their inherent compositionality, these networks can resolve the characteristic flow underlying the transport equations and achieve approximation rates independent of the parameter dimension.
ADVANCES IN COMPUTATIONAL MATHEMATICS
(2021)
Article
Mathematics, Applied
Van Tiep Do, Ron Levie, Gitta Kutyniok
Summary: This paper discusses the problem of superposition of different geometric characteristics in natural images and proposes a method for simultaneously decomposing and inpainting the image. The method utilizes an l(1) minimization approach with two dictionaries to sparsify and inpaint the two parts of the image.
ANALYSIS AND APPLICATIONS
(2022)
Article
Mathematics
Remi Gribonval, Gitta Kutyniok, Morten Nielsen, Felix Voigtlaender
Summary: This study focuses on the expressivity of deep neural networks, analyzing the complexity of networks based on the number of connections or neurons. The research investigates a class of functions whose error of best approximation decays at a certain rate with increasing complexity budget. The findings suggest that certain types of skip connections do not affect the resulting approximation spaces, and discuss the impact of network nonlinearity and depth on these spaces.
CONSTRUCTIVE APPROXIMATION
(2022)
Article
Mathematics
Gitta Kutyniok, Philipp Petersen, Mones Raslan, Reinhold Schneider
Summary: By utilizing the low dimensionality of the solution manifold, we have constructed neural networks for approximating the solution maps of parametric partial differential equations with significantly superior approximation rates compared to classical results.
CONSTRUCTIVE APPROXIMATION
(2022)
Article
Engineering, Electrical & Electronic
Ron Levie, Cagkan Yapar, Gitta Kutyniok, Giuseppe Caire
Summary: This paper proposes a deep learning method for accurately estimating the pathloss function in urban environments, showing significant improvement over previous methods.
IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS
(2021)
Article
Mathematics, Applied
Hector Andrade-Loarca, Gitta Kutyniok, Ozan Oektem, Philipp Petersen
Summary: In this paper, we propose a deep-learning-based algorithm that tackles both the reconstruction and wavefront set extraction problems in tomographic imaging. By using the wavefront set information of X-ray data, the algorithm requires the neural networks to extract the correct ground truth wavefront set and image simultaneously, leading to improved reconstruction results.
APPLIED AND COMPUTATIONAL HARMONIC ANALYSIS
(2022)
Article
Mathematics, Applied
Sohir Maskey, Ron Levie, Gitta Kutyniok
Summary: We study the transferability of spectral graph convolutional neural networks (GCNNs) and provide several contributions. Firstly, we prove that any fixed GCNN with continuous filters is transferable based on graphon analysis. Secondly, we establish the transferability for graphs that approximate unbounded graphon shift operators. Lastly, we obtain non-asymptotic approximation results and prove the linear stability of GCNNs. These findings extend the current state-of-the-art results in transferability of GCNNs.
APPLIED AND COMPUTATIONAL HARMONIC ANALYSIS
(2023)
Editorial Material
Psychology, Educational
Enkelejda Kasneci, Kathrin Sessler, Stefan Kuechemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Guennemann, Eyke Huellermeier, Stepha Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia Nerdel, Juergen Pfeffer, Oleksandra Poquet, Michael Sailer, Albrecht Schmidt, Tina Seidel, Matthias Stadler, Jochen Weller, Jochen Kuhn, Gjergji Kasneci
Summary: Large language models are a significant advancement in AI, and despite criticism and bans, they are here to stay. This commentary discusses the potential benefits and challenges of using these models in education, emphasizing the need for competencies, critical thinking, and strategies for fact checking. Teachers and learners must understand the technology and its limitations, and educational systems need clear strategies and pedagogical approaches to maximize the benefits. Challenges such as bias, human oversight, and potential misuse can provide insights and opportunities for early education on societal biases and risks of AI applications.
LEARNING AND INDIVIDUAL DIFFERENCES
(2023)
Proceedings Paper
Computer Science, Artificial Intelligence
Stefan Kolek, Duc Anh Nguyen, Ron Levie, Joan Bruna, Gitta Kutyniok
Summary: This paper presents a framework called Rate-Distortion Explanation (RDE) for explaining black-box model decisions. The framework is based on perturbations of the target input signal and can be applied to any differentiable pre-trained model, such as neural networks. Experiments demonstrate the adaptability of the framework to diverse data modalities, including images, audio, and physical simulations of urban environments.
XXAI - BEYOND EXPLAINABLE AI: International Workshop, Held in Conjunction with ICML 2020, July 18, 2020, Vienna, Austria, Revised and Extended Papers
(2022)
Proceedings Paper
Computer Science, Artificial Intelligence
Mariia Seleznova, Gitta Kutyniok
Summary: In this paper, the NTK of fully-connected ReLU networks with depth comparable to width is studied. It is found that the properties of NTK depend significantly on the depth-to-width ratio and the distribution of parameters at initialization. The importance of the ordered phase in the hyperparameter space for the stability of NTK is also highlighted.
INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 162
(2022)
Proceedings Paper
Computer Science, Artificial Intelligence
Stefan Kolek, Duc Anh Nguyen, Ron Levie, Joan Bruna, Gitta Kutyniok
Summary: CartoonX is a novel explanation method tailored towards image classifiers that extracts relevant piecewise smooth parts of an image in the wavelet domain to reveal valuable explanatory information, particularly for misclassifications, achieving lower distortion with fewer coefficients than state-of-the-art methods.
COMPUTER VISION, ECCV 2022, PT XII
(2022)
Proceedings Paper
Acoustics
Cagkan Yapar, Ron Levie, Gitta Kutyniok, Giuseppe Caire
Summary: This paper presents a solution to the problem of localization in dense urban scenarios. It introduces a deep learning-based method called LocUNet, which relies solely on the received signal strength from base stations for localization, offering high accuracy and robustness. By utilizing estimated pathloss radio maps and a neural network-based radio map estimator, the proposed method is suitable for real-time applications.
2022 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP)
(2022)
