Abstract
Machine learning is a complicated function of many elements.Our ultimate goal is to realize an intelligent agent thatcan learn to do the right things from reinforcement throughinteractions with the environment. As a dissertation theme,we have narrowed down a vast scope of ``agent learning'' toa small yet indispensable ``brain modeling'' subsidiary,called ``artificial'' neural-network (NN) learning.It is quite artificial because our approaches (describedin this dissertation) are tied to efficient implementationson modern computers designed from engineering and computerscience perspectives although its fundamental concept isbiologically inspired.Our primary interest in this dissertation resides inthe development of algorithmic and computational resultsapplicable to NN-learning, especially to supervisedlearning, where our NN model is optimized to learnthe designated outputs in response to certain input stimuli.A straightforward formulation often gives rise towhat we call ``neural networks nonlinear least squaresproblems.'' We attack the posed problems in conjunction withmodern numerical linear algebra techniques specially gearedto conspicuous characteristics arising in NN-learning;specifically, we identify and exploit data sparsity,symmetric stagewise architecture, and parameter separability.These key features are often neglected in the NN literature.We begin to explain how sparsity issues stand up inmultiple-response problems, which commonly entail twotypical sparse matrix formats: a block-arrow Hessian matrix,and a block-angular Jacobian matrix of the residual vector.Exploiting the data sparsity leads to very efficient learningalgorithms suitable for a wide variety of machine learningand optimization problems: In small- or medium-scaleoptimization problems, the sparsity-exploitationmakes an efficient matrix factorization on the Hessianmatrix, while in large-scale problems it fulfills a sparsematrix-vector multiply for extracting the Hessian informationin the Krylove subspace. The latter method comes into playas a new learning mode as ``iterative batch learning''implementable in either full-batch ormini-batch (i.e., block) mode.We next direct our special attention to a symmetric``stagewise'' structure embedded in a so-called multi-layerperceptron (MLP), a popular feed-forward NN model with multiplelayers (or stages); geometrically, an MLP forms a (symmetric)cone in the parameter space. The theory of discrete-stageoptimal control dictates advanced learning strategies such asthe introduction of ``stage costs'' in addition to theterminal cost, leading to what we call ``hidden-node teaching.''A remarkable result obtained by this new learning schemeis that it can develop insensitivity to initial parametersin a classical two-class classification parity benchmark problem.More significantly, the theory serves to exploit thenice multi-stage symmetric structure for evaluating the Hessianmatrix just as the well-known (first-order) backpropagationcomputes the gradient vector in a stagewise fashion.Our newly-developed ``stagewise'' second-order backpropagationalgorithm, derived from the second-order optimal control theory,can evaluate the full Hessian matrix faster than ``standard''methods that obtain only the Gauss-Newton Hessian matrix(e.g., see Matlab NN-toolbox for such a procedure); this isa truly tremendous breakthrough in the nonlinear least squaressense. In reality, the full Hessian matrix may not bepositive (semi-)definite during the learning phase, butthe widely-employed trust-region nonlinearoptimization method can deal excellently with the indefiniteHessian since the underlying theory has thrived on the``negative curvatures'' over the last two decades.The trust-region approach based onthe full Hessian matrix is of immense value in solvingreal-world ``large-residual'' nonlinear least squares problemsbecause the matrix of second derivatives is important toefficiency. In consequence, our stagewise second-orderbackpropagation approach would prove practically usefulfor general nonlinear optimization in a broadersense as long as a posed problem possesses a stagewiseconstitution.Furthermore, a model of mixed linear and nonlinear parametersmay become of great concern in various contexts of machine learning.In numerical linear algebra, the variable projection (VP) algorithmhas been the standard approach to the ``separable'' nonlinear(i.e., mixed linear & nonlinear) least squares problems sinceearly 1970s. For the sake of second-order algorithms, we desireto use as much Hessian information as possible while manipulatingcertain structural properties associated with a given NN model.Looking in this spirit toward further exploitation of parameterseparability, we have endeavored to devise an extension of VPalgorithms that employ the full Hessian matrix.The consequent method aims at solving large-residual machinelearning problems when both linear and nonlinear parametersco-exist in a given learning model. Although this approachstill needs further investigation, it would probably helpin optimizing other machine learning models such asgeneralized linear discriminant functions.Special structure should always be exploited when it arises.The multi-stage NN-learning is an excellent challenge, forit exhibits a great deal of structure; the principal ingredientsare analyzed out to be sparse, symmetric, stagewise, and separable.Along the guidance on structure exploitation,we emphasize the rigorous mathematical theory of optimal controlas well as the practical use of modern numerical linear algebraand nonlinear numerical optimization for algorithmic design purposes.Our proposed learning methods could apply broadlyto learning machines in yet unexplored domains and therefore haveenormous potential for diverse future extensions.