A new mathematical approach for deriving learning algorithms for various neural network models including the Hopfield model, Bidirectional Associative Memory, Dynamic Heteroassociative Neural Memory, and Radial Basis Function Networks is presented. The mathematical approach is based on the relaxation method for solving systems of linear inequalities. The newly developed learning algorithms are fast and they guarantee convergence to a solution in a finite number of steps. The new algorithms are highly insensitive to choice of parameters and the initial set of weights. They also exhibit high scalability on binary random patterns. Rigorous mathematical foundations for the new algorithms and their simulation studies are included.