Modeling of evolution is becoming increasingly important in biological and social systems. The idea of evolution presents a reasonable and convincible perspective to the problem of long term competition between different species or populations in nature. Evolutionary game theory has seen tremendous achievements and it exhibits good performance in modeling competition of species or populations. However, there remain many challenges, and the underlying scientific mechanism of competition is full of uncertainties. Bringing tools of math-biology and statistics to modeling the competitive phenomenons provides a way of throwing off the veil of nature to reveal itself to the world.

In this thesis, we study the spatial effects on evolution in game models numerically and theoretically, optimality and stability of symmetric evolutionary game and its applications to genetic selection and social network, and parameter estimation in game theoretic modeling of the biological and social systems. More precisely, we categorize the types of nonlinear games and investigate the simulation study of spatial effects on the evolution of cooperation in a nonlinear yeast game. We also incorporate the spatial diffusion effects in the replicator dynamics of nonlinear game models, and prove the asymptotic behavior of the solution to the corresponding adapted replicator diffusion equation. Furthermore, we apply the statistical techniques and methodologies to solve the inverse problem in evolutionary game dynamics, that is, the NB mixture model and Markov random field model incorporating replicator equations are built where penalized approximated negative log-likelihood method with generalized smoothing approach and Besag pseudo-likelihood method are implemented to facilitate the estimation and inference of model parameters. Finally, the theory for obtaining optimal and stable strategies for symmetric evolutionary games is explored, and new proofs and computational methods are provided. And the symmetric evolutionary game is applied to model the evolution of a population over a social network, then several different types of equilibrium states corresponding to social cliques are analyzed and a set of conditions for their stabilities are proved.