In this research the nonlinear modal interaction and the effect of the interaction on the stressed power system dynamic behavior including excitation control performance are discussed. A systematic scheme based on the normal forms method for the determination of nonlinear interaction between fundamental modes and excitation control modes in a stressed power system is developed;In a stressed power system, the interarea mode phenomenon may occur under large disturbance. Recent investigations revealed that the interarea mode may be among the power system fundamental modes of oscillation associated with the nonlinear modal interaction. If there is significant interaction, the controls will be affected. Because the conventional control system design techniques do not consider the interaction between modes, it is essential to develop a new approach for a clear understanding of the nonlinear modal interaction and its effect on the system dynamic performance;The proposed approach consists of Taylor series expansion, eigen-analysis, normal forms method, and time simulation. In normal form theory, a set of N-dimensional N system modes is said to be resonant of order r (where r is an integer) if [lambda][subscript]j=[sigma][limits][subscript]spk=1N m[subscript]k[lambda][subscript]k and r=[sigma][limits][subscript]spk=1N m[subscript]k for j = 1, 2, ·s, N. In this research work the second-order approxima-tion of the system equations is used. Second-order resonance condition is characterized by [lambda][subscript]k + [lambda][subscript]l = [lambda][subscript]j. If there are no second-order resonances then all the second-order nonlinear terms can be eliminated successively from the vector field using a set of nonlinear state space transformations. The terms of the nonlinear transformation provide important information regarding nonlinear modal interaction;After identifying the modes associated in the interaction and the extent to which they interact, initial conditions for the state variables corresponding to the excitation of the interacting modes are determined using the normal form transformation. These initial conditions are then used to analyze the effect of nonlinear modal interaction on the dynamic system behavior including the excitation control performance;The approach has been applied to two systems which are the four-generator test system and the IEEE 50-generator test system. The results show that excitation control modes interact with low frequency modes and the nonlinear modal interaction can substantially influence the dynamic system behavior.