A new problem formulation and numerical algorithm for an advanced phase-field approach (PFA) to martensitic phase transformation (PT) are presented. Finite elastic and transformational strains are considered using a fully geometrically-nonlinear formulation, which includes different anisotropic elastic properties of phases. The requirements for the thermodynamic potentials and transformation deformation gradient tensor are advanced to reproduce crystal lattice instability conditions under a general stress tensor obtained by molecular dynamics (MD) simulations. The PFA parameters are calibrated, in particular, based on the results of MD simulations for PTs between semiconducting Si I and metallic Si II phases under complex action of all six components of the stress tensor (Levitas et al. in Phys Rev Lett 118:025701, 2017a; Phys Rev B 96:054118, 2017b). The independence of the PFA instability conditions of the prescribed stress measure is demonstrated numerically for the initiation of the PT. However, it is observed that the PT cannot be completed unless the stress exceeds the stress peak points that depend on which stress measure is prescribed. Various 3D problems on lattice instability and following nanostructure evolution in single-crystal Si are solved. The effect of stress hysteresis on the nanostructure evolution is studied through analysis of the local driving force and stress fields. It is demonstrated that variation of internal stress fields due to differing boundary conditions may lead to completely different PT mechanisms.