Combined three-dimensional plastic flow and strain-induced phase transformation (PT) in boron nitride (BN) under high pressure and large shear in a rotational diamond anvil cell (rotational DAC or RDAC) are investigated. Geometrically nonlinear frameworks including finite elastic, transformational, and plastic deformations and finite element method (FEM) are utilized. Quantitative information is obtained on the evolutions of the stress tensor, plastic strain, volume fraction of phases in the entire sample, and slip-cohesion transitions, all during torsion under a fixed compressive load in RDAC. The effects of the applied compressive stress and the sample radius on PT and plastic flow are discussed. In comparison with DAC, the same amount of the high-pressure phase can be obtained at a much lower pressure in RDAC, which reduces the required force and the risk of diamond fracture. Also, RDAC has a potential to complete PT during torsion under pressure close to the minimum possible. A quasi-homogeneous pressure can be obtained in a transforming sample in RDAC under a proper choice of properties and parameters of a gasket. A number of experimental phenomena, including the pressure self-multiplication and quasi-homogeneous pressures in DAC and RDAC, are reproduced and interpreted. The simulation results provide a significant insight into coupled PTs and plastic flow in material in RDAC, and are important for the optimum design of experiments and the extraction of material parameters for PT, as well as for the optimization and control of PTs by the variation of various parameters.