The problem of efficient design of material microstructures exhibiting desired properties spans a variety of engineering and science applications. An ability to rapidly generate microstructures that exhibit user-specified property distributions transforms the iterative process of traditional microstructure-sensitive design. We reformulate the microstructure design process as a constrained Generative Adversarial Network (GAN). This approach explicitly encodes invariance constraints within a GAN to generate two-phase morphologies for photovoltaic applications obeying design specifications: specifically, various short circuit current density and fill-factor combinations. Such invariance constraints can be represented by deep learning-based surrogates of full physics models mapping microstructure to photovoltaic properties. To circumvent data generation bottlenecks, we utilize a multi-fidelity surrogate that reduces the requirements of expensive labels by 5X. Our approach enables fast generation of microstructures (in~190ms) with user-defined properties. Such physics-aware data-driven methods for inverse design problems are expected to democratize and accelerate the field of microstructure-sensitive design.