Though photoluminescence (PL) of Si quantum dots (QDs) has been known for decades and both theoretical and experimental studies have been extensive, their luminescence mechanism has not been elaborated. Several models have been proposed to explain the mechanism. A deep insight into the origin of light emissions in Si QDs is necessary. This work calculated the ground- and excited state properties of hydrogenated Si QDs with various diameters, including full hydrogen passivation, single SiO ligands, single epoxide and coexisting SiO and epoxide structures in order to investigate the dominant contribution states for luminescence. The results revealed that even a single oxygen atom in hydrogenated Si QDs can dramatically change their electronic and optical properties. Intriguingly, we found that a size-independent emission, the strongest among all possible emissions, was induced by the single SiO passivated Si-QDs. In non-oxidized Si-QDs exhibiting a core-related size-tunable emission, the luminescence properties can be modulated by the ligands of Si QDs, and a very small number of oxygen ligands can strongly influence the luminescence of nanocrystalline silicon. Our findings deepen the understanding of the PL mechanism of Si QDs and can further promote the development of silicon-based optoelectronic devices.