In this thesis, we develop a model that extends scalar sector of Two-Higgs-Doublet Model by adding a color-octet scalar.

Firstly, we construct the scalar potential with the color-octet scalar. The potential consists of three parts. The first part is the Two-Higgs-Doublet Model potential which describe the self interactions of the two Higgs doublets. The second part is the self interactions of the color-octet scalar. The third part arises from the interplay between the color-octet scalar and two Higgs doublets.

To comply with the experimental facts that strength of flavor changing neutral currents are strongly limited, we apply the principle of minimal flavor violation to our model to avoid tree-level flavor changing neutral current induced by Higgs-fermion interactions. With minimal flavor violation, the couplings of the scalars to fermions are greatly constrained.

We also impose another two important theoretical constraints on our model, custodial symmetry and perturbative unitarity. The purpose of imposing custodial symmetry is to comply with experimental results in the electroweak precision measurements. Unitarity is required for perturbation theory to work at high energy scatterings. With the two constraints applied, the degree of freedom of the parameter space is reduced, and the range of parameters can be estimated.

Next, we study the impact of the color-octet scalar on the decays of Higgs to two photons and two gluons. We make use of the existing fit to branching ratios of Higgs to two photons and Higgs to two gluons and show the points within the 1-$\sigma$ and 2-$\sigma$ contours of the fit. We also scan the parameter space and present the points that satisfy unitarity and stability conditions.

At last, we derive the one-loop renormalization group equations for the scalar self-couplings in our model. We mainly consider the contributions of themselves to study the evolution of the scalar self-couplings as energy scale is increased to a very high level. We solve the equations numerically, and evolve them from the electroweak scale up to the scale of Grand Unification Theory. In the meantime, we impose the requirements of unitarity and vacuum stability. We show that viable ranges of parameter space survive and shrink as the scale is increased.