The non-Gaussianity in the primordial density fluctuations is a key feature to clarify the early Universe and it has been probed with the Cosmic Microwave Background (CMB) bispectrum. In recent years, we have treated the novel-type CMB bispectra, which originate from the vector- and tensor-mode perturbations and include the violation of the rotational or parity invariance. On the basis of our current works, this thesis provides the general formalism for the CMB bispectrum sourced by the non-Gaussianity in the scalar, vector and tensor-mode perturbations. Applying this formalism, we calculate the CMB bispectra from the two scalars and a graviton correlation and primordial magnetic fields, and then outline new constraints on these magnitudes. Furthermore, this formalism can be easily extended to the cases where the rotational or parity invariance is broken. We also compute the CMB bispectra from the scalar-mode non-Gaussianities with a preferred direction and the tensor-mode non-Gaussianities induced by the parity-violating Weyl cubic terms. Here, we show that these bispectra include unique signals, which any symmetry-invariant models can never produce.