In axial compressors with a relatively large blade tip clearance, an unsteady phenomenon denoted as rotating instability (RI) can be detected when the compressor is throttled to the operating points near the stability limit. In the frequency domain, RIs are shown as a hump lower than the blade passing frequency. This indicates an increase in noise level and might cause blade vibration and other undesirable structural issues. In this thesis, a comprehensive study on RIs is performed based on an axial compressor rotor row of the Low Speed Research Compressor at Technische Universität Dresden. Three blade tip clearances are investigated, and a groove casing treatment is mounted over the shroud for flow control. Methods of numerical modeling are evaluated, and zonal large eddy simulation is selected as the numerical model. By analyzing the flow properties and applying the dynamic mode decomposition, the coherent flow structure corresponding to the dominant frequency of RIs is extracted and visualized as the waves located in the blade tip region. The criteria for the appearance of RIs in the investigated research object are concluded.