In recent years, extensive studies of many interesting aspects of online community dynamics promoted a better understanding of this area. One of the most challenging problems facing builders of online communities is the design of incentive mechanisms that can ensure user participation. However, running online community experiments in the real world is expensive, and requires a great deal of motivation from users. In this thesis two major approaches are explored: system dynamics modeling and agent-based modeling, to simulate the overall behaviours of participants in online communities. Although these models are developed by using two different methodologies, both of them can provide insights into the user motivation process, incentive mechanism evaluation and community development. The target online community for my study is called Comtella, which is used in several senior Computer Science classes in the Department of Computer Science, University of Saskatchewan. Simulation models for the Comtella online community have been developed and the simulation results are useful to provide future directions for incentive mechanism improvement.