The present study aimed to examine the influence of response format on the reliability and validity of the tests in question. To this end, the grammaticality judgment (GJ) test developed by Gass (1994) was selected as the instrument of the study which was manipulated in terms of the response format while the original items remained intact. Thus, a multiple-choice GJ test (MCGJT) was constructed to be compared with the traditional form, i.e., the dichotomous type (DGJT). In an interval of a month, the two tests were administered to a group of 110 engineering-major BS students. The results revealed that MCGJT was slightly more reliable (0.84 Cronbach's alpha) than the DGJT (0.83 Cronbach's alpha). With regard to validity, DGJT yielded slightly better validity indices in the first phase. Nevertheless, as a second phase to internal correlations, the correlations between the subsets and their totals were also explored; this yielded almost similar results.