The Effect of Generative AI Tools (Grammarly and ProWritingAid) on Students' Writing Skill: Pengaruh Alat Kecerdasan Buatan Generatif (Grammarly dan ProWritingAid) terhadap Kemahiran Menulis Siswa

Rosi Anjarwati; Daning Hentasmaka

doi:10.58194/eloquence.v4i3.3064

Authors

Rosi Anjarwati Universitas PGRI Jombang
Daning Hentasmaka Universitas PGRI Jombang

DOI:

https://doi.org/10.58194/eloquence.v4i3.3064

Keywords:

Generative AI, Grammarly, Prowritingaid, Writing Skill, Quasi Experimental Design

Abstract

Background: The integration of Generative Artificial Intelligence (GAI) into English language education has transformed writing instruction, yet empirical comparisons of its tools remain limited.

Purpose: This study investigates the comparative effect of Grammarly and ProWritingAid on EFL students’ academic writing skills.

Method: A quasi-experimental design was employed with three intact groups (n=37): Grammarly (n=13), ProWritingAid (n=11), and electronic dictionary control (n=13). Writing pre-test and post-test data were analyzed using the non-parametric Kruskal-Wallis test due to violations of normality and homogeneity assumptions.

Results and Discussion: A significant difference was found among groups (χ²=10.544, p=0.005). ProWritingAid yielded the highest mean rank (24.73), followed by Grammarly (21.81), and electronic dictionary (11.35). Post-hoc Mann-Whitney tests confirmed significant gains for both AI tools over the control (p<0.05), but no significant difference between Grammarly and ProWritingAid (p=0.792).

Conclusions and Implications: Both AI tools effectively enhance writing skills, with ProWritingAid showing slight superiority in stylistic and structural feedback.

References

[1] X. Fang, D. T. K. Ng, C. K. L. Leung, and S. K. W. Chu, “A Systematic Review of Artificial Intelligence Technologies Used for Story Writing,” Educ Inf Technol, pp. 1–37, 2023.

[2] A. Han and Z. Cai, “Design Implications of Generative AI Systems for Visual Storytelling for Young Learners,” in Proceedings of the 22nd Annual ACM Interaction Design and Children Conference, Chicago IL USA: ACM, June 2023, pp. 470–474. doi: https://doi.org/10.1145/3585088.3593867

[3] N. Nazari, M. S. Shabbir, and R. Setiawan, “Application of Artificial Intelligence Powered Digital Writing Assistant in Higher Education: Randomized Controlled Trial,” Heliyon, vol. 7, no. 5, p. e07014, May 2021, doi: https://doi.org/10.1016/j.heliyon.2021.e07014

[4] H. H. H. Ali, “The Importance of the Four English Language Skills: Reading, Writing, Speaking, and Listening in Teaching Iraqi Learners,” Humanit. Nat. Sci. J., vol. 3, no. 2, Feb. 2022, doi: https://doi.org/10.53796/hnsj3210

[5] J. M. Gayed, M. K. J. Carlon, A. M. Oriola, and J. S. Cross, “Exploring an AI-based Writing Assistant’s Impact on English Language Learners,” Comput. Educ. Artif. Intell., vol. 3, p. 100055, 2022, doi: https://doi.org/10.1016/j.caeai.2022.100055

[6] N. Amani and M. Bisriyah, “University Students’ Perceptions of AI-Assisted Writing Tools in Supporting Self-Regulated Writing Practices,” Indones. J. Engl. Lang. Teach. Appl. Linguist., vol. 10, no. 1, pp. 91–107, 2025. https://doi.org/10.21093/ijeltal.v10i1.1942

[7] J. Huang and A. Mizumoto, “Examining the Effect of Generative AI on students’ Motivation and Writing Self-Efficacy,” Digit. Appl. Linguist., vol. 1, p. 102324, Nov. 2024, doi: https://doi.org/10.29140/dal.v1.102324

[8] I. Perdana, J. Bungai, and I. Ilham, “The Students’ Perceptions of AI-Generated and Teacher Feedback in English Writing Skills: A Comparative Study,” J. Pendidik. Bhs. Ingg. Undiksha, vol. 13, no. 1, pp. 22–31, May 2025, doi: https://doi.org/10.23887/jpbi.v13i1.92668

[9] D. T. Y. G. Sumakul, F. A. Hamied, and D. Sukyadi, “Students’ Perceptions of the Use of AI in a Writing Class,” presented at the 67th TEFLIN International Virtual Conference & the 9th ICOELT 2021 (TEFLIN ICOELT 2021), Padang, Indonesia, 2022. doi: https://doi.org/10.2991/assehr.k.220201.009

[10] N. Pellas, “The Effects of Generative AI Platforms on Undergraduates’ Narrative Intelligence and Writing Self-Efficacy,” Educ. Sci., vol. 13, no. 11, p. 1155, Nov. 2023, doi: https://doi.org/10.3390/educsci13111155

[11] M. Ahmadzade and M. Farahian, “At a Crossroad: Comparing ChatGPT and Grammarly in Enhancing EFL Learners’ Use of the Article ‘The,’” J. Stud. Lang. Learn. Teach., vol. 2, no. 1, pp. 47–62, 2025, doi: https://doi.org/10.2139/ssrn.5699162

[12] E. Sulistyowati, “Penerapan Grammarly Tool Untuk Meningkatkan Keterampilan Menulis Teks Analisis Ekspositoris Siswa,” J. Educ. FKIP UNMA, vol. 7, no. 2, pp. 559–566, June 2021, doi: https://doi.org/10.31949/educatio.v7i2.1144

[13] S. Huang and P. Grady, “Generative AI: A Creative New World.” Sequoia Capital US/Europe., 2022.

[14] R. Marrone, A. Zamecnik, S. Joksimovic, J. Johnson, and M. De Laat, “Understanding Student Perceptions of Artificial Intelligence as a Teammate,” Technol. Knowl. Learn., vol. 30, no. 3, pp. 1847–1869, Sept. 2025, doi: https://doi.org/10.1007/s10758-024-09780-z

[15] A. Azennoud, “Enhancing Writing Accuracy and Complexity through AI-Assisted Tools among Moroccan EFL University Learners,” Int. J. Linguist. Transl. Stud., vol. 5, no. 4, pp. 211–226, 2024, doi: https://doi.org/10.36892/ijlts.v5i4.529

[16] C. K. Y. Chan, “A Comprehensive AI Policy Education Framework for University Teaching and Learning,” Int. J. Educ. Technol. High. Educ., vol. 20, no. 1, p. 38, July 2023, doi: https://doi.org/10.1186/s41239-023-00408-3

[17] Y. A. Muhie and A. B. Woldie, “Teaching-Learning, Higher Education, AI, AI-Based Education, Implementation Framework,” Sci. Technol., vol. 10, no. 1, pp. 1–7, 2020.

[18] T. N. Fitria, “ProWritingAid as AI-Powered Writing Tools: The Performance in Checking Grammar and Spelling of Students’ Writing,” vol. 12, no. 2, 2023.

[19] R. Schmidt-Fajlik, “ChatGPT as a Grammar Checker for Japanese English Language Learners: A Comparison with Grammarly and ProWritingAid,” AsiaCALL Online J., vol. 14, no. 1, pp. 105–119, June 2023, doi: https://doi.org/10.54855/acoj.231417

[20] G. S. Khalifa, A. M. Elshaer, S. A. El-Aidie, A. S. E. Yee, A. Farhan, and E. Gherbi A,H, “ChatGPT’s Challenges and Opportunities in Higher Education: A Technology-user (s)-environment lens.,” Int. J. Manag. Educ., vol. 18, no. 5, pp. 453–480, 2024, doi: https://doi.org/10.1504/IJMIE.2024.140898

[21] X. Zhai, X. Chu, C. S. Chai, M. S. Y. Jong, and A. Istenic, “A Review of Artificial Intelligence (AI) in Education from 2010 to 2020.,” Complexity, vol. 2021, no. 1, 2021, doi: https://doi.org/10.1155/2021/8812542

[22] G. J. Hwang, H. Xie, B. W. Wah, and D. Gašević, “Vision, Challenges, Roles and Research Issues of Artificial Intelligence in Education.,” Comput. Educ. Artif. Intell., vol. 1, p. 100001, 2020, doi: https://doi.org/10.1016/j.caeai.2020.100001

[23] R. Luckin and W. Holmes, Intelligence Unleashed: An Argument for AI in Education. London, UK: UCL Knowledge Lab, 2016.

[24] M. Stevenson and A. Phakiti, “Automated Feedback and Second Language writing,” in Feedback in Second Language Writing: Contexts and Issues, Second edition., United Kingdom: Cambridge University Press, 2019, pp. 125–142. https://doi.org/10.1017/9781108635547.009

[25] M. Li, “Automated Writing Evaluation,” in Researching and Teaching Second Language Writing in the Digital Age, Cham: Springer International Publishing, 2022, pp. 151–181. https://doi.org/10.1007/978-3-030-87710-1_7

[26] S. Cao and T. Zhou, “Exploring the Efficacy of ChatGPT-Based Feedback Compared With Teacher Feedback and Self-Feedback: Evidence From Chinese-English Translation,” SAGE Open, vol. 15, no. 3, p. 21582440251369204, Jan. 2025, doi: https://doi.org/10.1177/21582440251369204

[27] P. Wei, X. Wang, and H. Dong, “The impact of automated writing evaluation on second language writing skills of Chinese EFL learners: a randomized controlled trial,” Front. Psychol., vol. 14, p. 1249991, Sept. 2023, doi: https://doi.org/10.3389/fpsyg.2023.1249991

[28] N. Zhai and X. Ma, “Automated Writing Evaluation (AWE) Feedback: A Systematic Investigation of College Students’ Acceptance.,” Comput. Assist. Lang. Learn., vol. 35, no. 9, pp. 2817–2842, 2022, doi: https://doi.org/10.1080/09588221.2021.1897019

[29] W. Holmes, M. Bialik, and C. Fadel, Artificial Intelligence in Education Promises and Implications for teaching and learning. USA: Center for Curriculum Redesign, 2019.

[30] G. L. Parra and X. S. Calero, “Automated Writing Evaluation Tools in the Improvement of the Writing Skill,” Int. J. Instr., vol. 12, no. 2, pp. 209–226, Apr. 2019, doi: https://doi.org/10.29333/iji.2019.12214a

[31] G. Dizon and J. M. Gayed, “A Systematic Review of Grammarly in L2 English Writing Contexts,” Cogent Educ., vol. 11, no. 1, p. 2397882., 2024, doi: https://doi.org/10.1080/2331186X.2024.2397882

[32] K. D. Damayanti and M. H. Santosa, “The Effect of ProWritingAid Writing Tool on the Writing Skills of English as a Foreign Language (EFL) Students: A Systematic Literature Review,” Indones. J. Multidiciplinary Res., vol. 4, no. 2, pp. 469–478, Oct. 2024, doi: https://doi.org/10.17509/ijomr.v4i2.78943

[33] D. Miranty, U. Widiati, B. Y. Cahyono, and T. I. S. T. Sharif, “The Effectiveness of Using Grammarly in Teaching Writing Among Indonesian Undergraduate EFL Students,” presented at the International Seminar on Language, Education, and Culture (ISoLEC 2021), Malang, Indonesia, 2021. doi: https://doi.org/10.2991/assehr.k.211212.008

[34] M. Rahma and E. L. Zen, “Errors in Students’ Writing (Reports from ProWritingAid),” Scope J. Engl. Lang. Teach., vol. 7, no. 2, p. 202, Apr. 2023, doi: https://doi.org/10.30998/scope.v7i2.15528

[35] P. Chauhan, “Fundamentals of Academic Writing: A Literature Review,” J. NELTA, vol. 27, no. 1, 2022. https://doi.org/10.3126/nelta.v27i1-2.53201

[36] H. Aljuaid, “The Impact of Artificial Intelligence Tools on Academic Writing Instruction in Higher Education: A Systematic Review,” Arab World Engl. J., vol. 1, no. 1, pp. 26–55, Apr. 2024, doi: https://doi.org/10.24093/awej/ChatGPT.2

[37] P. D. Deep and Y. Chen, “The Role of AI in Academic Writing: Impacts on Writing Skills, Critical Thinking, and Integrity in Higher Education,” Societies, vol. 15, no. 9, p. 247, Sept. 2025, doi: https://doi.org/10.3390/soc15090247

[38] K. Hyland, Second language writing. New York: Cambridge University Press, 2019. https://doi.org/10.1017/9781108635547.009

[39] S. Millah, “Peningkatan Kemampuan Menulis Deskripsi Bahasa Inggris Melalui Strategi Pembelajaran Partisipatif,” Bahtera J. Pendidik. Bhs. Dan Sastra, vol. 24, no. 1, pp. 77–85, Jan. 2025, doi: https://doi.org/10.21009/bahtera.241.08

[40] Astari, “Barriers to Writing Skill Development in English Learners: A Systematic Review of Quantitative Studies (2020–2024),” Asian J. Lang. Lit. Cult. Stud., vol. 8, no. 2, pp. 432–439, 2025.

[41] Y. Suharyat and L. Lusiana, “The Investigation of Students’ Writing Skills in Learning English as a Second Language in Indonesia,” Eur. J. Engl. Lang. Stud., vol. 3, no. 1, pp. 47–57, June 2023, doi: https://doi.org/10.12973/ejels.3.1.47

[42] M. Obisuru and W. Purbani, “Kemampuan Menulis Bahasa Inggris Siswa Melalui self-editing and self-correcting berdasarkan Analisis Kesalahan Gramatikal dan Kosakata,” LingTera, vol. 3, no. 1, p. 51, May 2016, doi: https://doi.org/10.21831/lt.v3i1.8473

[43] A. Susanti and A. T. Azizan, “Enhancing Writing Skills Through Chat GPT: A Literature Review,” vol. 2, no. 2, 2025.

[44] R. Rahmaniah, “The Effect of Peer Feedback on English Writing Skills Among EFL Students,” vol. 13, no. 2, 2025.

[45] J. W. Creswell, Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Sage Publication, 2022.

[46] A. Asari, Zulkarnaini, A. C. Hartatik, Anam, and Suparto, Pengantar Statistika. Solok, Sumatra Barat: PT Mafy Media Literasi Indonesia, 2023.

[47] J. Pallant, SPSS Survival Manual: A Step by Step Guide to Data Analysis using IBM SPSS, 7th ed. London: McGraw-Hill Education., 2020. https://doi.org/10.4324/9781003117445

[48] M. I. Wahyuda, “The Effectiveness of Grammarly and ProWritingAid Application toward Writing Skill across Students writing level of MAN Batu students,” Universitas Islam Negeri Maulana Malik Ibrahim, Malang, 2022.