The Effect of Chatgpt 3.0 as a Personalized Learning Tool in Answering Atomic Structure Assessments
DOI:
https://doi.org/10.54536/ajet.v4i3.4700Keywords:
Atomic Structure, ChatGPT 3.0, General Chemistry Assessment, Generative AI, TAM SurveyAbstract
This study investigates the effect of ChatGPT 3.0, on Grade 11 STEM students’ performance in Atomic Structure assessments. Using a one-group pretest-post-test quasi-experimental design, the study applied the Wilcoxon Signed Rank test, Cohen’s D, and Hake’s Gain index to measure changes in cognitive assessment scores and technology acceptance via a TAM survey. Results revealed a significant increase in test scores with a large effect size (Cohen’s D = 0.9), though gains in specific subtopics were modest. The findings suggest that ChatGPT 3.0 can enhance learning outcomes and promote AI acceptance in education, warranting further investigation with extended intervention periods and larger sample sizes.
Downloads
References
Agbong-Coates, I. J. G. (2024). ChatGPT Integration Significantly Boosts Personalized Learning Outcomes: A Philippine study. International Journal of Educational Management and Development Studies, 5(2), 165–186. https://doi.org/10.53378/353067
Alayacyac, J. R. S., Regidor, J. C., Caballo, J. H. S., Abellanosa, J. E., & Monaghan, G. J. G. (2024). Computer Self-Efficacy and Effectiveness of Quipper Learning Management System. American Journal of Smart Technology and Solutions, 3(1), 17–21. https://doi.org/10.54536/ajsts.v3i1.2428
Ali, Z., & Bhaskar, S. (2016). Basic Statistical Tools in Research and Data Analysis. Indian Journal of Anaesthesia, 60(9), 662. https://doi.org/10.4103/0019-5049.190623
Andrade, C. (2020). The Inconvenient Truth about Convenience and Purposive Samples. Indian Journal of Psychological Medicine, 43(1), 86–88. https://doi.org/10.1177/0253717620977000
Ardyansyah, A., Yuwono, A. B., Rahayu, S., Alsulami, N. M., & Sulistina, O. (2024). Students’ perspectives on the application of a generative pre-trained transformer (GPT) in chemistry learning: a case study in Indonesia. Journal of Chemical Education, 101(9), 3666-3675. https://doi.org/10.1021/acs.jchemed.4c00220
Arguson, A., Mabborang, M., & Paculanan, R. (2023). The acceptability of generative ai tools of selected senior high school teachers in schools divisions office-manila, philippines. Journal of Artificial Intelligence, Machine Learning and Neural Network, 3(6), 1-10. https://doi.org/10.55529/jaimlnn.36.1.10
Bai̇doo-Anu, D., & Ansah, L. O. (2023). Education in the Era of Generative Artificial Intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Journal of AI, 7(1), 52–62. https://doi.org/10.61969/jai.1337500
Bandi, A., Adapa, P. V. S. R., & Kuchi, Y. E. V. P. K. (2023). The power of generative ai: A review of requirements, models, input–output formats, evaluation metrics, and challenges. Future Internet, 15(8), 260. https://doi.org/10.3390/fi15080260
Baroni, I., Calegari, G. R., Scandolari, D., & Celino, I. (2022). AI-TAM: a model to investigate user acceptance and collaborative intention in human-in-the-loop AI applications. Human Computation, 9(1), 1-21. https://doi.org/10.15346/hc.v9i1.134
Baum, Z. J., Yu, X., Ayala, P. Y., Zhao, Y., Watkins, S. P., & Zhou, Q. (2021). Artificial intelligence in chemistry: current trends and future directions. Journal of Chemical Information and Modeling, 61(7), 3197-3212.https://doi.org/10.1021/acs.jcim.1c00619
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2021). Introduction to meta-analysis. John wiley & sons. https://doi.org/10.1002/9780470743386
Bourban, M., & Rochel, J. (2020). Synergies in Innovation: Lessons Learnt from Innovation Ethics for Responsible Innovation. Philosophy & Technology, 34(2), 373–394. https://doi.org/10.1007/s13347-020-00392-w
Bozkurt, A., Junhong, X., Lambert, S., Pazurek, A., Crompton, H., Koseoglu, S., ... & Romero-Hall, E. (2023). Speculative futures on ChatGPT and generative artificial intelligence (AI): A collective reflection from the educational landscape. Asian Journal of Distance Education, 18(1), 53-130. https://doi.org/10.5281/zenodo.7636568
Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature reviews neuroscience, 14(5), 365-376.5
Campbell, S., Greenwood, M., Prior, S., Shearer, T., Walkem, K., Young, S., ... & Walker, K. (2020). Purposive sampling: complex or simple? Research case examples. Journal of research in Nursing, 25(8), 652-661. https://doi.org/10.1177/1744987120927206
Clark, J. M., & Paivio, A. (1987). A Dual Coding Perspective on Encoding Processes. In Springer eBooks. 5–33. https://doi.org/10.1007/978-1-4612-4676-3_1
Clark, J. M., & Paivio, A. (1991). Dual Coding Theory and Education. Educational Psychology Review, 3(3), 149–210. https://doi.org/10.1007/bf01320076
Clark, M. (2021). What is Dual Coding Theory and How Can it Help Teaching?. CENTURY. https://www.century.tech/news/what-is-dual-coding-theory-and-how-can-it-help-teaching/
Clark, T. M. (2023). Investigating the Use of an Artificial Intelligence Chatbot with General Chemistry Exam Questions. Journal of Chemical Education, 100(5), 1905–1916. https://doi.org/10.1021/acs.jchemed.3c00027
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
Danili, E., & Reid, N. (2006). Cognitive Factors that can Potentially Affect Pupils Test Performance. Chemistry Education Research and Practice, 7(2), 64–83. https://doi.org/10.1039/b5rp90016f
Das, B., Majumder, M., Phadikar, S., & Sekh, A. A. (2021). Automatic question generation and answer assessment: a survey. Research and Practice in Technology Enhanced Learning, 16(1), 5. https://doi.org/10.1186/s41039-021-00151-1
David, H. A., & Gunnink, J. L. (1997). The Paired T-test Under Artificial Pairing. The American Statistician, 51(1), 9–12. https://doi.org/10.1080/00031305.1997.10473578
Department of Education, Philippines. (n.d.). Academic track STEM. Retrieved January 10, 2025.
DepEd Computerization Program | Department of Education. (n.d.). Retrieved January 10, 2025, from https://www.deped.gov.ph/2018/04/06/deped-computerization-program/
Doe, J. K. (n.d.). Toward a Firm Technology Adoption Model (F-TAM) in A Developing Country Context. AIS Electronic Library (AISeL). Retrieved October 8, 2024, from https://aisel.aisnet.org/mcis2017/23/
Du Plessis, W. P. (2020). Automated Generation of Test Questions and Solutions. 2020 IFEES WorldEngineering Education Forum - Global Engineering Deans Council (WEEF-GEDC) (pp. 1–5). https://doi.org/10.1109/weef-gedc49885.2020.9293635
E. Morales, M. P. (2006). Development and Validation of a Concept Test in Introductory Physics for Biology Students. De La Salle University. https://www.dlsu.edu.ph/wp-content/uploads/pdf/research/journals/mjs/MJS07-2-2012/MJS07-2-4-morales.pdf
Editor & Editor. (2024). The Role of Generative AI in Education: Use Cases, Benefits and Challenges in 2024. Fullestop Blogs. https://www.fullestop.com/blog/generative-ai-in-education-use-cases-benefits-and-challengs
Eke, D. O. (2023). ChatGPT and the Rise of Generative AI: Threat to Academic Integrity? Journal of Responsible Technology, 13, 100060. https://doi.org/10.1016/j.jrt.2023.100060
Estrellado, C. J., & Miranda, J. C. (2023). Artificial Intelligence in the Philippine Educational Context: Circumspection and Future Inquiries. Social Science Research Network. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4442136
Farrelly, T., & Baker, N. (2023). Generative Artificial Intelligence: Implications and Considerations for Higher Education Practice. Education Sciences, 13(11), 1109. https://doi.org/10.3390/educsci13111109
García-Martínez, I., Fernández-Batanero, J. M., Fernández-Cerero, J., & León, S. P. (2023). Analysing the impact of artificial intelligence and computational sciences on student performance: Systematic review and meta-analysis. Journal of New Approaches in Educational Research, 12(1), 171-197. https://doi.org/10.7821/naer.2023.1.1240
Gasteiger, J. (2020). Chemistry in Times of Artificial Intelligence. ChemPhysChem, 21(20), 2233–2242. https://doi.org/10.1002/cphc.202000518
Giray, L., De Silos, P. Y., Adornado, A., Buelo, R. J. V., Galas, E., Reyes-Chua, E., ... & Ulanday, M. L. (2024). Use and Impact of Artificial Intelligence in Philippine Higher Education: Reflections from Instructors and Administrators. Internet Reference Services Quarterly, 28(3), 315–338. https://doi.org/10.1080/10875301.2024.2352746
Gray, N. (1988). Artificial Intelligence in Chemistry. Analytica Chimica Acta, 210, 9–32. https://doi.org/10.1016/s0003-2670(00)83874-x
Gruenhagen, J. H., Sinclair, P. M., Carroll, J. A., Baker, P. R., Wilson, A., & Demant, D. (2024). The Rapid Rise of Generative AI and its Implications for Academic Integrity: Students’ Perceptions and Use of Chatbots for Assistance with Assessments. Computers and Education Artificial Intelligence, 7, 100273. https://doi.org/10.1016/j.caeai.2024.100273
Guipitacio, J. A. D., Aleman, A. V. B., Bonsubre, C. M., Galleto, J. M. T., Tapere, B. N. B., Caballo, J. H., & Caangay, R. B. (2025). The Nexus Between AI Self-Efficacy and Attitude Towards AI of University Students in Davao City as Moderated by Sex. American Journal of Smart Technology and Solutions, 4(1), 1–7. https://doi.org/10.54536/ajsts.v4i1.3788
Hadžibegović, Z., & Sulejmanović, S. (2014). Fundamental Thermal Concepts Understanding: The First-year Chemistry Student Questionnaire Results. Bulletin of the Chemists and Technologists of Bosnia and Herzegovina, 42, 21-30.
Hashmi, N., & Bal, A. S. (2024). Generative AI in Higher Education and Beyond. Business Horizons. https://doi.org/10.1016/j.bushor.2024.05.005
Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Routledge.
He, L., Bai, L., Dionysiou, D. D., Wei, Z., Spinney, R., Chu, C., ... & Xiao, R. (2021). Applications of Computational Chemistry, Artificial Intelligence, and Machine Learning in Aquatic Chemistry Research. Chemical Engineering Journal, 426, 131810. https://doi.org/10.1016/j.cej.2021.131810
Hedberg, E., & Ayers, S. (2014). The Power of a Paired T-test with a Covariate. Social Science Research, 50, 277–291. https://doi.org/10.1016/j.ssresearch.2014.12.004
Honors Chemistry Test: Atomic History and Structure. (2015). Scribbd. https://www.scribd.com/doc/293180968/atomic-structure-exam
How Generative AI is Reshaping Education in Asia-Pacific. (2024). UNESCO. https://www.unesco.org/en/articles/how-generative-ai-reshaping-education-asia-pacific
Hutt, S., & Hieb, G. (2024). Scaling Up Mastery Learning with Generative AI - Exploring How Generative AI Can Assist in the Generation and Evaluation of Mastery Quiz Questions. University of Denver. https://doi.org/10.1145/3657604.3664699
ICT. (n.d.). Retrieved January 10, 2025, from www.depedimuscity.com. https://www.depedimuscity.com/services/ict.php
Kabudi, T., Pappas, I., & Olsen, D. H. (2021). AI-enabled adaptive learning systems: A systematic mapping of the literature. Computers and education: Artificial intelligence, 2, 100017. https://doi.org/10.1016/j.caeai.2021.100017
Khan, I., Al Sadiri, A., Ahmad, A. R., & Jabeur, N. (2019, January). Tracking student performance in introductory programming by means of machine learning. In 2019 4th mec international conference on big data and smart city (icbdsc) (pp. 1-6). IEEE.https://doi.org/10.1109/ICBDSC.2019.8645608
Kim, T. K. (2015). T-test as a Parametric Test. Kamje. https://doi.org/10.4097/kjae.2015.68.6.540
Leelavathi, R., & Surendhranatha, R. C. (2024). ChatGPT in the Classroom: Navigating the Generative AI Wave in Management Education. Journal of Research in Innovative Teaching & Learning. https://doi.org/10.1108/jrit-01-2024-0017
Lim, W. M., Gunasekara, A., Pallant, J. L., Pallant, J. I., & Pechenkina, E. (2023). Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators. The international journal of management education, 21(2), 100790. https://doi.org/10.1016/j.ijme.2023.100790
Lo, C. K. (2023). What is the Impact of CHATGPT on Education? A Rapid Review of Literature. Education Sciences, 13(4), 410. https://doi.org/10.3390/educsci13040410
Lowe, R. (2021). OpenAI’s GPT-2: The Model, the Hype, and the Controversy. Medium. https://towardsdatascience.com/openais-gpt-2-the-model-the-hype-and-the-controversy-1109f4bfd5e8?gi=2d3fca86e7c0
Lutkevich, B., & Schmelzer, R. (2023). GPT-3. Enterprise AI. https://www.techtarget.com/searchenterpriseai/definition/GPT-3
Marangunić, N., & Granić, A. (2014). Technology Acceptance Model: A Literature Review from 1986 to 2013. Universal Access in the Information Society, 14(1), 81–95. https://doi.org/10.1007/s10209-014-0348-1
Marikyan, D. & Papagiannidis, S. (2023). Technology Acceptance Model: A Review. In S. Papagiannidis (Ed), TheoryHub Book. https://open.ncl.ac.uk / ISBN: 9781739604400
Marsden, E., & Torgerson, C. J. (2012). Single Group, Pre- and Post-test Research Designs: Some Methodological Concerns. Oxford Review of Education, 38(5), 583–616. https://doi.org/10.1080/03054985.2012.731208
Marshall, G., & Jonker, L. (2010). An Introduction to Descriptive Statistics: A Review and Practical Guide. Radiography, 16(4), 1–7. https://doi.org/10.1016/j.radi.2010.01.001
Metcalfe, L. (1996). The Investigation of Student Understanding of Atomic Structure and Bonding.
Fui-Hoon Nah, F., Zheng, R., Cai, J., Siau, K., & Chen, L. (2023). Generative AI and ChatGPT: Applications, Challenges, and AI-human Collaboration. Journal of Information Technology Case and Application Research, 25(3), 277–304. https://doi.org/10.1080/15228053.2023.2233814
Naseer, F., Khalid, M. U., Ayub, N., Rasool, A., Abbas, T., & Afzal, M. W. (2024). Automated assessment and feedback in higher education using generative AI. In Transforming Education With Generative AI: Prompt Engineering and Synthetic Content Creation (pp. 433-461). IGI Global Scientific Publishing. https://doi.org/10.4018/979-8-3693-1351-0.ch021
Nick, T. G. (2007). Descriptive Statistics In: Ambrosius, W.T. (eds) Topics in Biostatistics Methods in Molecular Biology™, v. 404. Humana Press. https://doi.org/10.1007/978-1-59745-530-5_3
Paired T-tests. (n.d.). Coventry. Retrieved November 17, 2024
Paivio, A. (2018). Dual Coding Theory. InstructionalDesign.org. https://www.instructionaldesign.org/theories/dual-coding/
Passi, S., & Vorvoreanu, M. (2024). Over Reliance on AI: Literature review - Microsoft Research. Microsoft Research. https://www.microsoft.com/en-us/research/publication/overreliance-on-ai-literature-review/
Pesovski, I., Santos, R., Henriques, R., & Trajkovik, V. (2024). Generative AI for customizable learning experiences. Sustainability, 16(7), 3034. https://doi.org/10.3390/su16073034
PHL CHED Connect - We Educate as One. (n.d.). Phlconnect.ched.gov.ph. Retrieved January 10, 2025, from https://phlconnect.ched.gov.ph/content/view/governance-of-artificial-intelligence
Raftery, D. (2023). Will ChatGPT Pass the Online Quizzes? Adapting an Assessment Strategy in the Age of Generative AI. Irish Journal of Technology Enhanced Learning, 7(1). https://doi.org/10.22554/ijtel.v7i1.114
Rahman, M. A., Kabir, M. A., Haque, M. E., & Hossain, B. M. (2021). A Machine Learning-Based Price Prediction for Cows: http://doi. org/10.5281/zenodo. 4817941. American Journal of Agricultural Science, Engineering and Technology, 5(1), 64-69. https://doi.org/10.54536/ajaset.v5i1.63
Rai, N. (2016). A Study on Purposive Sampling Method in Research. Ksl. https://www.academia.edu/28087388/a_study_on_purposive_sampling_method_ in_research
Ramdurai, B. (2023). The Impact, Advancements, and Applications of Generative AI. ResearchGate. https://www.researchgate.net/publication/371314493_The_Impact_Advancements_and_Applications_of_Generative_AI
Schnotz, W., & Horz, H. (2010). New Media, Learning From. In Elsevier eBooks, p. 140–149. https://doi.org/10.1016/b978-0-08-044894-7.00740-5
Sigmoid. (2024). GPT-3: All You Need to Know About AI Language Models. Sigmoid. https://www.sigmoid.com/blogs/gpt-3-all-you-need-to-know-about-the-ai-language-model/
Su, J., & Yang, W. (2023). Unlocking the Power of ChatGPT: A Framework for Applying Generative AI in Education. ECNU Review of Education, 6(3), 355–366. https://doi.org/10.1177/20965311231168423
Sun, L., & Zhou, L. (2024). Does Generative Artificial Intelligence Improve the Academic Achievement of College Students? A Meta-Analysis. Journal of Educational Computing Research, 62(7), 1896–1933. https://doi.org/10.1177/07356331241277937
Talikan, A. I., Salapuddin, R., Aksan, J. A., Rahimulla, R. J., Ismael, A., Jimlah, R., ... & Ajan, R. A. (2024). On paired samples T-test: Applications, examples and limitations. Ignatian, 2(4), 943-951. https://doi.org/10.5281/zenodo.10987546
Tamer, H., & Knidiri, Z. (2023). University 4.0: Digital Transformation of Higher Education Evolution and Stakes in Morocco. American Journal of Smart Technology and Solutions, 2(1), 20–28. https://doi.org/10.54536/ajsts.v2i1.1300
Thanh, B. N., Vo, D. T. H., Nhat, M. N., Pham, T. T. T., Trung, H. T., & Xuan, S. H. (2023). Race with the machines: Assessing the capability of generative AI in solving authentic assessments. Australasian Journal of Educational Technology, 39(5), 59-81. https://doi.org/10.14742/ajet.8902
Tobler, S. (2023). Smart Grading: A Generative AI-based Tool for Knowledge-grounded Answer Evaluation in Educational Assessments. MethodsX, 12, 102531. https://doi.org/10.1016/j.mex.2023.102531
Tongco, M. D. C. (2007). Purposive Sampling as a Tool for Informant Selection. Scholar Space. http://hdl.handle.net/10125/227
Tsai, D. C., Huang, A. Y., Lu, O. H., & Yang, S. J. (2021, July). Automatic question generation for repeated testing to improve student learning outcome. In 2021 International Conference on Advanced Learning Technologies (ICALT) (pp. 339-341). IEEE. https://doi.org/10.1109/icalt52272.2021.00108
Wang, Y., Tang, Y., & Ye, Z. S. (2022). Paired or partially paired two-sample tests with unordered samples. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(4), 1503-1525. https://doi.org/10.1111/rssb.12541
Wong, L. H., Chen, W., & Jan, M. (2012). How artefacts mediate small-group co-creation activities in a mobile-assisted seamless language learning environment?. Journal of Computer Assisted Learning, 28(5), 411-424. https://doi.org/10.1111/j.1365-2729.2011.00445.x
Wood, D., & Moss, S. H. (2024). Evaluating the Impact of Students’ Generative AI Use in Educational Contexts. Journal of Research in Innovative Teaching & Learning. https://doi.org/10.1108/jrit-06-2024-0151
Manfei, X. U., Fralick, D., Zheng, J. Z., Wang, B., Xin, M. T., & Changyong, F. E. N. G. (2017). The differences and similarities between two-sample t-test and paired t-test. Shanghai archives of psychiatry, 29(3), 184. https://doi.org/10.11919/j.issn.1002-0829.217070
Yilmaz, R., & Yilmaz, F. G. K. (2023). The Effect of Generative Artificial Intelligence AI-based Tool Use on Students’ Computational Thinking Skills, Programming Self-efficacy and Motivation. Computers and Education Artificial Intelligence, 4, 100147. https://doi.org/10.1016/j.caeai.2023.100147
Zhang, L., & Xu, J. (2024). The Paradox of Self-efficacy and Technological Dependence: Unraveling Generative AI’s Impact on University Students’ Task Completion. The Internet and Higher Education, 65, 100978. https://doi.org/10.1016/j.iheduc.2024.100978
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Nick Andrei V. Cayetano, Francisza C. De Dios, Leonardo M. Francisco, Ishbel Zanthie N. Hernandez, Frances Lei V. Sequito, Sophia Marie C. Sevilla, Louise Beatriz Sison
This work is licensed under a Creative Commons Attribution 4.0 International License.
