Abstract
Teacher assessment literacy is a key factor in the success or failure of teaching. Studies investigating in-service and pre-service teachers’ assessment literacy have revealed that even teachers who have taken assessment courses lack assessment literacy. The aim of this research is to propose an effective strategy for improving pre-service teacher assessment literacy. The strategy is named ‘P2R-WATA (Practicing, Reflecting and Revising with WATA system) assessment literacy development model’, and was developed on the basis of the WATA system (Web-based Assessment and Test Analysis system) and the Standards for Teacher Competence in Educational Assessment of Students (STCEAS) developed by American Federation of Teachers (AFT), the National Council on Measurement in Education (NCME) and the National Education Association (NEA). The WATA system offers personalized and situated supporting resources and provide opportunities for pre-service teachers to assemble tests and administer them to students online. Furthermore, the WATA system facilitates performance of test analysis and item analysis, and enables pre-service teachers to review statistical information from the test and item analyses to understand student learning failures and revise the test items. In order to test the effectiveness of the P2R-WATA assessment literacy development model, the Quasi-experimental method was adopted. In this research, 60 pre-service teachers were assigned to an experimental group and a control group (30 pre-service teachers each). The experimental group used the P2R-WATA assessment literacy development model, while the control group used the P2R assessment literacy development model, which was lacking in the WATA system. After one semester, pre-service teacher improvement of assessment knowledge, assessment perspectives, assessment skill and assessment item quality in the experimental group and the control group were compared. In addition, pre-service teacher attitudes towards the WATA system in the experimental group were also surveyed. Furthermore, the relative superiority of the P2R-WATA assessment literacy development model and factors influencing experimental group pre-service teacher learning effectiveness were also investigated. Finally, improvements to the current P2R-WATA assessment literacy development model were discussed. The results and conclusions of this research are following: 1. The assessment literacy improvement of experimental group was significantly better than that of the control group. 2. The assessment item quality of experimental group was significantly better than that of control group. This demonstrates that there is a significant correlation between assessment literacy and assessment item quality. 3. Experimental group pre-service teachers hold a positive attitude towards the WATA system and expressed a belief that the WATA system can help them improve their assessment literacy. 4. The superiority of the P2R-WATA assessment development model over the P2R assessment literacy development model lies in the fact that the WATA system is equipped with the Triple-A Model. The Triple-A Model provides pre-service teachers with the ‘scaffolding of complete assessment administration’. Pre-service teachers are able to assemble and administer tests, and then perform test and item analysis, analyze learning failures and revise their tests. Experimental group pre-service teachers said that these designs were effective in improving their assessment literacy. 5. Experimental group pre-service teachers will have lower learning effectiveness if they confront troubles like ‘unfamiliar with the WATA system’, ‘unfamiliar with the e-Learning method’, ‘more assessment materials in the WATA system are required’ and ‘more practicing task are required’. If pre-service teachers are familiar with the ‘scaffolding of complete assessment administration’ provided by the Triple-A Model in the WATA system, and use the statistics of the test and item analysis to revise their assessment mistakes and understand student learning failures, they will improve the effectiveness of their learning. 6. The interface of the WATA system for the P2R-WATA assessment literacy development model should be enhanced to promote user friendliness. In addition, individual differences should be considered in the design of the human-computer interface and the electronic assessment materials to provide equal learning opportunities for different types of learners. More examples of electronic assessment materials to help learners understand the concepts of educational assessment are necessary. Furthermore, there also should be more opportunities to practice the tasks in the P2R-WATA assessment literacy development model, because these opportunities will enable learners to have sufficient experience in revising items and to verify the effectiveness of their item-revision strategies.