Beginning in 2013, Ohio instructors participated in workshops at CETE, part of The Ohio State University, to create test item banks for Career-Technical Education End-of-Course (EOC) Tests for Ohio’s CTE Testing System. Required tests are post-tests, given after instruction and funded by the Ohio Department of Education, that provide scores that can be used in the calculation of the state’s technical skill attainment indicator of performance (2S1) specified by the Carl D. Perkins Career and Technical Education Improvement Act of 2006. Pretests are an optional, but valuable, way to measure gain. As of 2018-19, generation of initial course-based tests (pre- and post-) is complete for all Career Fields
When designing test forms for Technical Testing Project, the primary goals are content coverage, reliability and validity. Panels of subject matter experts (SME) – Ohio secondary and postsecondary instructors, and business and industry practitioners – write test questions linked to outcomes-competencies in Career Field Technical Content Standards. Quality assurance of test questions is performed using a two-part review, first by checking the content and accuracy of the questions, and then by rating each test question. Test items are then field tested with students in Ohio CTE programs across the state. Results are analyzed by the CETE psychometric staff. Poorly performing questions are dropped, and final forms are chosen to optimize test reliability, content coverage, and instructor ratings of essentially and quality.
Before each school year, CETE staff choose two 40-item test forms for finalized EOC Tests – one a pretest and the other a posttest. The two tests are not identical, but a percentage of questions are common. An even number of items are not written for each outcome. Instead, more items are written for more important outcomes. For an overview of what percentage of items are written to each outcome for each test, please see the blueprints under About the Tests.
In general, each EOC Test is a blend of questions at two levels of challenge: C1 and C2. Approximately 30-40 percent of items will be at the C2 level (higher in Bloom’s taxonomy – mainly application, analysis, or evaluation). The remaining items are at the C1 level (knowledge and comprehension). In addition, approximately 30-40 percent of items for each test are scenario-based. A scenario describes a short entry-level workplace situation that provides information needed to answer associated items. Graphics (tables, charts, figures, maps) may also be included to provide information needed to answer the test question.
Each EOC Test used for pretesting corresponds to a course as defined by the Ohio Department of Education. Specific information regarding each individual pathway test— EOC Tests to administer, EOC Tests included in each Pathway Test, and number of questions in each EOC Test—follows in the next section. To locate the more information about the Ohio Career Field Technical Content Standards covered by each course outline and associated EOC Test, go to the WebXam Information Resources page at https://www.webxam.org/public/AboutTheTests.
As stated above, not every pathway and course had an associated pretest at the time of system launch. Available course-based pretests represent all courses released by ODE across Career Fields except for the Career-Fields in field testing. Item writing follows content standards revision, and the ODE-CTE website includes the proposed renewal schedule.
Other than reliability and validity for assessing student knowledge and skill, CETE as test developer provided some evidence for content and criterion validity. Staff are always looking to partner in collecting data that will help to support score interpretations for purposes of teacher effectiveness; this means that gain is related to other measures of teacher effectiveness. The pretest user is defined as the school district. Finally, we want to remind local districts that, according to Ohio Revised Code interpretation contained in the ODE Administrative Rule, districts are NOT required to use Category B vendors for OTES if another acceptable option is available (for example, Category C local options using Student Learning Objectives—SLO).