Rescoring of FY2014 OCTCA Field Tested Assessments

Rescoring of 2013-2014 Field Tested Assessments

During 2013-2014, a field test of newly-written End-of-Course (EOC) tests was conducted by the maintainers of WebXam, CETE at The Ohio State University, who coordinated with the Ohio Department of Education. The field test process, just like the large group review involving instructors, is an important part of psychometric quality assurance. The purpose of field testing is to examine scores on both individual items and entire modules. This QA process helps CETE staff to select quality items to create operational test forms.

When students took field tested EOC Tests this spring, the tests had a large number of test items, typically 80 or more. Also, students did not receive a pass/fail indication. In May, CETE staff analyzed these field test results. For EOC Tests with a sufficiently large number of testers, or approximately 125 students, CETE psychometric staff dropped poorly performing items and selected final test forms of 40 items so that tests could be rescored for accountability purposes. Every student’s test result is based on the set of items chosen. The percentage score may increase or decrease from the raw score shown to the student during field testing.

Districts can now run reports on the WebXam site to get these EOC scores. Reports will show the following:

  • Rescored field tests appear with a score and a pass/fail indicator. These scores will be combined to create the composite Pathway Test score.
  • Non-rescored field tests appear without a cut score and without a pass/fail indicator. These scores are not part of the composite Pathway Test score.

So, for field tested assessments, the overall score for a Pathway Test will consist only of scores for EOC Tests that were rescored.

The chart below shows which EOC Tests had a sufficient number of test takers for psychometric analysis. For these EOC Tests, the proficient and advanced cutoff scores are also shown. EOC Tests not listed were not rescored; they will be given another year to increase participation. When additional EOC Tests are rescored at the end of 2014-2015, you will be able to see students’ scores on the final forms for EOC Tests taken in 2013-2014.

EOC Test Name # Tests Proficient % Advanced %
Health Science Career Field
Athletic Injuries & Prevention 329 61 82
Biomedical Engineering 180 72 89
Clinical Laboratory Techniques 185 77 92
Exercise & Athletic Training 365 67 79
Exercise Physiology & Biochemistry 131 53 80
Fitness Evaluation & Assessment 293 54 83
Human Anatomy and Physiology 755 54 75
Lifespan Development and Medical Intervention 269 58 82
Medical Terminology 993 52 81
Nutrition & Wellness 289 60 81
Nutrition and Wellness 219 60 81
Patient Centered Care 719 73 89
Patient Centered Care and Diagnostics 250 64 84
Principles and Practices of Biomedical Technology 283 67 89
Principles of Allied Health 888 63 77
Engineering and Science Career Field
Digital Electronics 197 72 94
Engineering Design 1014 50 93
Engineering Principles 545 64 91
Manufacturing Technologies Career Field
Gas Metal Arc Welding 334 74 85
Machine Tools 209 61 90
Machining with Industrial Lathes 199 64 92
Machining with Industrial Milling Machines 155 70 85
Manufacturing Operations 335 73 91
Construction Technologies Career Field
Carpentry and Masonry Technical Skills 191 75 94
Construction Safety 682 64 92
Construction Technology–Core and Sustainable Construction 387 65 88
Residential Electrical Systems 228 75 93
Structural Coverings and Finishes 185 81 93
Law and Public Safety Career Field
The American Criminal Justice System 494 47 70
The Correctional System and Services 196 60 80
Foundations of Firefighting and EMS 185 71 82
Homeland Security Protecting America’s Critical Infrastructure 223 70 81
Investigations and Forensics in Criminal Investigations 313 61 84
Police Work and Practice in Public Safety 473 56 80
Security and Protective Services 187 53 77

 

New Indicator of Incomplete Pathway Test

Please note that, although the format of the score output file to the state’s data system, EMIS, has not changed, there is a new score designation this year. Pathway tests that have moved to the EOC format require at least three EOC scores to be considered a valid overall test score. Students should take the EOC test for each course completed.  However, in some cases, students have started testing but have fewer than three EOC scores. In this case, a numeric score and pass/fail value will not be reported. Instead, there will be an uppercase letter I in place of the percent score and a letter I in place of the pass/fail indicator. This is now reflected in the WebXam score reports, which show a “N/A” for passing status if the Pathway Test is considered incomplete.

 

OCTCA Score Reporting Changes for 2013-2014

Some of the WebXam cutoff scores have been adjusted this year.  This has caused instances
in which a few students who had passed certain modules last year could be shown as failures.
CETE and the Ohio Department of Education have repaired this problem
so that any student who passed a modular sequence last year will now still be reported with a passing score.  Read below for additional detail.

 

Reversion of Cutoff Scores

Between the 2012-2013 and 2013-2014 school years, the developers-maintainers of WebXam, CETE at The Ohio State University, conducted maintenance on operational test forms. An announcement (available at https://go.osu.edu/FY2014maintenance) explained the scope and impact of test maintenance on length of test forms (# items) and cutoff scores. Note that this is not related to the field test for end-of-course forms, which is a separate issue.

As announced, many pathway-level cutoff scores changed in 2013-2014 due to the selection of different items for this year’s form. Cutoff scores are determined in part by instructors’ ratings of individual items. Cutoffs could increase, decrease, or be unchanged based on the items in the revised forms.

In some cases, students who are CTE Concentrators in 2013-2014 may have taken all or some of a Pathway Test in 2012-2013. CETE and the Ohio Department of Education have determined that these students may be unfairly disadvantaged if one of the following occurs:

  1. The student completes a Pathway Test in 2012-2013 and achieves a passing score. Rather than report the score to EMIS during 2012-2013, the district reports the score in 2013-2014. This causes the student’s score on the 2012-2013 test to be compared to the 2013-2014 cutoff score. If the student’s score was above the 2012-2013 cutoff but below the 2013-2014 cutoff, the student will be unfairly reported as failing.
  1. The student takes some modules of a modular Pathway Test in 2012-2013 and completes the assessment in 2013-2014. Even if the majority of modules were taken during 2012-2013, the student’s overall Pathway Test score will be compared to the 2013-2014 cutoff score. If the student’s score was above the 2012-2013 cutoff but below the 2013-2014 cutoff, the student may be unfairly reported as failing.

For 2013-2014 reporting, CETE and ODE have taken the following action to address scenarios 1 and 2:

If the cutoff score for a Pathway Test increased from 2012-2013 to 2013-2014, that cutoff score is reverted to the FY2013 value.

Please note that, as a result of this action, some students who were reported as failing after testing in 2013-2014 will now be reporting as passing. These students can be considered passing for the purposes of reporting CTE assessment scores for 2S1 Technical Skill Attainment measurements.

 

Passing all End-of-Course Tests

Additionally, there is third scenario which may occur with End-of-Course (EOC) testing:

  1. A student takes an EOC Test for all courses in the CTE Program and passes all EOC Tests. However, the student fails the overall Pathway Test because the cutoff score for the Pathway Test is higher than the cutoffs for each EOC Test taken.

CETE and ODE have taken the following action to address the third scenario:

If a student has a complete set of EOC Tests and has passed all EOC Tests, then the student will be indicated as passing the Pathway Test even if the student’s overall score is less than the overall cutoff score.

Again, students who were marked as failing earlier in the year may now be marked as passing because of this change. Students who do not pass all EOC Tests are not affected by this action.

The changed scores will be seen in two areas of WebXam, a) score reporting for users (teachers, local administrators) and b) data output files that CETE makes available to ITC via secure log-in and download.

All changes listed above are effectively immediately and will be included in the yearly output files that will be made available on or before June 13, 2014. For more information about reporting to EMIS, please see: https://go.osu.edu/EMIS .

 

Questions?

Contact WebXam Support at webxam@osu.edu if you need more information about this change.

May 23, 2014 Presentation

On May 23, 2014, at a workshop conducted by Central Region Tech Prep Center at Delaware Area Career Center‘s North Campus, Jim Austin and Dan Keck of CETE presented on WebXam to administrators and teachers associated with career-technical education (CTE) programs.

 

Pre-test question when activating

Activation Question
The Ohio CTE assessments in WebXam are designed as post-tests, to be taken after students have received the relevant instruction in their CTE program. However, we are aware that many districts, looking for assessments to use to measure Student Learning Objectives, have been using WebXam tests in the junior year as a pre-test and in the senior year as a post-test.

In order to analyze score data properly, we need to know how assessments are being used. Beginning April 24, when you as an administrator or teacher log in to WebXam and activate a test for students to take, you will be asked whether the students have received instruction in the content – in other words, whether it is a post-test or a pre-test. Please see the blue box and the rounded box below it in the screenshot.

The activation procedure is the same, except for the added step of responding to the pre-test/post-test question.  The score reporting procedure is also the same: scores for all tests taken this year will be included in score reports.

Please contact WebXam Support with any questions.

Changes to names in 47MC Biomedical

The names of the End-of-Course Tests in the 47MC Biomedical Pathway Test have been changed to align to the course names published by The Ohio Department of Education. The content of the End-of-Course Tests has not changed. The test items and standards covered are the same as before – only the names have changed.
Please refer to the following chart for the changes.

 

Old name in WebXam Current name in WebXam
Laboratory Techniques Principles and Practices of Biomedical Technology
Cell Culturing/Fermentation Biomedical Engineering
Biochemistry Biochemistry of Health
Molecular Biology Techniques Biotechnology for Health and Disease
DNA and Protein Analysis Genetics of Disease

February 2014 WebXam Newsletter

The February 2014 WebXam newsletter is now available at the following link:

http://news.webxam.org/category/newsletters/2014-february/

Topics for this newsletter include:

  • Deadlines for Testing and Uploading
  • Test Security Policies Reminder
  • Delay of 28MC Manufacturing Operations
  • New Student Testing Page
  • New Item Writing for 2014-2015 Field Testing
  • Revised Operational Assessments for 2013-2014
  • WebXam User’s Workshop Recording

Deadlines for Testing and Uploading

Friday, April 11, 2014 is the deadline for submitting upload files of student data.  If you have questions about uploads, please contact your school district’s EMIS coordinator, your ITC, or WebXam Support.

After April 11, students can still be added manually by logging on to WebXam, clicking the Secondary Students menu item followed by the New menu item, and filling out the form.

 

Friday, May 2, 2014 is the last day of online testing for Ohio Career-Technical Competency Assessment (OCTCA) secondary students for the 2013-2014 school year taking the field tested assessments:

Construction Technologies Career Field

  • 15MC Construction Design and Management
  • 27MC Mechanical, Electrical and Plumbing
  • 45MC Structural Systems

Engineering and Science Technologies Career Field

  • 26MC Engineering Design

Health Science Career Field

  • 22MC Health Information Management
  • 23MC Exercise Science & Sports Medicine
  • 25MC Allied Health and Nursing
  • 47MC Medical Bioscience

Law and Public Safety Career Field

  • 41MC Firefighting and Emergency Medical Services
  • 43MC Criminal Justice

Manufacturing Career Field

  • 28MC Manufacturing Operations

Friday, June 6, 2014 is the last day of online testing for Ohio Career-Technical Competency Assessment (OCTCA) secondary students for the 2013-2014 school year taking all other assessments.

Friday, June 13, 2014 is the deadline for WebXam end-of-year score output files to be made available to ITCs for download.

Be sure to complete your testing on or before June 6, to ensure that scores will be reflected in our end-of-year reports. For more information on the reporting of scores to the state EMIS system, see https://www.webxam.org/ASP/documentation/qs_emis_reporting.asp.

 

Test Security Policies Reminder

Once a year, in the interest of test validity, users who administer or take OCTCA assessments in WebXam are presented with the User Agreement again to read and agree to before proceeding to the website.

The User Agreement links to the WebXam Test Security Rule Book, also available at the top of our Information Resources web page at http://www.webxam.org/ASP/documentation/info_docs.asp. Please be reminded of the following security policies:

  • An individual who proctors a test for a student must:
    • be a paid employee of the school district.
    • not be a student, student teacher, or unpaid volunteer.
    • not be a teacher of the pathway or course being tested.
  • BYOD Policy: While WebXam staff understands that local school districts and CTPD may allow students to bring their own devices for instruction and other educational activities, it is NOT permitted to use personal devices (tablets, netbooks, smartphones, etc.) to take WebXam tests. If the district owns the device, and if students do not have administrative rights on the machine, then a tablet may be used for testing.
  • Use of a non-programmable, non-graphing calculator is permitted for WebXam assessment.  So, for example, the TI-30X IIS calculators used for the OGT are also appropriate for WebXam testing.  Calculator apps on cell phones are not permitted.