|Certification Exam Interim Report|
The Exam Subcommittee under the guidance of the Exam Subcommittee Chair, John Matthews, is now well along in its work to create the first national certification exams for Ayurvedic professionals. These exams represent an important milestone for NAMA and the Ayurvedic profession in the U.S.
The Certification Exam Interim Report, included below in its entirety and also available to download here, is meant to bring our community up to speed and answer questions about the exam creation process.
Certification Exam Interim Report
August 20, 2015
SCOPE OF WORK
The National Exam Subcommittee has been charged with facilitating the process of producing two national certification exams (Ayurvedic Health Counselor and Ayurvedic Practitioner). We have organized a panel of approximately 30 Subject Matter Experts (SMEs) from among experts in our field who are working under the guidance of Kryterion Global Testing Solutions’ chief psychometrician to:
The work is being done by the SME panel with expert support and guidance. The SME panel periodically reports to the National Exam Subcommittee who then reports to the full Standards Committee. The Chair of the Standards Committee reports to the Board of Directors, with additional reporting provided to NAMA’s membership and other sectors of interest.
In the course of their work, the SMEs inevitably encounter questions or suggestions that are outside their charge, but which may be important to the work of the Standards Committee as it develops and revises competencies and guidelines; and to the board as it considers policies and procedures to guide and support the creation of the exams and the operational and administrative structures that implement them. As these questions and suggestions arise, I take careful note and take them to the subcommittee who then determines what is appropriate for further referral up the chain to the full Standards committee and perhaps to the board.
The next subcommittee meeting will take place right after the panel completes the JTA process (see below). My next interim report will be published once we have completed the blueprint survey and are set to begin item writing.
We began in late April 2015 and have completed ten virtual meetings lasting approximately two hours each. We have had an average attendance of fifteen SMEs, along with two administrators (myself and Sheila Higgins of my staff) and Kim Thayn, the psychometrician. We have completed the JTA for the Counselor exam and will start the JTA process for the Practitioner exam during the first week of September. A more detailed description of the process follows.
Detailed Time Frame
FEBRUARY 23, 2015 – APRIL 30, 2015
Prework – TASK COMPLETE
Psychometrician worked with one or two lead SMEs to align the competencies for each area to the scope of practice.
MAY 1, 2015 –SEPTEMBER 15, 2015
Job Task Analysis – TASK IS COMPLETE FOR THE FIRST OF TWO EXAMS; NOW BEGINNING FOR THE SECOND EXAM
A JTA is a systematic process for collecting information regarding functions (i.e., responsibilities and duties) and tasks performed on a job, as well as the knowledge and skills (i.e., competencies) required to perform those tasks.
The results of the job task analysis describe the breadth and depth of knowledge and skills that must be covered by the certification exam in order for it to be deemed valid, credible and useful.
Results of the JTA are used to create test objectives for the exam. Test objectives help ensure that the knowledge and skills measured by the exam are the same as those used to perform the job (i.e., measure applied job knowledge). SMEs will identify the primary tasks, subtasks, and supporting knowledge, skills and abilities expected of both an Ayurvedic Practitioner and Ayurvedic Health Counselor certified professional.
SEPTEMBER 15, 2015 – SEPTEMBER 30, 2015
Following the JTA, Kryterion will use the SME-identified testing objectives to create an online test blueprint survey. The survey will be distributed to a large population of the profession and will be available for a minimum of two weeks to allow for maximum flexibility in administration to accommodate different professional schedules.
The psychometrician, along with a focus group of SMEs, will review the analysis of the survey results in a one-hour web meeting
Kryterion will summarize the test development processes that have been completed in a final report. The report will include:
OCTOBER 1, 2015 – JANUARY 31, 2016
Once the blueprint is complete, items (test questions) must be written. Kryterion will facilitate a virtual item-writing workshop (two hours) to train SMEs in the principles of effective and efficient item writing. Each SME will be asked to write 3-5 items; Kryterion will then host another two-hour review meeting to ensure an understanding of item writing principles and improve the quality of the additional items to be written.
The goal of these workshops is a deliverable of 150 items per exam for a total of 300 draft test items that are ready for the item review process.
Psychometric & Grammar Edits
Kryterion will then perform psychometric, sensitivity and English grammar edits on all items. The psychometric edit verifies that each item conforms to applicable psychometric item writing standards. The sensitivity edit ensures that the items do not appear to favor any particular nationality, race, religion or gender. The English grammar edit corrects grammar, usage, readability, clarity and consistency of usage.
After editing, SMEs will review the items to ensure the substantive nature of the item was not inadvertently altered during the editing process.
Virtual Technical Review
At each step of the way, Kryterion will work with groups of SMEs in technical reviews of the items for:
This will be the most time-intensive part of the test development process. It takes approximately 1 hour to review 8-10 items. Each review session is conducted via telephone conference call and a virtual classroom.
Kryterion will perform a final edit for spelling, punctuation and style before beta testing.
February 1, 2016 – March 31, 2016
The months of February and March will be devoted to the Alpha Review process. During this time, Kryterion will set up the exams in Webassessor for review. The SMEs will be asked to indicate the correct answer to the item as well as identify “enemy items” (i.e., an item that provides the answer to another item or item that measures the same content) and note any other issues with the items. Kryterion will consolidate comments and suggested edits from the SMEs, and facilitate a two-hour virtual meeting to review and resolve the noted issues, if necessary.
Kryterion will use feedback from the alpha review to inform the assembly of the beta test form(s).
NAMA is responsible for conducting a final quality control review of said forms before the beta testing window begins. This process is to be determined by NAMA’s Board of Directors.
April 1, 2016 – June 30, 2016
The purpose of beta testing is to collect data from Test Taker responses on each item to statistically analyze the item’s performance and determine whether it should be retained, discarded or revised. During beta testing, the items must be administered to a minimum of 60 examinees representing the target audience. The beta exam is administered at secure testing sites with certified proctors.
July 1, 2016 – July31, 2016
After beta testing is complete, Kryterion will perform an item analysis, which provides valuable information on item difficulty, discrimination (i.e., how well an item discriminates between high and low test performers), and answer choice analysis (i.e.; proportion of test takers choosing each option). A psychometric analysis is compiled by Kryterion, which identifies items that exhibited unusual statistical performance. NAMA will make the final determination to retain, discard, rework or retire items from the test bank.
August 1, 2016 – August 15, 2016
Kryterion will use analysis and form selection data to produce two comparable test form(s). Iterative analysis will continue until the forms are as statistically equivalent as possible in both difficulty and content.
August 16, 2016 – October 31, 2016
Standard Setting: Pass/Cut Score Study
Kryterion will perform a modified Angoff standard setting procedure to provide a recommended passing/cut score range. The modified Angoff is conducted during two two-hour virtual meetings. In the first meeting, SMEs will learn about using the modified Angoff test range. In the Angoff method, SMEs are asked to predict the proportion of minimally competent examinees that would answer each item correctly. Each SME will independently complete one round of test predictions, which Kryterion will compile and have SMEs review the results in a second virtual meeting.
After the second meeting, each SME will independently review the items for a third and final prediction. Kryterion will compile and analyze the final predictions and provide a recommended cut score range. NAMA is responsible for determining the final cut score.
After the score is finalized, beta test takers may be notified of their scores.
November 1, 2016 - onward
Kryterion will use all data, feedback and analysis to produce the final certification exam, which will be offered to the public in early December 2016.