NABCEP Exam results

Wondering why it’s taking so long to get the results of your recent NABCEP exam?

A Conversation with NABCEP Board member Jane Pulaski and Dr. Lynn Webb of Professional Testing, Inc.

Since 2003, when NABCEP offered its first certifying exam, NABCEP has used Professional Testing, Inc. (PTI) as a full-service provider of assessment, evaluation and certification services.  Headquartered in Orlando, Florida, with offices in Tallahassee and the Metropolitan Washington D.C. area, PTI provides effective, valid, fair, reliable, and legally defensible assessment and evaluation services, helping businesses, industries, professional associations, and licensure/certification agencies ensure that today”s workforce is competent, and that the organizations administering the assessments are viable and credible.

Because this facet of certification is complicated and frequently misunderstood, NABCEP visited with Lynn Webb, Ed.D, psychometrician at PTI, to learn how it works with a client like NABCEP to develop highly specific certifying exams for PV and solar thermal.

Here’s our conversation:

JP:  Lynn, NABCEP has been offering its certifying exams since 2003.  From time to time, NABCEP gets questions about how the tests are developed and vetted.  NABCEP has used Professional Testing Inc. to assist with the development of the certification examination programs.  Can you talk a little bit about how PTI works with NABCEP on its very specific PV and Solar Thermal certifying exams?

LW:  The test development process is quite thorough, probably more so than most people would suspect.  It begins with a study of the tasks that people in the field of photovoltaic energy actually do; in this case, the PV and Solar Thermal Task Analyses.   From the results of the study of what people do on the job, we develop the blueprint for the certification examination.

A panel of content experts volunteer their time and expertise and serve as the examination committee. The examination committee uses the blueprint to write questions that will appear on the test.  All questions are referenced to a published source, rather than reflecting someone’s individual practice or opinion.  After an Examination Committee member writes a question, it is reviewed by the full panel of experts. Because the certification is national in scope, the Committee ensures that each question uses terminology that would be understood nationally, and if there is a regional component to the question (e.g., longitude or latitude), it is stated clearly.

JP:  So it sounds like this takes a long time.  I’m guessing it’s not a process where a couple of folks sit around a table and conjure up questions.

LW:  Exactly.  The Examination Committee is a very dedicated group that donates a lot of time and expertise to NABCEP.  Professional Testing guides the Committee to ensure that each question has only ONE best answer. There may be more than one answer that could be plausible, but one is clearly the best.  Professional Testing also guides the Committee to ensure that each question is properly referenced, stated clearly, and is free from bias or sensitivity issues. We ensure that test-wise examinees cannot guess the answer quickly from a clue within the question, rather than having the requisite knowledge.

The Examination Committee chooses from a pool the 60 questions that will comprise a test. No one is given a score on the test until Professional Testing checks the statistical performance of every question. Even after all the work described above, it may be that a question is interpreted differently by the various examinees – and the statistics will show a problem.  It may be that far more examinees choose an option that was not selected by the Committee as the best. Or, it could be that for a specific question, the people who scored highest on the test answered incorrectly, while the people who scored lowest on the test overall answered this question correctly.

That indicates to me that something about the question was confusing or misleading. The Examination Committee reviews all questions with unexpected statistical performance and decides if there was something vague or flawed in any question. If so, everyone is given credit for the question.  We don’t have many instances of giving everyone credit – but it is a great way to make sure that we are being fair about every question on every test.  The Examination Committee also considers the comments that examinees write when taking the examination.

JP:  So is this one of the reasons why it takes so long to get a score report?  Most people wish they could know their results sooner.

LW:  Yes, reviewing the statistical performance of every question and making the final scoring decisions, and sometimes re-scoring everyone just for one question – that takes some of the time between when people leave the test until they receive results.

JP:  Still, it seems to take forever for people to get the results of the test.

LW:  There is another step that adds to the time of the process. The Examination Committee conducts a study at the beginning of the certification program to recommend the score to pass the examination to the NABCEP Board. The standard to pass is criterion-referenced, which means that everyone who scores at the passing score or higher will pass, and everyone who scores lower than the passing score will fail. It’s the opposite of grading on a curve – it doesn’t matter if you test with the most able group or the least able group – you must meet the criterion to pass (or passing score).  The passing score is based on the 60 items of a specific test.

Instead of conducting passing score studies every six months for the new PV installer tests, Professional Testing statistically equates the tests. It is unlikely that we could make the test exactly the same in difficulty of questions every six months. Yet, if some people take the test in March, and other people take an easier test in September, we can all see that this would be unfair to the people who tested in March.

In order to ensure that the test maintains a constant level of difficulty, we statistically equate the passing standard from one administration to the next.  If the test in March, for example, is a little bit easier than the one from the previous fall, then examinees will need to answer more questions correctly than was required in the fall. The criterion to pass, set by the Examination Committee and approved by the Board, is upheld through this statistical process of equating.  (We’ll use statistical equating on the Solar Thermal examination once the number of people taking the test grows.)

JP:  So let me see if I follow:  after the verification that each of the 60 questions is okay and will count towards the examinees’ scores, AND the standard to pass is statistically equated to the criterion set by the Examination Committee and approved by the Board, then people receive their test results?

LW:  Yes – but the work of the Examination Committee keeps going!  If there were problems with any of the questions, they have to agree on how to fix the questions or delete them from the question pool. The test is different every six months, so the committee writes new questions annually. All those questions are reviewed….

JP:  Over and over and over?

LW:  Yes.  It’s not an inconsequential commitment to be on the Examination Committee.  But it’s a great way to network with other interested professionals, and they have the satisfaction of ensuring that only people with the requisite knowledge are NABCEP-certified.  The certification program is designed to ensure the safety of the public.