STAR BASE Consulting's
NOW YOU KNOWTM
Not All Assessments are Created Equal
IT skills assessment should provide objective measurements that are critical for understanding the strengths of potential and current employees. But gaining these advantages requires using the right methods.
What characteristics should you look for in a technical skills assessment to ensure you get a reliable measure of IT proficiency?
Adaptive Testing :
Random Question Selection/Multiple Levels of Difficulty :
Independently Adapting Sub-Topics :
Degrees of Difficulty Assigned to Each Question :
Multiple Correct Responses—Partial Understanding :
Percentile Rankings :
Absolute Strengths/Weaknesses :
Sample Question Type :
Proctored vs. Self-Administered Assessments :
Access to TeckChek :
Test Development :
Test Results
STAR BASE Consulting can provide you a complete list of NOW YOU KNOWTM assessments.
Adaptive testing
What is adaptive testing?
Adaptive testing methodologies select questions with a specific level of difficulty based on previous responses. Thus, the adaptive testing engine "adapts" the question selection process according to the professional's abilities, eliminating questions that are too easy or too difficult for the test taker. This method of testing allows for the collection of the required feedback with far fewer questions.
Why is adaptive questioning important?
Adaptive questioning is the most efficient, effective means of knowledge-based testing. Responses provide the adaptive testing engine with the information it needs to deliver only those questions that are appropriate for individual abilities. The benefits of this approach include:
- Time isn’t wasted on inappropriate questions
- Assessments are not burdened with information that isn’t needed for a reliable measure of technical proficiency
- Results show areas of strength and weakness clearly and accurately
Can you give an example showing why adaptive testing is so efficient?
Adaptive testing can be compared to the Olympic High-Jump event (which employs adaptive methodology) to illustrate efficiency. The bar is initially set at a level—let’s use 5 feet for this illustration. If a person fails to clear 5 feet after several attempts, the bar can be lowered to 4 feet. Raising the bar above 5 feet now would be futile since it’s already established that the person cannot clear 5 feet. At 4 feet, the person clears the bar, so it’s raised to 4.5 feet. There’s no reason to lower the bar further because the person has already demonstrated his ability to jump over 4 feet. If the person fails to jump over 4.5 feet, the bar could be set between 4 feet—which it’s known the person can clear—and 4.5 feet—which it’s known the person cannot clear. The bar continues to move in ever-smaller increments based on subsequent successes and failures.
This is exactly how an adaptive exam works. Question difficulty is adjusted up or down based upon the test taker's prior responses.
Why don't all companies offer adaptive testing?
The adaptive test development process is much more complex than that required by a non-adaptive assessment. Few companies that specialize in testing and assessment actually deploy adaptive testing methodology. The use of adaptive testing should be the primary consideration and requirement when evaluating an assessment product or service.
« page top
Random question selection/multiple levels of difficulty
How does an adaptive test select questions for a test taker?
Once the adaptive testing engine has evaluated a response and determined the appropriate level of difficulty for the next question, a follow-up is randomly selected from a pool of available questions at the determined difficulty level. For this purpose, TeckChek maintains several pools of questions at various difficulty levels. The random selection process allows individuals to take an assessment more than once and receive different questions that are assigned the same level of difficulty each time they take the test. This process helps ensure the test result is a true measure of the individual's knowledge, and not a reflection of their ability to learn and study test questions.
« page top
Independently adapting sub-topics
Does knowledge in one aspect of the technology impact the difficulty level of questions selected in other areas of the technology?
Not in a TeckChek assessment. TeckChek’s unique methodology breaks each technology into 12-18 sub-topics. Each sub-topic contains a pool of questions at all difficulty levels. The adaptive testing engine selects the next question for that sub-topic based on prior responses within that sub-topic. Thus the assessment adapts independently within each of the sub-topics.
What is the benefit of independently adapting sub-topics?
By adapting independently within sub-topics, the knowledge in one sub-topic does not impact the difficulty level of questions in other sub-topics. This allows each sub-topic to be independently evaluated and identified as a specific strength, weakness or proficiency. Assessments that do not adapt independently within sub-topics cannot provide accurate strengths and weaknesses because the difficulty of questions in each sub-topic has been influenced by knowledge levels in other sub-topics.
« page top
Degrees of difficulty assigned to each question
What are question ‘weights’?
Question weights are values assigned to each question measuring the difficulty level and relative importance of the material being tested. Typically, the higher the weight, the greater the degree of difficulty or importance. This is not unique to testing and is often employed in sporting events. Diving fans know that weights are assigned to different dives based upon difficulty. The more difficult dives get higher weights. A diver's final score is factored by the difficulty (weight) of his dive and his/her actual performance on the dive.
In an adaptive test, the number of correctly answered questions is not as important as the difficulty and relevance of those questions. For this reason, all TeckChek questions are weighted for difficulty and importance. The more difficult the question, the more credit received for a correct answer and the less credit lost for incorrect answers.
What are the benefits of assigning weights to the questions?
Weighted questions allow for much more granular insight into proficiency levels, thus enabling the individual(s) using that result to make better, more educated decisions related to hiring, training, professional development and resource management.
Further, in an adaptive assessment, test takers will receive questions of varying difficulty levels based upon prior responses. If there were no weights, the scoring would not be fair to those who were doing well and receiving more difficult questions.
Assigning each question with weights representing different areas of knowledge enables independent scoring in those areas. The various weights used enable TeckChek results to include five distinct scores and percentiles (Overall Proficiency, Book Knowledge, Practical Experience, Conceptual Understanding and Work Speed), thus allowing comparisons of book knowledge to practical experience and other deep insights into the nature of proficiency.
« page top
Multiple correct responses—partial understanding
What are multiple correct responses?
TeckChek questions are uniquely formulated to provide the maximum feedback, enabling the test taker to express a very wide range of understanding in each question. This is accomplished through a methodology called ‘Multiple Correct Response.’ Each TeckChek question has five possible answers, of which up to three can be correct. The test taker is never told how many correct answers there are to any given question, but are allowed to select up to three answers. Credit is gained for every correct answer selected and lost for every wrong answer selected. Credit is also lost for every correct answer not selected.
Just as questions are weighted, so are individual answers. The amount of credit gained or lost with each answer is determined by the comparative correctness of that answer. Some incorrect answers are so far off the mark that selecting them indicates that the test taker is guessing, which results in the loss of a lot of credit. Other incorrect answers are more intelligent and indicate a certain amount of partial knowledge. Therefore, less credit is lost for a more intelligent incorrect answer.
What are the benefits of using multiple correct response?
Multiple correct answers allow for very detailed feedback since they provide increased accuracy, reliability, and usability. By enabling the selection of up to three answers, there are actually 25 unique answer combinations to every question. Each one of these 25 combinations implies a different level of knowledge about the subject, each has its own unique credit value based upon the combination of answers selected and not selected, and each of the 25 answer combinations leads to a different level of follow-up question within the adaptive algorithm.
Where a traditional test with one correct answer typically provides binary feedback (correct or incorrect), TeckChek’s 25 answer combinations allow the test taker to express a wide range of understanding and receive the appropriate amount of credit—with each question. When combined over the entire test, this detailed feedback on each question assures a more reliable and accurate assessment of proficiency.
« page top
Percentile rankings
What are percentile rankings?
Percentiles are used as a form of ranking. A score in the 60th percentile means that score is higher than 60 percent of all scores ever given in that exam. The value of a percentile is determined by the make-up of the population contributing to the test scores. A percentile is a relative measure determined by its population. For example, being in the 66th percentile of professional baseball players is not considered a negative comparison. Although 34 percent of professionals play better, someone in the 66th percentile of professional ball players is still a better ball player than 99.999 percent of the general population.
TeckChek’s percentile pools are populated entirely with scores from highly skilled professionals who make their living in the tested technology. A percentile of 60 indicates greater proficiency than 60 percent of the professionals who have taken the assessment.
What insights do percentiles provide?
TeckChek percentiles provide a window of insight into the marketplace of IT professionals, and how the individual tested ranks in that marketplace. TeckChek provides more than an overall ranking, as each of the five scores provided by TeckChek also include a percentile which shows how that score ranks among the scores of all test takers in that specific subject.
In older technologies, where the average test taker has more than five years of experience, scores are high and a generally high score may result in an average percentile. That same score in an emerging technology (one in which professionals have not yet had time and opportunity to develop expertise) may result in a very high percentile.
« page top
Absolute strengths and weaknesses
How does TeckChek determine strengths and weaknesses?
Each TeckChek assessment breaks the test subject down into 12-18 sub-topics that are unique and specific to that test subject. TeckChek assessments adapt independently within each of these sub-topics (see Independently Adapting Sub-topics above). This means that performance in one sub-topic does not impact the difficulty level of questions in other sub-topics, allowing proficiency in each sub-topic to be independently evaluated.
What is an absolute measure of strength and weakness?
TeckChek’s unique detailed feedback from multiple answer combinations, and the efficiencies of adaptive testing within sub-topics, allow it to measure absolute proficiency in each sub-topic. Most other assessments lack the amount of feedback provided by TeckChek’s unique question methodology. They can only offer relative strengths and weaknesses. In such assessments, results indicate that the test taker is stronger in topic A than topic B. However, this relative measurement of skill does not indicate if the test taker is proficient in either topic! When selecting an assessment, first make sure that it offers an analysis of strengths and weaknesses and then make sure they are absolute and not relative measurements.
How can I benefit from knowing strengths and weaknesses?
TeckChek’s review of absolute strengths and weaknesses is an important tool in both individual and group skill analysis. TeckChek’s analysis helps identify individuals with the specific skills needed on a project. It can also be used in establishing individual training needs. At the department or enterprise level, it identifies skill gaps to help pinpoint skills for new hires and evaluate the skill mix on project teams. Many clients use TeckChek’s Skill Gap Analysis, a report that aggregates results from individuals into a graph showing collective strengths and weaknesses, to help trainers customize course delivery or content to the needs of the group.
By eliminating areas of collective proficiency from the curriculum and concentrating the training on areas of collective weakness, clients are able to reduce classroom time and more effectively focus courses to achieve greater return on training investments.
« page top
Sample question
The question below is not an actual TeckChek question (we have no test for U.S. geography) but is intended to illustrate the multiple correct response methodology.
Which of the following are states in the United States of America?
A. Texas
B. Paris
C. California
D. Jupiter
E. Washington D.C.
Let us examine a few of the 25 answer combinations to this question:
| Selected Answers |
Interpretation |
Follow-up Question |
| Texas and California |
Correct answer. Receives the most credit and hardest follow-up question. |
Very Difficult |
| Texas |
Partially correct. Credit received for selecting Texas but credit is lost for not selecting California. |
Difficult |
| Jupiter |
Incorrect. Test taker is most likely guessing. |
Easy |
| Jupiter, Paris, Washington, D.C. |
Worst possible answer. Maximum credit lost and easiest follow-up question selected. |
Very Easy |
| Texas and Washington, D.C. |
Partially correct. Provides more credit than ‘Texas and Paris’ because, although it is incorrect, selection of Washington D.C. shows more understanding of U.S. Geography then Paris. |
Medium |
« page top
Proctored vs. self-administered assessments
What is the importance of a proctored exam?
Proctored assessments are administered in an environment where the identity of the test taker can be confirmed and where the absence of reference materials and outside assistance can be monitored. The Internet has made it convenient to administer assessments at home or in other unsecured environments. While any test can be given in a proctored environment, score comparisons (including percentiles) must be made independently for those taking the exam in an unproctored or self-administered testing environment.
Does TeckChek offer self administered and proctored assessments?
Yes, TeckChek offers self administered assessments for the convenience of the test taker.
TeckChek is also the only vendor-independent technical proficiency assessment that provides a separate test for proctored (supervised) administration in addition to its self-administered assessments. The percentile scores reported on a TeckChek proctored assessment reflect only those scores from other proctored assessments in the selected subject.
« page top
Access to TeckChek
Who can take a TeckChek assessment?
TeckChek assessments can only be initiated by a corporate sponsor who has purchased a license for access to the exams. The corporate sponsor can test current or potential employees. TeckChek is not available to the general public or to individuals.
Why is it important that the assessment not be available to the general public?
When assessments are available to the general public, it is not clear how often an individual has taken a particular assessment. Some testing organizations provide unlimited public access to their tests. This drastically impacts the validity of the test result. When selecting an assessment service, it is important to know how accessible the assessment has been in the past to the individual being tested and the impact of prior access on the validity of result.
« page top
Test development
Who develops the assessment questions?
Contracted panels of outside subject matter experts write TeckChek assessments. These experts, typically independent consultants, are selected for their extensive practical experience and varied application expertise. We avoid the use of academic experts who might lack in-field, hands-on expertise in the use of a technology.
How are the questions developed?
The expert panels use TeckChek’s proprietary authoring tool to craft questions according to the unique TeckChek methodology. The authoring tool assists the experts in writing questions, formatting assessments and assigning question weights used by TeckChek’s test engine to determine detailed proficiency (see Question Weighting above). The experts independently review each other’s questions for accuracy and to confirm assigned weights. This independent peer review is a fundamental part of TeckChek’s rigorous test development process.
Are question results monitored for exceptional conditions?
Once an assessment has been technically reviewed by the panel of experts, copy edited, and accepted by the TeckChek Test Development Group, the assessment is made available to TeckChek clients in ‘field test’ mode. During this period, percentiles are not provided on the assessment results while the pool of completed tests is being built. At the same time, the results of every question are analyzed to determine if the question is accurate and not misleading or confusing (actual results should approximate expected results for each question based upon the question’s assigned weights). Any question that appears to have abnormal or unexpected results is reviewed by the panel of experts for anomalies that could affect results.
Test results
What kind of information is included in TeckChek results and where can they be accessed?
Unparalleled insight into knowledge—the combination of the efficiencies inherent in adaptive methodology, the detailed responses, and the recognition of partial knowledge provided by the 25 answer combinations all enable TeckChek to collect a large amount of relevant detailed feedback about the test taker's proficiency in a subject.
This large body of detailed feedback is analyzed by the TeckChek scoring engines and reported in the detailed Technical Proficiency Profile, which can be accessed online immediately following the conclusion of the assessment.
« page top
