What is ?

is a secure online application designed to analyze multiple-choice test data. It evaluates the statistical integrity of tests and the academic integrity of students taking tests. uses data files submitted by our clients to provide detailed item-level and test-level statistical information, as well as detailed statistical collusion detection reports. The item and test statistics produced by are designed to help you evaluate and improve the performance of your tests and test items. The collusion detection reports produced by are tools designed to help you detect potentially unscrupulous test-taking behavior (e.g., answer copying).

(Return to Table of Contents)

Why is an online application rather than a software package that I can install on my desktop?

is an online application rather than desktop software for a number of reasons.

Castle Rock Research is constantly improving its products, and an online application allows customers to access these improved features immediately. As well, Castle Rock Research is happy to hear customer suggestions for improving our products; our online application allows us to incorporate these suggestions when appropriate to improve our product for all clients. This is in contrast to desktop software, for which the customer is required to download and install updates, or to periodically purchase an updated version of the software in order to take advantage of improvements or new features.

Large data sets can be processor-intensive (e.g., some collusion detection analyses can take substantial time when large samples are processed on one desktop computer). takes the burden off the customer by processing the data on our systems. Our servers process the data submitted by our customers in a fraction of the time it would take one desktop computer to analyze the same information. As a result, our clients' computers are not bogged down with processor-intensive analyses.

Desktop software requires certain operating systems, memory, and processor speed. These requirements can limit some users who do not have access to up-to-date computer technology. In addition, some desktop software performs poorly on individual computers because certain operating system versions or other software on the computer can affect the performance of the desktop software. All that requires is access to the Internet; it is virtually platform independent.

Once signed up, clients can access from anywhere in the world through an Internet connection. This makes our product especially user-friendly for clients who do not always have access to the same desktop computer.

(Return to Table of Contents)

How much does cost?

License packages for are very reasonably priced and designed to meet the needs of individuals and large organizations alike. Please refer to our Purchase page for pricing options.

(Return to Table of Contents)

How do I submit my test data to ?

Once you purchase a license, you are issued a password that you use along with your e-mail address to log in to . Once logged in, you can submit a data file of the individual examinee responses to your test questions along with a key file containing the correct item responses.

A step-by-step submission process with straightforward instructions guides you through the data-submission process. Once the data is submitted, our servers take over, producing reports and making them available to you in convenient formats, ready for online viewing, downloading, or printing.

(Return to Table of Contents)

Since data I submit is processed on servers rather than on my desktop computer, how can I be certain that the data is secure?

Security and client confidentiality are of the highest priority at Castle Rock Research and our clients can be assured that their data will be secure at all times. Client data is only accessible through an authenticated session, and clients are free at any time to permanently delete their data from our servers.

uses the strongest Internet encryption available. This level of encryption is the same as that used by secure banking and government web sites. As such, it is virtually impossible for someone to break the encryption of the information in transit.

(Return to Table of Contents)

What can I do with all the statistical item and test information that I get from (i.e., how will this information help me in assessing the quality of my test items)?

The item and test information that generates provide clients with tools to evaluate the performance of the tests they administer to examinees. Included with the product is supporting documentation that guides users in interpreting the statistical information in practical terms. The combination of high-quality statistical reports and supporting interpretive resources allows educators who do not have specific training in the area of educational and psychological measurement to effectively evaluate and improve the performance of their tests and items. In other words, is designed not just for assessment experts who wish to evaluate the statistical performance of tests, but for anyone who administers multiple-choice tests in any setting (e.g., high school, certification examinations, post-secondary). The evaluation of test items lead to better assessment of examinees and therefore greater confidence in test results.

(Return to Table of Contents)

What is the role of the collusion detection features of ?

The collusion detection features of allow our clients to investigate cases of examinees who may have engaged in collusion on tests. should be used in conjunction with existing practices (e.g., such as having test invigilators patrol the test writing area and spacing examinees out in a room) which test administrators use on a regular basis. These practices do not assume examinees are cheating, but are put in place to reduce the possibility that collusion will occur. gives test administrators an advanced research-based tool to investigate the possibility of collusion among examinees: a tool that may succeed where others have failed.

(Return to Table of Contents)

The rationale for attempting to identify students who may have engaged in test collusion when taking tests fundamentally lies in the idea that creating a "level playing field" makes testing more fair for all students. If groups of students are cheating on tests and obtaining higher grades than they should, students who do not cheat may be disadvantaged by comparison. Cheating has far-reaching implications for examinees, education centers, businesses, and the public, in that they may be passing/hiring/selecting people who are not properly qualified. Consider, for example, the medical student who is not properly prepared and who by cheating on his/her final certification examination is able to receive her medical degree. The consequences for his/her patients could be significant. This may be an oversimplification of the issue in that one would hope the fate of people's lives does not lie solely in the results of one examination score, but it does illustrate that the misrepresentation of examinees' abilities that results from cheating can have serious consequences.

Some survey research at post-secondary institutions suggests that over 70% of examinees admit to copying answers from other examinees during an exam or quiz at some time during their post-secondary education. The suggested pervasiveness of this behavior indicates a need for alternatives to traditional approaches (e.g., test invigilators) to detect and prevent collusion. For more information regarding test collusion, please refer to the following sources.

(Return to Table of Contents)

flagged a pair of examinees as engaging in collusion. How do I know for certain that they cheated?

The collusion detection features of are based on identifying pairs of examinees whose response patterns to multiple-choice items are considered very unusual and therefore very rare. Note that employs conservative thresholds that pairs of examinees must surpass before being flagged as potentially engaging in collusion. This is done to limit flagging examinees who did not engage in collusion (called false positives).

Although rare, it is always possible that a pair of flagged examinees may not have engaged in collusion. As such, it is important for test administrators to investigate examinees flagged by cautiously - as cautiously as they would investigate a pair of examinees turned in by a fellow examinee or test invigilator. As part of the supporting services that provides to our clients, customers will have access to academic references that outline methods of responding to examinees flagged as potential colluders.

(Return to Table of Contents)

What statistical collusion detection techniques does use?

uses five separate methods of detecting collusion. Each of these was developed by academic researchers prior to the development of , and each has been published and peer-reviewed in academic research literature. All methods used by examine all possible pairs of examinees.

The detailed statistical aspects of the methods are complex; therefore those interested in the specific technical details of the methods are encouraged to explore the academic references cited. It is recommended that clients not familiar with statistical methods related to collusion detection request assistance from knowledgeable colleagues in the area of statistical methods. Brief descriptions of the five statistical collusion detection methods used by are given below.

Angoff's (1974) B-index - This method compares the number of identically incorrect responses for all pairs of examinees who fall into similar intervals of test performance (i.e., intervals are based on multiplying the number of incorrect responses for a pair of examinees). A statistical test is then performed to determine whether the observed value of the number of identically incorrect responses for a pair of examinees is significantly different than the mean value of identically incorrect responses for all examinees at that interval.

A histogram of B-index values for all possible pairs of examinees shows the distribution of values and the rarity of high B-index values (large positive B-index values suggest potential collusion). For example, if a pair of examinees was found to have a B-index value of 9.56 and the mean B-index value for all examinee pairs is 0 with a standard deviation of 1, a score of 9.56 would be considered extremely rare. On the histogram the B-index value of 9.56 would stand out far to the right of the distribution, away from the majority of cases.

For more information, see:

ACT Pair 1 method - This method was developed by the American testing organization ACT (American College Testing; Hanson, Harris, & Brennan, 1987). It uses two indexes calculated for each pair of examinees, the number of identically incorrect responses to test questions (JI1I2), and the length of the longest string of identically responses (STRINGL). If the values for the pair of these statistics are high enough (and therefore rare enough) for any pair of examinees, these examinees will be flagged as potentially engaging in collusion.

Histograms of PAIR1 values for all possible pairs of examinees show the distribution of values and the rarity of high PAIR1 values. Three histograms for the PAIR1 method are produced: a histogram of the JI1I2 values, a histogram of STRINGL values, and a histogram of the product of the JI1I2 values and STRINGL values. The product of the JI1I2 values and STRINGL values is used to determine if a pair of examinees engaged in collusion. High product values (i.e., values that are far to the right of the majority of values displayed in the histogram) indicate that collusion may have occurred.

For more information, see:

ACT Pair 2 method - This method, which was developed by ACT (Hanson, Harris, & Brennan, 1987), uses two statistics calculated for each pair of examinees. The first statistic is based on the number of incorrect responses in the longest string of identically responses (STRINGI1), and the second statistic (PJ) is based on several different measures such as the number of items on the test, the number of identical responses, and the number of identically incorrect responses. If the values for the pair of these statistics are high enough (and therefore rare enough) for any given pair of examinees, these examinees will be flagged as potentially engaging in collusion.

Histograms of PAIR2 values for all possible pairs of examinees show the distribution of values and the rarity of high PAIR2 values. Three histograms for the PAIR2 method are produced: a histogram of the STRINGI1 values, a histogram of PJ values, and a histogram of the product of the STRINGI1 values and PJ values. The product of the STRINGI1 values and PJ values is used to determine if a pair of examinees engaged in collusion. High product values (i.e., values that are far to the right of the majority of values displayed in the histogram) indicate that collusion may have occurred.

For more information, see:

MESA method - The Modified Error Similarity Analysis (MESA) method is one of five statistical methods used by to identify pairs of examinees who may have engaged in collusion. The MESA method is based on the ESA method (Error Similarity Analysis method) developed by Bellezza and Bellezza (1989). It computes the probability that a pair of examinees has a certain number of identically incorrect responses to test items. If the probability is low enough for any pair of examinees, that pair is flagged as potentially engaging in collusion. has modified the traditional ESA method slightly to provide customized estimates of certain variables and to process information more quickly.

A histogram of MESA values for all possible pairs of examinees shows the distribution of values and the rarity of low MESA values (low MESA values suggest potential collusion). For example, if a pair of examinees was found to have a MESA value of 1.23E-12, this would be considered an extremely rare event.

g2 method - This method (developed by Frary, Tideman, & Watts, 1977) is based on the total number of incorrect responses for pairs of examinees. It considers one examinee as the "answer copier" and the other as the "source." The popularity of alternatives and the potential copier's total score are used to estimate - for each item - the probability of selecting the source's answers (correct and incorrect). Two g2 values are produced for each pair of examinees: one tests the hypothesis that the first examinee copied from the second and the second tests the opposite theory. If the g2 values are large enough for any pair of examinees, they will be flagged as potentially engaging in collusion.

A histogram of g2 values for all possible pairs of examinees shows the distribution of values and the rarity of high g2 values (high positive g2 values suggest potential collusion). For example, if a pair of examinees was found to have a g2 value of 8.54 and the mean g2 value for all examinee pairs was 0 with a standard deviation of 1, their score of 8.54 would be considered extremely rare. On the histogram, the g2 value of 8.54 would stand out far to the right of the distribution, away from the majority of cases.

For more information, see:

(Return to Table of Contents)

What is a "key file" and "data file" in the context of ?

A key file is a text file that contains the answer key for a test submitted to . Each answer, given as a number value, is separated by a comma. For example, if the answers for a particular test are: A, C, B, A, D, D, A, C, B, A, then the answer key file will contain the corresponding values: 1,3,2,1,4,4,1,3,2,1.

A data file is an electronic file that contains the answers of examinees for a particular test. Typically, a data file is produced by optical scanners (e.g., Optical Mark Recognition OMR, Optical Character Recognition OCR, Scantron machines, etc.) that process "bubble sheets" onto which examinees have recorded their answers. Most scanner operators have no problem accommodating the data file needs of .

requires a simple text-based data format (i.e., comma-separated ASCII). Each examinee should be entered on a separate row, as follows:

Examinee identification in the first field, followed by a comma.

Writing center code in the second position, followed by a comma.

Group code in the third position, followed by a comma.

Examinee's answers in the fourth and subsequent positions, each separated by a comma.

This code must be numerical, and only values of 1 to 9 can be used. requires that item responses start at a value of "1" and continue up to "9" (A= 1,B=2,…H=8, I=9). considers a value of "0" to be no response; as such, users must not use 0 values as item responses (e.g., for a true/false item, the values "1" and "2" must be used rather than "0" and "1"). Completion of this field is mandatory.

Below is an example of the content of a data file, with optional writing center and group code information, for three students who each took a ten-item test:

An example of how a client submits assessments to .

Professor Jones teaches three classes of introductory psychology at a local college and administers multiple-choice tests regularly to his students. He is interested in evaluating and improving the quality of his assessments, so he purchased a single-user license for .

A few weeks before administering a midterm exam to his students, he spoke with the scanning (bubble-sheet processing) department at his college. The professor wanted to administer the same 80 item multiple-choice midterm to all three of his classes. Professor Jones showed the scanning technicians the data file requirements for and was assured that they would produce a data file for him when he submitted his bubble sheets for scanning. Professor Jones was interested in comparing how each of his classes would perform on the midterm, so he asked the scanner operator to set up an area on the bubble sheets where students would mark what class they were in. He was also interested in how male and female students performed on his midterm, so he asked the scanner operator to set up an area on the bubble sheet for students to fill in their gender as well.

On the day of the midterm, the professor asked his students to fill in the two areas on the bubble sheet to indicate their class and their gender. The students then took the test as they normally would. When all three of the professor's classes had taken the midterm, he took the bubble sheets to the scanning department for processing. The scanner operators scanned all the bubble sheets and produced a data file that corresponded with the example data file on the web site. The scanner operator sent a data file called "psych101_test.dat" to the professor via the college's secure e-mail system. The professor then created an answer key file called "psych101_key.dat" and logged into the web site.

Once logged into the secure web site, the professor clicked on the "Add a job" button in order to submit the midterm exam results for his classes. He gave the job the name "Jones psych midterm" and attached the data and the answer key file. He then followed the step-by-step wizard, answering a few simple questions about his test (e.g., how many questions are there), and finished submitting his assessment. In under a minute, the reports for his test were completed and he began looking over the results. As a first-time user, Professor Jones felt that he needed some support in interpreting the reports, so he went into the "Knowledge base" section of and found a number of useful guides that walked him through how to interpret his results. Professor Jones found a few items that were not performing as well as he would have liked, so he revised those items so that he could administer the improved version of the test next semester.

(Return to Table of Contents)

Sometimes I reorder the items in my tests and administer several forms of the exam to my examinees. How does know which form is which?

At this time cannot differentiate the same items administered on different tests. Our product would treat each test form submitted to our system as a separate test. If our clients wish to combine test data, they would need to reorganize the data files before they submit them to us. For example, clients could reorganize the items in one data file to match the order of items in another data file. They could then combine the two data files into one data file and submit it to as one test.

(Return to Table of Contents)

Can I get statistical breakdowns of the performance of different groups (e.g., males versus females) on my tests using ?

Yes. Our product has the ability to define groups in the data submitted (e.g., male and female) so that a statistical breakdown of group performance is possible. The reports generated by will contain detailed information regarding the performance of each group on the test.

(Return to Table of Contents)

Can I get statistical breakdowns of the performance of different writing centers (e.g., three introductory psychology classes) on my tests using ?

Yes. We have built into our product the ability to define writing centers (e.g., classes, schools, etc.) in the data submitted so that a statistical breakdown of writing center performance is possible. The reports generated by will contain detailed information regarding the performance of each writing center on the test.

(Return to Table of Contents)

Can I download a file with the statistical data produced by for my test?

Yes. provides the option of downloading data files that contain all the statistical information generated in the reports. This allows our clients to archive or bank all statistical information regarding their tests.

(Return to Table of Contents)

Can I download a PDF document that contains all the reports generated by ?

Yes. provides the option of downloading a PDF document with all the reports generated for a test. This allows our clients to archive all reports generated by for their tests.

(Return to Table of Contents)

Yes, we have made every effort to make the interpretation of the item and test information as visually pleasing and easy to interpret as possible. We have included graphical information whenever possible to accommodate our clients who would like a visual representation of the information.

(Return to Table of Contents)

Does produce Item Response Theory (IRT) statistics as well as Classical Test Theory (CTT) statistics?

At this stage, does not produce IRT statistics because these statistics require large sample sizes. In future versions of , statistics such as IRT and factor analysis/principal component analysis may become available.

(Return to Table of Contents)

How are new features added to ?

We at Castle Rock Research are always looking for ways to improve the quality of our products and are happy to respond to the needs of our clients. Castle Rock Research is continuously adding features to in order to make it more useful to our clients. If there are features that you wish to add to , or if you have suggestions on how to improve and you are a registered client, you can send suggestions through the support page.

(Return to Table of Contents)

While I was logged into and tried to access reports for one of my jobs, I received an error message that told me that another user has replaced my session. What does this mean?

When you log into you establish a session. If another user at a different computer uses your e-mail and password to login, they will replace your session, forcing you off of the system. This is because only allows a single session per user. In order to avoid this situation, it is important not to reveal your login information to others. If you believe someone has obtained your login information, you should change your password by logging into , going into the "User profile" section, and following the instructions for changing your password. Another way to avoid this situation is to ensure that your institution has purchased enough licenses to accommodate all users.

(Return to Table of Contents)

I had logged into and did not use the application for 30 minutes. When I tried to access my reports, gave me a message stating that my session had timed out. What does this mean?

If a user logs into and does not use the system for 30 minutes, he or she will be asked to re-login to the application. It is important that if you are away from your computer, you log out of the application. Logging out of the application prevents others from accessing your information while you are away from your computer. To log out after you are logged in to the application, click on the link in the top right corner of the application labeled "Log out." If you forget to log out, will do it for you after 30 minutes.

(Return to Table of Contents)

How long does retain my test data and reports?

The reports produced by can be stored indefinitely, at your discretion. They can be accessed as long as you have a valid license for . Downloadable information produced by (e.g., PDF, data files) are stored for 30 days on our secure servers after which time they are subject to deletion. If you need to retain this information you must download and save necessary files to your local system.

(Return to Table of Contents)

I'm having problems downloading the PDF (or Data File) created by as part of my job. What is the cause and how do I solve this?

When attempting to download the Report PDF, Item PDF, or Data File, you may encounter an error message on Internet Explorer. On some browsers, you may not receive a message at all. If you have a client firewall installed, it may be preventing you from downloading these files. This is not an error, but a consequence of the way these files are sent to you. Typically, files for download are located on a file server and its hyperlink is placed on a web page. This type of download does not cause problems for the client firewall, but it does pose a security risk since the files are publicly available on the internet. We have taken steps so that only you have access to your files. When a link to download a file is clicked (either the Report PDF, Item PDF, or Data File), we first authenticate the user to ensure that the user is the owner of the document, and then stream the document using 128 bit encryption over port 443. By default, your firewall may not be configured to allow this type of download.

In order to download your files, you must configure your client firewall to add "integrity.castlerockresearch.com" as one of your trusted sites. If you are on Windows and have Symantec Client Firewall installed, you may configure your firewall to trust Integrity by following the instructions below. Otherwise, please refer to your firewall documentation or contact us for support.

    Firewall Configuration Instructions (Symantec)

(Return to Table of Contents)