Frequently asked questions
Table of contents
Questions & answers
-
What is
?
is a secure online application designed to
analyze multiple-choice test data. It evaluates the statistical integrity of tests and
the academic integrity of students taking tests.
uses data files submitted by our clients to
provide detailed item-level and test-level statistical information, as well as detailed
statistical collusion detection reports. The item and test statistics produced
by are designed to help you evaluate
and improve the performance of your tests and test items. The collusion detection
reports produced by are tools
designed to help you detect potentially unscrupulous
test-taking behavior (e.g., answer copying).
(Return to Table of Contents)
-
Why is
an online application rather
than a software package that I can install on my desktop?
is an online application rather than
desktop software for a number of reasons.
Castle Rock Research is constantly improving its products, and an online application
allows customers to access these improved features immediately. As well, Castle Rock
Research is happy to hear customer suggestions for improving our products; our online
application allows us to incorporate these suggestions when appropriate to improve our
product for all clients. This is in contrast to desktop software, for which the
customer is required to download and install updates, or to periodically purchase an
updated version of the software in order to take advantage of improvements or new features.
Large data sets can be processor-intensive (e.g., some collusion detection analyses
can take substantial time when large samples are processed on one desktop
computer). takes the burden off
the customer by processing the data on our systems. Our servers process the data
submitted by our customers in a fraction of the time it would take one desktop
computer to analyze the same information. As a result, our clients' computers are not
bogged down with processor-intensive analyses.
Desktop software requires certain operating systems, memory, and processor speed.
These requirements can limit some users who do not have access to up-to-date computer
technology. In addition, some desktop software performs poorly on individual computers
because certain operating system versions or other software on the computer can affect
the performance of the desktop software. All
that requires is access to the Internet;
it is virtually platform independent.
Once signed up, clients can access from
anywhere in the world through an Internet connection. This makes our product especially
user-friendly for clients who do not always have access to the same desktop computer.
(Return to Table of Contents)
-
How much does
cost?
License packages for are very reasonably
priced and designed to meet the needs of individuals and large organizations alike.
Please refer to our Purchase page for pricing options.
(Return to Table of Contents)
-
How do I submit my test data to
?
Once you purchase a license, you
are issued a password that you use along with your e-mail address to log in to
. Once logged in, you can submit a data
file of the individual examinee responses to your test questions along with a key file
containing the correct item responses.
A step-by-step submission process with straightforward instructions guides you
through the data-submission process. Once the data is submitted, our servers take over,
producing reports and making them available to you in convenient formats,
ready for online viewing, downloading, or printing.
(Return to Table of Contents)
-
Since data I submit is processed on
servers rather than on my desktop computer, how can I be certain that the data is secure?
Security and client confidentiality are of the highest priority at Castle Rock Research
and our clients can be assured that their data will be secure at all times. Client data
is only accessible through an authenticated session, and clients are free at any time
to permanently delete their data from our servers.
uses the strongest Internet
encryption available. This level of encryption is the same as that used by secure
banking and government web sites. As such, it is virtually impossible for someone to
break the encryption of the information in transit.
(Return to Table of Contents)
-
What can I do with all the statistical item and test information that I get
from
(i.e., how will this information
help me in assessing the quality of my test items)?
The item and test information that
generates provide clients with tools to evaluate the performance of the tests they
administer to examinees. Included with the
product is supporting documentation that guides users in interpreting the statistical
information in practical terms. The combination of high-quality statistical reports
and supporting interpretive resources allows educators who do not have specific training
in the area of educational and psychological measurement to effectively evaluate
and improve the performance of their tests and items. In other words,
is designed not just for assessment
experts who wish to evaluate the statistical performance of tests, but for anyone who
administers multiple-choice tests in any setting (e.g., high school, certification
examinations, post-secondary). The evaluation of test items lead to better assessment
of examinees and therefore greater confidence in test results.
(Return to Table of Contents)
-
What is the role of the collusion detection features
of
?
The collusion detection features of allow
our clients to investigate cases of examinees who may have engaged in collusion on
tests. should be used in conjunction
with existing practices (e.g., such as having test invigilators patrol the test
writing area and spacing examinees out in a room) which test administrators use
on a regular basis. These practices do not assume examinees are cheating, but are
put in place to reduce the possibility that collusion will
occur. gives test administrators an advanced
research-based tool to investigate the possibility of collusion among examinees: a tool
that may succeed where others have failed.
(Return to Table of Contents)
-
What is the rationale for collusion detection, and how prevalent is cheating?
The rationale for attempting to identify students who may have engaged in test collusion
when taking tests fundamentally lies in the idea that creating a "level playing field"
makes testing more fair for all students. If groups of students are cheating on tests and
obtaining higher grades than they should, students who do not cheat may be disadvantaged
by comparison. Cheating has far-reaching implications for examinees, education centers,
businesses, and the public, in that they may be passing/hiring/selecting people who are
not properly qualified. Consider, for example, the medical student who is not properly
prepared and who by cheating on his/her final certification examination is able to receive
her medical degree. The consequences for his/her patients could be significant. This may
be an oversimplification of the issue in that one would hope the fate of people's lives
does not lie solely in the results of one examination score, but it does illustrate that
the misrepresentation of examinees' abilities that results from cheating can have serious
consequences.
Some survey research at post-secondary institutions suggests that over 70% of examinees
admit to copying answers from other examinees during an exam or quiz at some time during
their post-secondary education. The suggested pervasiveness of this behavior indicates
a need for alternatives to traditional approaches (e.g., test invigilators) to detect
and prevent collusion. For more information regarding test collusion, please refer to the
following sources.
Cizek, G. J. (1999). Cheating on Tests, How to Do It, Detect It, and Prevent It. Lawrence
Erlbaum Associates: Mahwah, New Jersey.
Cizek, G. J. (2003). Detecting and Preventing Classroom Cheating: Promoting Integrity
in Assessment. Corwin Press: Thousand Oaks, California.
(Return to Table of Contents)
-
flagged a pair of examinees as engaging
in collusion. How do I know for certain that they cheated?
The collusion detection features of
are based on identifying pairs of examinees whose response patterns to multiple-choice
items are considered very unusual and therefore very rare. Note
that employs conservative thresholds
that pairs of examinees must surpass before being flagged as potentially engaging in
collusion. This is done to limit flagging examinees who did not engage in collusion
(called false positives).
Although rare, it is always possible that a pair of flagged examinees may not have engaged
in collusion. As such, it is important for test administrators to investigate examinees
flagged by cautiously - as cautiously
as they would investigate a pair of examinees turned in by a fellow examinee or test
invigilator. As part of the supporting services
that provides to our clients, customers
will have access to academic references that outline methods of responding to examinees
flagged as potential colluders.
(Return to Table of Contents)
-
What statistical collusion detection techniques
does
use?
uses five separate methods of detecting
collusion. Each of these was developed by academic researchers prior to the development
of , and each has been published and
peer-reviewed in academic research literature. All methods used
by examine all possible pairs of examinees.
The detailed statistical aspects of the methods are complex; therefore those interested
in the specific technical details of the methods are encouraged to explore the academic
references cited. It is recommended that clients not familiar with statistical
methods related to collusion detection request assistance from knowledgeable colleagues
in the area of statistical methods. Brief descriptions of the five statistical
collusion detection methods used by
are given below.
Angoff's (1974) B-index - This method compares the number of identically incorrect
responses for all pairs of examinees who fall into similar intervals of test
performance (i.e., intervals are based on multiplying the number of incorrect
responses for a pair of examinees). A statistical test is then performed to
determine whether the observed value of the number of identically incorrect
responses for a pair of examinees is significantly different than the mean value
of identically incorrect responses for all examinees at that interval.
A histogram of B-index values for all possible pairs of examinees shows the
distribution of values and the rarity of high B-index values (large positive
B-index values suggest potential collusion). For example, if a pair of
examinees was found to have a B-index value of 9.56 and the mean B-index value for
all examinee pairs is 0 with a standard deviation of 1, a score of 9.56 would be
considered extremely rare. On the histogram the B-index value of 9.56 would
stand out far to the right of the distribution, away from the majority of cases.
For more information, see:
Angoff, W. (1974). The development of statistical indices for detecting cheaters.
Journal of the American Statistical Association, 69(345), 44-49.
ACT Pair 1 method - This method was developed by the American testing
organization ACT (American College Testing; Hanson, Harris, & Brennan, 1987).
It uses two indexes calculated for each pair of examinees, the number of
identically incorrect responses to test questions (JI1I2), and the length
of the longest string of identically responses (STRINGL). If the values for
the pair of these statistics are high enough (and therefore rare enough) for
any pair of examinees, these examinees will be flagged as potentially engaging
in collusion.
Histograms of PAIR1 values for all possible pairs of examinees show the distribution
of values and the rarity of high PAIR1 values. Three histograms for the PAIR1
method are produced: a histogram of the JI1I2 values, a histogram of STRINGL values,
and a histogram of the product of the JI1I2 values and STRINGL values. The
product of the JI1I2 values and STRINGL values is used to determine if a pair
of examinees engaged in collusion. High product values (i.e., values that are
far to the right of the majority of values displayed in the histogram) indicate
that collusion may have occurred.
For more information, see:
Hanson, B. A., Harris, D. J., & Brennan, R. L. (1987). A comparison of several
statistical methods for examining allegations of copying (ACT Research Report
Series No. 87-15). Iowa City, IA: American College Testing.
ACT Pair 2 method - This method, which was developed by ACT (Hanson, Harris, &
Brennan, 1987), uses two statistics calculated for each pair of examinees.
The first statistic is based on the number of incorrect responses in the longest
string of identically responses (STRINGI1), and the second statistic (PJ)
is based on several different measures such as the number of items on the test,
the number of identical responses, and the number of identically incorrect
responses. If the values for the pair of these statistics are high enough (and
therefore rare enough) for any given pair of examinees, these examinees will be
flagged as potentially engaging in collusion.
Histograms of PAIR2 values for all possible pairs of examinees show the
distribution of values and the rarity of high PAIR2 values. Three histograms
for the PAIR2 method are produced: a histogram of the STRINGI1 values, a histogram
of PJ values, and a histogram of the product of the STRINGI1 values and PJ values.
The product of the STRINGI1 values and PJ values is used to determine if a pair
of examinees engaged in collusion. High product values (i.e., values that are
far to the right of the majority of values displayed in the histogram) indicate
that collusion may have occurred.
For more information, see:
Hanson, B. A., Harris, D. J., & Brennan, R. L. (1987). A comparison of several
statistical methods for examining allegations of copying (ACT Research Report
Series No. 87-15). Iowa City, IA: American College Testing.
MESA method - The Modified Error Similarity Analysis (MESA) method is one of
five statistical methods used by to identify pairs of examinees who may have
engaged in collusion. The MESA method is based on the ESA method (Error
Similarity Analysis method) developed by Bellezza and Bellezza (1989). It
computes the probability that a pair of examinees has a certain number of
identically incorrect responses to test items. If the probability is low
enough for any pair of examinees, that pair is flagged as potentially
engaging in collusion. has modified
the traditional ESA method slightly to provide customized estimates of certain
variables and to process information more quickly.
A histogram of MESA values for all possible pairs of examinees shows the
distribution of values and the rarity of low MESA values (low MESA
values suggest potential collusion). For example, if a pair of examinees
was found to have a MESA value of 1.23E-12, this would be considered an
extremely rare event.
Bellezza F., & Bellezza, S. (1991). Detection of cheating on multiple-choice
tests by using error-similarity analysis. Teaching of Psychology, 16(3), 151-155.
g2 method - This method (developed by Frary, Tideman, & Watts, 1977) is based
on the total number of incorrect responses for pairs of examinees. It
considers one examinee as the "answer copier" and the other as the "source."
The popularity of alternatives and the potential copier's total score are
used to estimate - for each item - the probability of selecting the source's
answers (correct and incorrect). Two g2 values are produced for each
pair of examinees: one tests the hypothesis that the first examinee
copied from the second and the second tests the opposite theory. If
the g2 values are large enough for any pair of examinees, they will be
flagged as potentially engaging in collusion.
A histogram of g2 values for all possible pairs of examinees shows
the distribution of values and the rarity of high g2 values (high positive
g2 values suggest potential collusion). For example, if a pair of examinees
was found to have a g2 value of 8.54 and the mean g2 value for all examinee
pairs was 0 with a standard deviation of 1, their score of 8.54 would be
considered extremely rare. On the histogram, the g2 value of 8.54 would
stand out far to the right of the distribution, away from the majority of cases.
For more information, see:
Frary, R. B., Tideman, T. N., & Watts, T. M. (1977). Indices of cheating
on multiple-choice tests. Journal of Educational Statistics, 2, 235-256.
(Return to Table of Contents)
-
What is a "key file" and "data file" in the context
of
?
A key file is a text file that contains the answer key for a test
submitted to . Each answer, given as a
number value, is separated by a comma. For example, if the answers for a particular
test are: A, C, B, A, D, D, A, C, B, A, then the answer key file will contain the
corresponding values: 1,3,2,1,4,4,1,3,2,1.
A data file is an electronic file that contains the answers of examinees for a
particular test. Typically, a data file is produced by optical scanners (e.g.,
Optical Mark Recognition OMR, Optical Character Recognition OCR, Scantron
machines, etc.) that process "bubble sheets" onto which examinees have recorded
their answers. Most scanner operators have no problem accommodating the data
file needs of .
requires a simple text-based data
format (i.e., comma-separated ASCII). Each examinee should be entered on a
separate row, as follows:
Examinee identification in the first field, followed by a comma.
Examinee identification can be a number, a name, or a combination of characters.
Identifiers are limited to 128 characters and cannot include commas (e.g., do not
use "Jones, Jane" as an examinee identifier, instead use "Jane Jones" or "Jones
Jane"). Completion of this field is mandatory.
Writing center code in the second position, followed by a comma.
This code can be numerical, textual, or a combination (e.g., "Morning session
101A"). Completion of this field is optional.
Group code in the third position, followed by a comma.
This code can be numerical, textual, or a combination (e.g., "Males," "Females,"
"Mature students," "Out of country students"). Completion of this field is optional.
Examinee's answers in the fourth and subsequent positions, each separated by a comma.
This code must be numerical, and only values of 1 to 9 can be used.
 requires that item responses start at a
value of "1" and continue up to "9" (A= 1,B=2,…H=8, I=9).
 considers a value of "0" to be no
response; as such, users must not use 0 values as item responses (e.g., for a
true/false item, the values "1" and "2" must be used rather than "0" and "1").
Completion of this field is mandatory.
Below is an example of the content of a data file, with optional writing center and
group code information, for three students who each took a ten-item test:
Examinee ID 1,Writing center 1,Group A,2,1,3,1,1,2,2,4,1,2
Examinee ID 2,Writing center 1,Group A,1,3,2,4,2,4,3,1,2,4
Examinee ID 3,Writing center 3,Group A,3,4,4,2,3,1,2,3,2,4
Below is an example of the content of a data file, with optional writing center code
information, for three students who each took a ten-item test:
Examinee ID 1,Writing center 1,3,4,3,1,1,2,2,4,1,1
Examinee ID 2,Writing center 2,2,3,2,4,2,4,3,1,2,2
Examinee ID 3,Writing center 1,3,4,4,2,3,1,2,3,2,2
Below is an example of the content of a data file, with NO optional writing center or
group code information, for three students who each took a ten-item test:
Examinee ID 1,4,4,3,1,3,2,2,4,1,4
Examinee ID 2,4,3,2,4,1,4,3,1,2,4
Examinee ID 3,4,4,4,2,2,1,2,3,2,4
An example of how a client submits
assessments to .
Professor Jones teaches three classes of introductory psychology at a local college
and administers multiple-choice tests regularly to his students. He is interested in
evaluating and improving the quality of his assessments, so he purchased a single-user
license for .
A few weeks before administering a midterm exam to his students, he spoke with the
scanning (bubble-sheet processing) department at his college. The professor wanted
to administer the same 80 item multiple-choice midterm to all three of his classes.
Professor Jones showed the scanning technicians the data file requirements
for and was assured that they
would produce a data file for him when he submitted his bubble sheets for
scanning. Professor Jones was interested in comparing how each of his classes
would perform on the midterm, so he asked the scanner operator to set up an area
on the bubble sheets where students would mark what class they were in. He was
also interested in how male and female students performed on his midterm, so he
asked the scanner operator to set up an area on the bubble sheet for students to
fill in their gender as well.
On the day of the midterm, the professor asked his students to fill in the two areas
on the bubble sheet to indicate their class and their gender. The students then took
the test as they normally would. When all three of the professor's classes had
taken the midterm, he took the bubble sheets to the scanning department for processing.
The scanner operators scanned all the bubble sheets and produced a data file
that corresponded with the example data file on
the web site. The scanner operator
sent a data file called "psych101_test.dat" to the professor via the college's secure
e-mail system. The professor then created an answer key file called "psych101_key.dat"
and logged into the web site.
Once logged into the secure web
site, the professor clicked on the "Add a job" button in order to submit the midterm
exam results for his classes. He gave the job the name "Jones psych midterm" and
attached the data and the answer key file. He then followed the step-by-step wizard,
answering a few simple questions about his test (e.g., how many questions are there),
and finished submitting his assessment. In under a minute, the reports for his test
were completed and he began looking over the results. As a first-time user, Professor
Jones felt that he needed some support in interpreting the reports, so he went into
the "Knowledge base" section of and
found a number of useful guides that walked
him through how to interpret his results. Professor Jones found a few items that were
not performing as well as he would have liked, so he revised those items so that
he could administer the improved version of the test next semester.
(Return to Table of Contents)
-
Sometimes I reorder the items in my tests and administer several forms of the
exam to my examinees. How does
know
which form is which?
At this time cannot differentiate
the same items administered on different tests. Our product would treat each test
form submitted to our system as a separate test. If our clients wish to combine
test data, they would need to reorganize the data files before they submit them to
us. For example, clients could reorganize the items in one data file to match
the order of items in another data file. They could then combine the two data
files into one data file and submit it
to as one test.
(Return to Table of Contents)
-
Can I get statistical breakdowns of the performance of different groups
(e.g., males versus females) on my tests
using
?
Yes. Our product has the ability to define groups in the data submitted
(e.g., male and female) so that a statistical breakdown of group performance
is possible. The reports generated
by will contain detailed information
regarding the performance of each group on the test.
(Return to Table of Contents)
-
Can I get statistical breakdowns of the performance of different writing
centers (e.g., three introductory psychology classes) on my
tests using
?
Yes. We have built into our product the ability to define writing
centers (e.g., classes, schools, etc.) in the data submitted so that a
statistical breakdown of writing center performance is possible. The reports
generated by will contain
detailed information regarding the performance of each writing center on the test.
(Return to Table of Contents)
-
Can I download a file with the statistical data produced
by
for my test?
Yes. provides the option of
downloading data files that contain all the statistical information
generated in the reports.
This allows our clients to archive or bank all statistical information
regarding their tests.
(Return to Table of Contents)
-
Can I download a PDF document that contains all the reports generated
by
?
Yes. provides the option of downloading
a PDF document with all the reports generated for a test. This allows our clients
to archive all reports generated
by for their tests.
(Return to Table of Contents)
-
Do the statistical reports that I receive include graphical representations
of the information as well as numerical information?
Yes, we have made every effort to make the interpretation of the item and test
information as visually pleasing and easy to interpret as possible. We have
included graphical information whenever possible to accommodate our clients
who would like a visual representation of the information.
(Return to Table of Contents)
-
Does
produce Item Response
Theory (IRT) statistics as well as Classical Test Theory (CTT) statistics?
At this stage, does not
produce IRT statistics because these
statistics require large sample sizes. In future versions of
, statistics such as IRT
and factor analysis/principal
component analysis may become available.
(Return to Table of Contents)
-
How are new features added to
?
We at Castle Rock Research are always looking for ways to improve the
quality of our products and are happy to respond to the needs of our
clients. Castle Rock Research is continuously adding features to
in order to make it
more useful to our clients. If there
are features that you wish to add
to , or if you have
suggestions on how to improve
and you are a registered client,
you can send suggestions through
the support page.
(Return to Table of Contents)
-
While I was logged
into
and tried to access reports
for one of my jobs, I received an error message that told me that another user
has replaced my session. What does this mean?
When you log into you establish a
session. If another user at a different computer uses your e-mail and password to
login, they will replace your session, forcing you off of the system. This is
because only allows a single
session per user. In order to avoid this situation, it is important not to reveal
your login information to others. If you believe someone has obtained your
login information, you should change your password by logging
into , going into the "User profile"
section, and following the instructions for changing your password. Another way
to avoid this situation is to ensure that your institution has purchased
enough licenses to accommodate all users.
(Return to Table of Contents)
-
I had logged into
and did not
use the application for 30 minutes. When I tried to access my
reports, gave me a message stating
that my session had timed out. What does this mean?
If a user logs
into and does not use the system
for 30 minutes, he or she will be asked to re-login to the application. It is
important that if you are away from your computer, you log out of the application.
Logging out of the application prevents others from accessing your information
while you are away from your computer. To log out after you are logged in to
the application, click on the link in the top right corner of the
application labeled "Log out." If you forget to log
out, will do it for you after 30 minutes.
(Return to Table of Contents)
-
How long does
retain my test
data and reports?
The reports produced by can
be stored indefinitely, at your discretion. They can be accessed as long as
you have a valid license for .
Downloadable information produced
by (e.g., PDF, data files) are stored
for 30 days on our secure servers after which time they are subject to deletion.
If you need to retain this information you must download and save necessary
files to your local system.
(Return to Table of Contents)
-
I'm having problems downloading the PDF (or Data File) created by
as part of my job. What is the cause and how do I solve this?
When attempting to download the Report PDF, Item PDF, or Data File, you may encounter
an error message on Internet Explorer. On some browsers, you may not receive a message at all.
If you have a client firewall installed, it may be preventing you from downloading these files.
This is not an error, but a consequence of the way these files are sent to you. Typically, files
for download are located on a file server and its hyperlink is placed on a web page. This type
of download does not cause problems for the client firewall, but it does pose a security risk
since the files are publicly available on the internet. We have taken steps so that only
you have access to your files. When a link to download a file is clicked (either the Report PDF, Item PDF,
or Data File), we first authenticate the user to ensure that the user is the owner of the document,
and then stream the document using 128 bit encryption over port 443.
By default, your firewall may not be configured to allow this type of download.
In order to download your files, you must configure your client firewall to add
"integrity.castlerockresearch.com" as one of your trusted sites. If you are on Windows and have
Symantec Client Firewall installed, you may configure your firewall to trust Integrity by following
the instructions below. Otherwise, please refer to your firewall documentation or
contact us for support.
Firewall Configuration Instructions (Symantec)
(Return to Table of Contents)
|