Pilot Test Outcomes

Chinese (Mandarin)

Multiple-Choice Items (MCQs) and Short-Answer (SAs) items

Open-Response Items (ORIs)

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

Dance

Multiple-Choice Items (MCQs)

NOTE: As described in the "Pilot Testing" section of the Technical Manual, in addition to the review of test items by expert judges (members of the BRC and CACs), when permitted by sufficient candidate populations, empirical data were collected about statistical and qualitative characteristics of the new items through pilot testing. For this field there exists an insufficient candidate population for collecting statistical information through pilot testing. Therefore, pilot test information for multiple-choice questions is unavailable for the field.

Open-Response Items (ORIs)

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

English as a Second Language

Multiple-Choice Items (MCQs)

Open-Response Items (ORIs)

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

German

Multiple-Choice Items (MCQs)

NOTE: As described in the "Pilot Testing" section of the Technical Manual, in addition to the review of test items by expert judges (members of the BRC and CACs), when permitted by sufficient candidate populations, empirical data were collected about statistical and qualitative characteristics of the new items through pilot testing. For this field there exists an insufficient candidate population for collecting statistical information through pilot testing. Therefore, pilot test information for multiple-choice questions is unavailable for the field.

Open-Response Items (ORIs)

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

Italian

Multiple-Choice Items (MCQs)

NOTE: As described in the "Pilot Testing" section of the Technical Manual, in addition to the review of test items by expert judges (members of the BRC and CACs), when permitted by sufficient candidate populations, empirical data were collected about statistical and qualitative characteristics of the new items through pilot testing. For this field there exists an insufficient candidate population for collecting statistical information through pilot testing. Therefore, pilot test information for multiple-choice questions is unavailable for the field.

Open-Response Items (ORIs)

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

Music

Multiple-Choice Items (MCQs)

Open-Response Items (ORIs)

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

Portuguese

Multiple-Choice Items (MCQs)

NOTE: As described in the "Pilot Testing" section of the Technical Manual, in addition to the review of test items by expert judges (members of the BRC and CACs), when permitted by sufficient candidate populations, empirical data were collected about statistical and qualitative characteristics of the new items through pilot testing. For this field there exists an insufficient candidate population for collecting statistical information through pilot testing. Therefore, pilot test information for multiple-choice questions is unavailable for the field.

Open-Response Items (ORIs)

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

Russian

Multiple-Choice Items (MCQs)

NOTE: As described in the "Pilot Testing" section of the Technical Manual, in addition to the review of test items by expert judges (members of the BRC and CACs), when permitted by sufficient candidate populations, empirical data were collected about statistical and qualitative characteristics of the new items through pilot testing. For this field there exists an insufficient candidate population for collecting statistical information through pilot testing. Therefore, pilot test information for multiple-choice questions is unavailable for the field.

Open-Response Items (ORIs)

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

Sheltered English Immersion

Multiple-Choice Items (MCQs)

*This chart includes information regarding multiple-choice items piloted prior to the first operational administration and may be a subset of all items developed. Additional items are piloted as non-scorable on operational test forms, until all items are piloted.

Open-Response Items

In consultation with the Massachusetts Department of Education and the Department of Justice review, the open-response item for the Sheltered English Immersion test was designed as a single, five-part open-response item requiring candidates to show their knowledge of sheltering content. Candidates use one of 10 mentor texts for the basis of their response. Each mentor text is an informational text that is representative of the kind of content and academic language features a student might encounter in texts in a given content area. Candidates are expected to spend about 20–30 minutes to prepare an adequate response to each of the five parts of the assignment, for a total of about 2–2 ½ hours for the section. The multiple-choice section counts for 60 percent and the open-response section counts for 40 percent of the candidate's total test score.

Piloting of the new open-response item type focused on evaluating the item format. Pilot test responses to the open-response item format were reviewed for the following characteristics:

• Candidate ability to complete the assignment
• Testing time required to complete the assignment
• Administration process and computer screen layout
• Application of the proposed score scale to the responses

A review of the data and item comments collected indicated that the candidates were able to appropriately and adequately respond to the new open-response item type and were able to produce a response within the targeted response time of 2 ½ hours, leaving sufficient time to respond to the multiple-choice questions. In addition, candidates were able to move throughout the various open-response item components with relative ease.

Theater

Multiple-Choice Items (MCQs)

Open-Response Items (ORIs)

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.