Test Development Activities

Timeline of Test Development Activities

The Massachusetts Tests for Educator Licensure® (MTEL®) program has been developed as one of the requirements for candidates seeking educator licensure. Candidates for educator licensure in Massachusetts must take and pass a communication and literacy skills test and appropriate subject matter test(s) as part of the requirements to obtaining an educator license. By passing the MTEL, candidates demonstrate that they have the subject matter knowledge and skills to teach in Massachusetts public schools.

Request for Test Development by the Department of Elementary and Secondary Education

The Department of Elementary and Secondary Education (the Department) begins the test development process by determining which test fields are to be newly developed or updated. Once a request for test development has been received, Evaluation Systems works with the Department to determine the subject matter for the new tests, referencing the state regulations, the Massachusetts curriculum frameworks, national benchmarks, and other relevant documents and materials related to the field.

Identify Massachusetts Educators for Review Committees

Test development for the MTEL is accomplished through the close collaboration of the Department, Evaluation Systems, and Massachusetts public school educators and college faculty. The contribution of Massachusetts educators is essential to the success and integrity of the MTEL program. Educators participate at every stage of test development by serving on review committees, responding to surveys, assisting in pilot test activities, and providing input used by the Massachusetts Commissioner of Elementary and Secondary Education in setting the qualifying (passing) score for the tests. In addition, educator candidates make a valuable contribution to the test development process by participating in the pilot testing of test items.

One of the most important ways educators contribute to the MTEL is by serving as members of test development committees. Educators can be nominated by their colleagues, department heads, principals, college and university deans and department chairs, association personnel, or another educator organization, or they can apply directly to participate on MTEL committees. Nomination and application materials can be found on the MTEL committee recruitment website . Successful applicants are educators and faculty who are committed to ensuring the accuracy, quality, relevance, freedom from bias, and rigor of the MTEL tests.

Committee members are educators licensed and teaching in Massachusetts public schools or faculty who prepare undergraduate and graduate educator candidates in education, liberal arts and sciences, and fine arts courses related to the test content. Committees are composed of educators who reflect the racial, gender, ethnic, and regional diversity of Massachusetts. Typically, committees include:

• public school educators (a majority of committee members) and higher education faculty;
• educators from different levels of teaching (e.g., early childhood, elementary, middle, and secondary levels) as appropriate for the license;
• members of professional associations and other organizations;
• diverse racial, ethnic, and cultural groups;
• educators with expertise in special needs; and
• educators and faculty from different geographic locations and diverse school settings (e.g., urban areas, rural areas, large schools, small schools).

The following test development committees are formed for participation in MTEL development activities.

• Content Advisory Committee (CAC): The Content Advisory Committee members review objectives and test items for accuracy and freedom from bias. The committee ensures that items match the test objectives, are aligned with the Massachusetts Curriculum Frameworks and student learning standards, and are relevant to the job of a Massachusetts educator.

• Bias Review Committee (BRC): The Bias Review Committee reviews testing materials for potential bias. Typically, the BRC reviews test materials prior to the review by the Content Advisory Committee. Bias Review Committee members are asked to review each item for potential bias based upon a set of criteria related to content, language, offense, and stereotypes.

  Committee members work at every stage of MTEL test development to ensure that test materials are accurate, reflective of the standards of Massachusetts, and free from bias. Educators review MTEL materials with a focus on excluding language, content, or perspectives that might disadvantage educator candidates based on background characteristics irrelevant to the purpose of the test. In addition, committee members work to ensure that the MTEL includes content and perspectives that reflect the diversity of the Massachusetts population. Bias review is the responsibility of every educator who reviews MTEL test materials.

• Qualifying (Passing) Score Panel: Panelists' qualifications for the Qualifying Score Panel are similar to those required to participate on the Content Advisory Committee. The panel consists of educators licensed and teaching in Massachusetts public schools; college faculty who are teaching (or have taught) undergraduate or graduate arts and sciences courses in which educator candidates are enrolled, or who prepare undergraduate or graduate education students for educator licensure; and other educators in the content area who are familiar with issues of bias and special needs. About 25 percent of the Content Advisory Committee members are invited to participate in passing score activities, while the majority of the panel consists of Massachusetts educators who have not yet participated in the test development process for the field. The Department makes the final determination of the membership of the Qualifying Score Panel.

Prepare Draft Test Objectives and Specifications

Evaluation Systems prepares test objectives and assessment specifications for newly developed and updated test fields. Test objectives are broad, meaningful statements of the subject matter knowledge required for entry-level teaching in Massachusetts classrooms. They are designed to communicate the structure and content of the tests to candidates, educator preparation faculty, and other interested parties.

Test objectives are organized into major areas of content called subareas. Each subarea is further defined by a set of objectives. Each test objective includes descriptive statements that provide examples of content covered by the objective.

The following chart shows the relationship among subareas, test objectives, descriptive statements, and test items in a typical test field.

The title of the chart is Sample. The first line says General Science. A red callout indicates that this is the Test. The next line is indented one level and says Science and Technology/Engineering. A red callout indicates that this is the Subarea. Next is a paragraph that is indented two levels. It says 0014. Understand and apply basic concepts and principles of life science to interpret and analyze phenomena. The red callout indicates that this is the Objective. Next is a bulleted paragraph that is indented three levels. It says For example: basic characteristics and needs of living things; basic concepts and processes related to cells and organisms; plant structures, functions, and processes (e.g., photosynthesis); the systems of the human body; basic principles of genetics and heredity; and how organisms interact with one another and their environments. The red callout indicates that this is the Descriptive Statement. Next is a boxed question with responses, identified by the red callout as a Test Question. It reads: A major role the nucleus plays in the functioning of a cell is to: A. regulate the movement of materials in and out of the cell. B. provide energy that the cell needs to operate C. store food, waste products generated in the cell, and water. D. control the division and growth of cells.

Committee Members Review Test Objectives and Specifications

For each test, an objective review conference is held at which Bias Review and Content Advisory committees review both the test objectives and specifications for bias and content-related issues. The committees each convene for a one-day meeting to review the draft test objectives and test specifications. The Bias Review Committee first reviews the draft objectives for potential bias and makes recommendations for the Content Advisory Committee to consider and act upon. The Content Advisory Committee then reviews the content of the draft test objectives for a field for inclusion of content that is appropriate in terms of both scope and depth.

After the draft test objectives have been reviewed and revised by the Bias Review and the Content Advisory Committees, the Department reviews and approves the draft test objectives. Evaluation Systems incorporates approved revisions into the objectives and prepares them for the content validation survey.

Conduct Content Validation Survey of Test Objectives

For every newly developed or updated test, a content validation survey is conducted. The content validation survey provides evidence that the test objectives are valid—that is, they specify content for testing that is considered important for the job of a Massachusetts educator. Through the survey process, educators in the field rate the objectives on their degree of importance for entry-level teaching.

A large sample of educators, school administrators, and educator preparation faculty is selected to participate in the survey. The survey asks about:

• the importance of each objective for entry-level teaching in Massachusetts public schools; and
• how well the set of objectives represents the subject matter knowledge required for entry-level teaching in Massachusetts public schools.

Educators who participate in the content validation survey can contribute to the test development process by rating and commenting on the proposed set of test objectives.

Committee Members Review Test Items and Validation Survey Results

Following the content validation survey, test items are developed to match the test objectives that educators have rated as important and, therefore, valid. The newly developed items are the product of the combined work of content experts, educators, item development specialists, psychometricians, and content reviewers.

Bias Review and Content Advisory committees review the results of the content validation survey and then review the newly developed draft test items at the item review conference, which typically spans one to two days for each committee review. Bias Review Committee members are asked to review each item for potential bias based upon a set of review criteria relating to areas of content, language, offense, and stereotypes. Content Advisory Committee members are asked to review each item and consider its alignment to the relevant validated test objective, as well as its accuracy, freedom from bias, and job-relatedness.

The committee-approved test items are finalized, reviewed and approved by the Department, and prepared for pilot testing.

Test Items Pilot Tested

All newly developed, committee-approved items are pilot tested before they are used on a test form. Items are pilot tested in order to gather information about their quality and technical characteristics. Acceptable item statistics based on pilot testing serve as another source of evidence regarding the importance and relevance of the test content for educator candidates.

Candidates for educator licensure and other eligible students participate in pilot tests, either at preparation institutions or operational administrations of the MTEL. The pilot tests provide educator candidates with the opportunity to respond to test items that may be included on future test forms and to get a sense of how they might perform on the test in the future. Candidate responses to pilot test items contribute to the test development process; the responses can be analyzed and the technical quality of the items reviewed. In addition, candidates provide their comments about individual test items.

Committee Members Establish Marker Responses

Members of the Content Advisory Committee meet to review pilot test responses to the open-response item components of the tests. The purpose of the Marker Response Review Meeting is to identify a set of responses that correspond to, or "mark," the points on the MTEL scoring scale. The committee selects a set of responses for the test field that will serve as "markers," or examples, of each possible score point (1–4) for use in training scorers at the scoring sessions held following each test administration.

Committee members independently review a set of responses. For each score point, they are asked to come to consensus and select the one response that best represents the score point description. Committee members are encouraged to discuss the selections of the set of marker responses that would best serve as both an anchor set for the open-response item used at the first operational administration and as a "historic anchor set" for the test field for the duration of the program. This historic anchor set is used at the beginning of the training process for all future scorer training sessions.

As approved by the Department, Evaluation Systems uses the marker responses identified by the MTEL marker response committees during scorer orientation for operational scoring.

Test Administration Begins

Newly developed and updated tests are released and administered according to the MTEL schedule for the type and volume of the test field. The test information page for a field includes the testing and score reporting schedule.

Commissioner Establishes Qualifying Score with Input from Massachusetts Educators

Following the release of a newly developed or updated test, a Qualifying Score Meeting date is established and a Qualifying Score panel is identified. The panel is made up of Massachusetts public school educators and college and university faculty—some of whom served on the Content Advisory Committee or Bias Review Committee.

During the Qualifying Score Meeting, the committee participates in a structured process that includes a test-taking activity, training, a discussion about the requirements for entry-level teaching, and rating of test items. This process is known as the modified Angoff procedure and is considered industry standard. Following the training, which includes a practice component, panelists provide their professional judgments concerning the performance of the "just acceptably qualified" candidate on the items on the first operational test form (percent of items expected to be answered correctly). Then, the panelists participate in a second round of ratings. In this round, they are provided with information about actual test takers' performance and the ratings of the other panelists, and are given the opportunity to reconsider, as they deem appropriate, any of their ratings. The composite of the panelists' judgments becomes the recommendation for the qualifying (passing) score for the test. At this meeting, panelists may also comment on items based on the review criteria of accuracy and freedom from bias.

Following the Qualifying Score Meeting, the Massachusetts Commissioner of Elementary and Secondary Education reviews the panel's recommendation and the performance of candidates on the first test form and sets the qualifying score for the test.

Educators Score the MTEL Tests / Test Results Reported

Once a test field has been developed or updated, it is administered at regular MTEL test administrations. Following each administration, all test materials are returned to Evaluation Systems for scoring. MTEL tests consist of multiple-choice and open-response items. The multiple-choice items are scored by computer. The open-response items are scored by Massachusetts scorers, including Massachusetts educators and educator preparation and arts and sciences faculty, qualified in the specific test areas.

While scorers' qualifications may vary depending on the types of items they will score, in general, scorers have qualifications such as

• a Massachusetts educator license/certificate;
• teaching experience in Massachusetts public schools;
• experience as a faculty member at an educator preparation program, teaching courses in which educator candidates are enrolled; and/or
• job experience in editing, proofreading, or writing.

The open-response items are scored holistically according to standardized procedures, using the scoring scales approved by the Department. Scorers are trained in the scoring process. At least two scorers independently score each candidate's responses. If there is a discrepancy in the scores assigned to a candidate's response, the response is scored by another scorer to resolve the discrepancy. Scorers are monitored during the scoring process to ensure consistency in scoring. Score scales and sample responses for the different types of open-response items on each test can be found in the Test Information Guide for each test.

Once scoring is complete, the combined multiple-choice and open-response scores for a candidate's test are compared with the approved qualifying (passing) score for the field. Score reports are generated for candidates and released within five weeks of the test administration date.