Race to the Top

According to a recent KUOW FM report, John Cronin of NWEA, Inc. apparently laughed about the Garfield MAP test protest, saying, “(…) I really can’t say that I know what’s motivating them.” See/hear: Seattle Superintendent And Testing Company Defend Standardized Test.

But this same fellow wrote a letter to the Charleston School District in 2010 advising them that MAP should not be used for teacher evaluations. Yet the Seattle Public School District is using it that way, effectively making it a high-stakes test. (This memo has since disappeared from the Internet.)

(Also see: MAP test manufacturer warns: MAP test should NOT be used to evaluate teachers. — So why is Seattle Public Schools doing just that?)

I contacted Mr. Cronin about this memo back in 2011, to express my concern that the Seattle School District was misusing the MAP test to evaluate teachers – exactly what he warned the Charleston School District not to do. I asked if he would write a similar letter to our school board and superintendent, and I warned him about the various reasons why Seattle may sour on the MAP test product.

For the record, here is our exchange:

4/12/11

to john.cronin

John Cronin,
Kingsbury Center
Northwest Evaluation Association
5885 SW Meadows Road Suite 200
Lake Oswego, OR 97035
(503) 624-1951
john.cronin@nwea.org

Dear John,

I am a Seattle Public Schools parent with serious concerns about how the MAP® test is being used in Seattle. I want to bring to your attention the fact that our school district is misusing NWEA’s product and this is causing a growing resentment toward it.

In short, SPS is using the MAP test to evaluate its teachers.

That is not what the test was designed for nor how the Seattle schools community was told it would be used when SPS purchased it. We were told it would help teachers understand our children’s learning needs. Instead, it is being used by district administrators as a tool to assess, reward or punish teachers and principals. This is an unfair and inaccurate way to measure teachers and it is affecting what is taught in our schools — not for the better.

Thus the Seattle Public School District is effectively misusing the MAP as a high-stakes test.

I recently came across your letter to the Charleston School District (http://media.charleston.net/2010/pdf/kingsburymemoccsd_102210.pdf) which warned against using NWEA’s MAP test to evaluate teachers. Would you consider writing a similar letter to our new interim school superintendent and school board with the advice you gave the Charleston school district?
Here are their e-mail addresses: (Dr. Susan Enfield, Interim Superintendent) superintendent@seattleschools.org, <michael.debell@seattleschools.org>, betty.patu@seattleschools.org, <ksblum@seattleschools.org>, <peter.maier@seattleschools.org>, <sherry.carr@seattleschools.org>, <harium.martin-morris@seattleschools.org>, <steve.sundquist@seattleschools.org>

Also, the circumstances under which the MAP test was purchased by SPS has resulted in suspicion, leading many of us to doubt whether MAP was truly the best product available, or chosen for some other reason. As you may know, the Seattle Public School District purchased a subscription to the MAP in 2009. At that time, our superintendent was Maria Goodloe-Johnson. She was a member of the board of directors of NWEA when the purchase took place, and failed to disclose this fact. In 2010, the state auditor cited this as an ethics violation/conflict of interest and Goodloe-Johnson was forced to step down from the NWEA board.

This appearance of impropriety has potentially undermined any legitimate value the product may have.

The result of all this is the MAP test is developing a negative reputation and connotation in Seattle. I thought you should know this. Surely this was not NWEA’s intent. I would assume that this is not how NWEA wants a major school district to feel about its product — or to sour on the MAP entirely. Unfortunately, that is where the sentiment is headed. Those of us with blogs will continue to report this accordingly. (15 Reasons Why the Seattle School District Should Shelve the MAP® Test—ASAP.)

I believe the way the MAP test was introduced and is being used by SPS has been handled in a less than optimal and upfront manner. The district recently fired the superintendent who was associated with NWEA. We are aiming to turn a new page in this district. Don’t you agree that it would be better for everyone if MAP were not part of a lingering resentment associated with our previous superintendent?

Perhaps the MAP can be used at the beginning of the year in the way it was intended – as a tool designed to help teachers know where their incoming students are academically. I do not believe it needs to be administered three times a year to all students. That is proving to be excessive and costly in many ways.

If a better arrangement cannot be reached and this misuse of the test continues, we the parents of SPS will likely advocate for the elimination of MAP altogether.

Thank you in advance for your consideration of my thoughts and concerns. I look forward to your reply.

Sincerely,

Sue Peters

Seattle Public Schools parent
Co-editor, Seattle Education 2010
Founding member, Parents Across America
Education blogger, the Huffington Post

p.s. As a journalist, I also appreciated your comments about the L.A. Times’ teacher rankings (http://www.kingsburycenter.org/blog/johncronin/2010/09/07/la-times-teacher-rankings-and-code-journalistic-ethics) and share your sentiments (My (unanswered) letter to the L.A. Times about its teacher-ranking witch hunt: “Have you no decency?” — Also: New Univ. of Colorado report says the Times’ research was seriously flawed).

John Cronin <john.cronin@nwea.org>
4/13/11

to me

Ms. Peters,

Thank you for contacting us and expressing your concerns.

As you noted, we did comment on a particular board policy that was proposed for the Charleston School District. In this case, the district asked us for our comments on the proposed policy, which we offered. We do stand by those comments, with the caveat they were intended to address the policy as it was proposed to be implemented in Charleston. However, NWEA’s policy is not to comment on specific school system policies and practices in regard to the use of our assessment unless the school system requests it, so I can’t offer a comment on the Seattle situation.

As you know, many states and school systems are considering adapting policies that would use assessments, most commonly the state assessment, but sometimes others including ours, in the professional evaluation of teachers. We understand the controversy around the issue of using tests in teacher evaluation, and we are in the process of drafting an official position statement in regard to these practices. When the statement is released, we would be happy to forward it to you. Just let me know if you would like a copy.

I appreciate your concern for the teachers and principals who are trying their best to serve Seattle’s children and hope there will be an opportunity for everyone to move forward.

John Cronin

Director – The Kingsbury Center at NWEA

KINGSBURYCENTER.ORG | NWEA.ORG | Partnering to Help All Kids Learn

s. peters

Post Script:

Just in case it disappears again, here is the memo in full:

Charleston County School District Teacher Evaluation Policy

The Kingsbury Center at NWEA is an independent center within the Northwest Evaluation Association, created to conduct research on trends in educational policy and educational assessment. Through collaborative research studies with foundations, think‐tanks, universities and NWEA partner schools, the Kingsbury Center is helping to change the conversations around education’s most challenging issues.

The Center and our partners strive to influence the thinking of leaders at all levels of the educational system. Our work ranges from research that influences national policy to reports that provide actionable information to school systems.

Recently, it came to our attention that a database was published by the Charleston Post and Courier that reported on the academic growth of students taught by Charleston County School District teachers.

In this database, CCSD teachers were ranked on the basis of the percentages of their students whose growth matched or exceeded the growth of Virtual Comparison Groups of students that were created by the Kingsbury Center for the district. As a consequence of that publication, the CCSD school board is now considering implementation of a teacher accountability policy in which 60% of a teacher’s evaluation would be dependent on student growth and achievement data of this type. When we learned of this, we asked the district administration for permission to send our comments on the proposed policy to the board, and we were encouraged to do so.

The Kingsbury Center supports efforts to implement accountability for both schools and teachers. We believe that Charleston students deserve no less. We also believe that student achievement data can inform the teacher evaluation process. But nearly all experts in our field agree that test results should not supersede the principal’s role in teacher evaluation. There are several reasons for this:

1. The proposed board policy establishes an expectation that each (we take this to mean each and every) student advance by no less than one academic year. This could be interpreted to mean that any situation in which one or more students failed to meet this objective would constitute cause for a personnel action against a teacher. We doubt that is the board’s intent.

The board should be aware that, according to our most recent norms, the top 10% of schools for growth only have 64% to 73% of their students (depending on grade level) meet the “one year of growth” target.

2. The statistical methods used for these kinds of evaluations, known as “value‐added” models, are useful for evaluating schools and can play a role in the professional evaluation process for teachers. However, these statistical models are designed under a specific set of assumptions about schools, which, when not met, limit the validity of their findings. For example, the accuracy of a value‐added measure can be compromised if students and teachers are not randomly assigned to classrooms.

For example, if young teachers were routinely assigned to the most difficult classes; if veteran teachers make their own choice of teaching assignment and/or students; or if certain advanced classes were reserved exclusively for students without behavior problems; all of these scenarios potentially introduce bias into value‐added measurements. Such bias, when it exists, can mistakenly attribute higher “effect” ratings to some teachers and lower “effect” ratings to others, leading to invalid results.1

3. As the stakes associated with value‐added measurement increase, the legal requirements around its application will grow considerably stricter. Because South Carolina requires due process for experienced teachers that are proposed for termination, any evaluation policy should be written to minimize the risk of a challenge stemming from the use of test results in the process. For instance, a teacher who is terminated or placed on probation because of test scores would have cause to challenge the action if it can be proved that the procedures for assigning students to classes introduced the kinds of biases cited in the previous point.

4. Measurement error should be considered and applied if test data are being used for performance evaluation. Studies of the Educational Value‐Added Assessment System (EVAAS) in use in Tennessee since 1993 found that only one‐third of teachers could be identified as clearly different from “average” when measurement error was considered.1

5. The statistical errors associated with value‐added measures decrease dramatically as you consider larger numbers of students. For example, the statistical error associated with a classroom of 30 students in Charleston would be about 3.4 times greater than the error associated with the average Charleston school. If the class size is reduced to 20 students, the statistical error would be 4.2 times greater. This means that classroom results are likely to be far more volatile than school results over time.

6. School districts implementing value‐added systems typically rely on student achievement data in reading and mathematics. The proposed board policy requires that assessments to measure academic growth be implemented in other core subjects. Measuring growth in some of the core subjects, particularly history and social science, has so far not been attempted by test publishers and the validity of value‐added methodologies applied to these subjects is unproven. Implementing this methodology with the level of stakes proposed, may invite legal challenges to personnel actions taken on the basis of poor results. The problem becomes profound at the high school level, because of the myriad number of subjects taught. In addition, high schools cannot generally identify any ingle teacher who would be responsible for a reading or mathematics score. This is why valueadded methodologies are rarely applied in high school settings.

1 Braun, H.I., Using Student Progress to Evaluate Teachers: A Primer on Value‐Added Models. Retrieved October 7, 2010, from http://www.ets.org/Media/Research/pdf/PICVAM.pdf

Recommendations

We would recommend that the board refrain from adopting a policy in this area until the district is able to study and propose concrete strategies for addressing these issues which, unaddressed, may expose the district to potential legal liability:

a. To address the question of exactly what data would be used for evaluation of teachers who are not teaching in subjects in which value‐added measurements are currently used, particularly social studies, history, science, art, vocational education, and music.

b. To establish explicit criteria for performance on the tested measures that are validated as reasonable by using data from test publisher norms, the district or state’s past performance, or other legally defensible standards.

c. The policy should also require that statistical error be considered when applying these criteria.

d. We would recommend that student test data not receive more weight than the principal’s evaluation in importance.

e. The impact of measurement error, while relatively large when measuring individual student growth, decreases dramatically when that growth is aggregated to large groups. When the groups under consideration are several hundred, such as school level aggregations, measurement error has a much more negligible impact. Consequently, we can support using value‐added metrics as one factor among others in identifying under‐performing schools.