Teacher Knowledge
The SII effort to design survey items measuring knowledge for teaching mathematics and language arts is rooted in the idea that measures representing school and classroom processes are necessary in assessing student performance outcomes, but that teachers' facility in using subject matter knowledge in the context of classroom teaching is also essential. These pedagogical content knowledge measures allowed SII to investigate the effects of teachers' knowledge on student achievement, and to understand how implementation of these reform programs is mediated by teachers' content knowledge. While many potential methods for exploring and measuring teachers' content knowledge exist (i.e., interviews, observations, structured tasks), we elected to focus our efforts on developing survey measures because of the large number of teachers (over 5000) participating in SII.
Mathematics
Beginning in 2000, we undertook the development of such survey measures. Using theory, research, the study of curriculum materials and student work, and our own experience, we wrote items we believe represent the knowledge teachers use in teaching elementary mathematics - representing numbers, interpreting unusual student answers or algorithms, anticipating student difficulties with material. Based on item-writing experience, Hill and Ball (2004) developed guidelines and recommendations for developing items that tap into teachers’ pedagogical content knowledge in mathematics (here). In mathematics, the University of California Office of the President (UCOP) assisted us in piloting these items as part of an evaluation of the state's Mathematics Professional Development Institutes. This California collaboration developed into a sister project of SII, Learning Mathematics for Teaching. In addition to UCOP, initial survey development and piloting was supported by the National Science Foundation, U.S. Department of Education, and Atlantic Philanthropies.
Each elementary (K-6) item has each been piloted with over 600 elementary teachers, yielding information about item characteristics and overall scale reliabilities for piloted forms. Intended for use only in researching the content knowledge of elementary-level school educators, we have items that can be used to measure teachers' mathematical content knowledge for teaching in:
- Number and operations
- Patterns, functions, and algebra
- Geometry
Items in each category capture whether teachers can not only answer the mathematics problems they assign students, but also how teachers solve the special mathematical tasks that arise in teaching, including evaluating unusual solution methods, using mathematical definitions, representing mathematical content to students, and identifying adequate mathematical explanations.
We also have items measuring teachers' knowledge of students and content for the domain of K-6 number and operation. These items probe whether teachers recognize common student errors and solution strategies, and can anticipate whether material will be difficult or easy for students. Using the SII teacher content knowledge in mathematics items, Hill, Rowan and Ball (2005) found a positive effect of teacher mathematical knowledge on first and third graders’ gain scores. Readers wanting to learn more about the development and scaling properties of the mathematics knowledge measures should consult the article by Hill, Schilling and Ball (2004).
The battery of items, presented in the Teacher Questionnaire, was administered as part of the longitudinal design across the fours years of the SII study. Readers may wish to view the items administered in Year 1, Year 2, Year 3 and Year 4.
When should projects elect NOT to use these measures?
These measures are not appropriate for all research and evaluation efforts. Reasons for not using these measures include:
- Length. Typically, between 12-25 items are required to achieve reliabilities of between .7 and .8 for each scale. Teachers complete roughly one item per minute. Users who want to gauge teacher knowledge growth in diverse areas of the K-8 curriculum (e.g., geometry, algebra, number and operation) may need multiple scales in order to assess such growth. Alignment with your curriculum, questions, or study. Our measures were written to provide an accurate picture of teachers’ overall ability to solve mathematics problems that arise in classrooms within each of the constructs listed above. They were not written to align with any particular professional preparation program or research question. For this reason, they may not be sensitive to the actual content of your program. These measures are also not appropriate for the evaluation of programs where improving teachers’ mathematical knowledge for teaching is not a primary focus.
- Complex analyses. Instruments which carry our measures can be more difficult to design and analyze than traditional surveys.
- Small sample size. These measures were not designed or validated to make statements about any individual teacher’s level of mathematics knowledge. Instead, one might use the measures to make statements about how content knowledge differs among groups of teachers, or how a group of teachers performs at one or more time points. Usually, 60 teachers per group – or 60 teachers over time – will be adequate for comparison purposes. Effects may be found with smaller groups, but the risk of insufficient statistical power rises as group size diminishes.
- Highly knowledgeable teachers. Our measures are not well-suited to studies investigating learning among already highly knowledgeable teachers.
- Inappropriate usage. These measures cannot be used to evaluate individual teachers for tenure, pay, hiring, or any other use with high-stakes consequence. They have not been validated for this purpose.
- Non-K-8 teacher populations. These items were piloted with K-8 teachers; the joint AERA/APA Standards recommend re-piloting before use with other populations.
When should projects elect to use these measures?
These measures are best suited for investigating:
- How knowledge of mathematics for teaching corresponds to other constructs, such as instructional practice, attitude toward teaching mathematics, etc.
- How knowledge of mathematics for teaching develops. Within this category, these measures are not suited to all programs, however. Instead, they are best used with:
- Programs that intend to broadly improve teachers’ knowledge in any of the content areas named above; programs which focus only on narrow bands of the K-8 math curriculum (e.g., programs which focus only on linear functions) will not likely see positive results. Programs which target significant resources at improving teachers’ knowledge—not 2-3 hours of professional development, but sustained, serious mathematical work.
- Programs which work to improve teachers’ mathematical reasoning and analysis skills.
- Programs with pre-existing curricula. Programs without pre-existing curricula—for instance, professional development where teachers can choose their own topics for investigation—complicate the design of evaluations with these measures.
- Programs with sufficient sample size to make general statements about differences between groups or over time.
In the fall and spring, Learning Mathematics for Teaching (LMT) project training sessions are held in Ann Arbor, Michigan for researchers interested in learning how to properly scale the mathematics teacher content knowledge items. During training we provide a description of measures development from both a practical and theoretical perspective, give a short background in measurement theory and test construction, and then offer advice on logistical issues associated with measures use. There is no cost for training or for the measures themselves, but users must cover transportation and accommodation expenses. Those interested in attending training should contact Geoffrey Phelps for information about dates. For more information, readers may view the LMT website.
READING
Item development in language arts was guided by three distinctions in content knowledge for teaching reading: (1) knowledge of content, (2) knowledge of students and content, and (3) knowledge of teaching and content. The primary difference between items in each of these categories is in how content knowledge is used in teaching reading.
The knowledge of content domain came closest to measuring teachers’ common knowledge of the subject. Items in this category did not require making teaching decisions; rather, the battery of items assessed teachers’ word analysis comprehension skills.
Answering items that focused on knowledge of content and students required respondents to use their knowledge of reading to understand the range of student products encountered teaching the subject. For example, a knowledge of students and content item may have required using knowledge of reading and students to recognize common student errors or approaches to reading.
Items on knowledge of teaching and content required respondents to use knowledge of reading to decide between different teaching actions. For instance, an item may have required using knowledge of reading to determine remedial action for a student struggling over a difficult word.
With one exception, these domains were not designed to be mutually exclusive. Respondents answering teaching decision items may have needed to use both knowledge of students and content. Likewise, respondents answering knowledge of students and content items may have needed to draw on content knowledge. In contrast, however, items assessing knowledge of content were designed to be largely distinct.
In addition to distinctions in the types of content knowledge for teaching noted above, items were also measuring content topic knowledge in (1) comprehension and (2) word analysis. Within comprehension, the range items included: morphology, vocabulary, comprehension strategies and questions, genre, fluency, and other topics related to comprehending the meaning of words and text. Word analysis included: phonemic awareness, letter sound relationships, word frequency, and other topics related to the reading and decoding of words and their print and sound elements.
Together these distinctions define six areas or domains of subject knowledge for teaching reading (see, Table 1 below). The actual items should be thought of as sampling from the wider range of possibilities that define these domains. To date, we have written items in five of the six domains. We currently have no knowledge of teaching and content items in the area of word analysis. Readers may view an article on the development and measurement properties of teacher content knowledge scales by Phelps and Schilling (2004) here.
| Table 1. Content Knowledge for Teaching Reading Domains | |||
|---|---|---|---|
| Knowledge of Content | Knowledge of Students and Content | Knowledge of Teaching and Content | |
| Comprehension | |||
| Word Analysis | XXXXXXXXXXXXXXXXXX | ||
As in mathematics, the battery of items in language arts (reading) was administered as part of the TQ longitudinal design. Readers may wish to view the items administered in Year 1, Year 2, Year 3 and Year 4.