A theory of change is a succinct description of a program or approach and the mechanisms through which it is expected to improve its targeted outcomes. A theory of change, which may be referred to by other terms such as pathway of change, engine of change, blueprint, logic model or theory of action, identifies the important outcomes and indicates what intermediate outcomes to look for to ensure that the program is on the right path to improving targeted outcomes (Reisman & Gienapp 2004).

A theory of change is useful throughout all stages of a program's development. Initially, creating a theory of change can help stakeholders reach a common understanding of the program and how it will accomplish desired outcomes. Later, it becomes a tool for stakeholders to communicate with others about the program and its expected benefits. An explicit theory of change also provides a foundation for evaluating the program. It guides the development of research questions, informs evaluation design, and aids in interpreting and presenting evaluation results.

The Teachers Institute Theory of Change Maps Pathways to Teacher and Student Outcomes.

The Teachers Institute theory of change describes how program founders designed the Teachers Institutes to improve teaching and student learning. The Understandings and Procedures (Yale National Initiative 2007) provide a clear description of the essential features of the Teachers Institute approach. The theory of change extends this to focus on pathways to the expected benefits of following the Understandings and Procedures.

At its most basic, the theory of change illustrates how teacher and university faculty backgrounds, characteristics, and interests, as well as the leadership of the subset of Fellows who serve as seminar Coordinators, shape the implementation of Teachers Institute seminars (Figure 1). The immediate products of the seminars are the curriculum units created by participants and the professional recognition and faculty privileges at the university that participating teachers receive upon completion.

These immediate products are expected to lead to three strands of outcomes corresponding to teachers, students, and university faculty. For teachers, seminar participation is expected to result in increased content and pedagogical knowledge, which in turn is expected to improve the quality of their instruction. For students, teachers' increased knowledge and improved instruction are expected to lead to greater motivation to learn. The experiences of university faculty who lead seminars are expected to improve their own pedagogical skills, enhance their disposition to collaborate, and enhance their sense of being part of a learning community.

Ultimately, these intermediate outcomes are expected to increase teacher retention and advancement and improve teachers' performance (as assessed in school district teacher evaluation systems), enhance student learning of curriculum topics, and enhance the contributions of university faculty to public education. All of these outcomes converge to support higher student achievement.

The Teachers Institute theory of change has a longitudinal dimension that is difficult to illustrate but is crucial for understanding the potential impact of Teachers Institutes over time. Although a relatively small number of teachers participate in seminars in a given year, over time a significant proportion of teachers will participate. Some teachers will participate in multiple years, which is expected to strengthen outcomes for those teachers and their students.

Higher teacher retention maximizes the benefits that compound over time. Teachers who remain teaching in the district continue to use Institute-developed curriculum units and apply their enhanced knowledge and classroom practices in teaching future cohorts of students. Participating teachers who remain in their district are likely to grow into leadership roles and continue to foster collaboration, higher morale and collegiality among teachers.

Curriculum units are available for use by other teachers, further extending the potential effects of the Teachers Institute seminars across teachers and over time. In New Haven Public Schools, where the Yale-New Haven Teachers Institute serves approximately 50 teachers each year, simulations show that under reasonable assumptions about class sizes, teacher attrition, and returning Fellows, two-thirds of students in the school district at any time during a five-year period would be exposed to a curriculum unit taught by a Teachers Institute Fellow. In addition, approximately two-thirds of students enrolled at any time during the five-year period would be exposed to Institute-developed curriculum units taught by other teachers (Arnold 2010).

Research and Best Practices Support the Teachers Institute Theory of Change.

The Teachers Institute approach was developed more than 30 years ago, but it remains a state-of-the-art program. The theory of change is grounded in the founders' vision for the program, affirmed by participating teachers' reports about their experiences and the benefits of participating, and backed by research and experts' current understanding of best practices.

What Participating Teachers and Program-Sponsored Research Say

Outcomes research conducted by the Yale-New Haven Teachers Institute and the Yale National Initiative shows that participating teachers consistently rate their Institute experience as valuable and report experiencing key intermediate outcomes. Annual surveys have consistently shown that participating teachers rate the Institute programs higher than other professional development programs in developing knowledge, skills, enthusiasm, high expectations of students, and capacities to motivate students (Smith 2004).

An analysis of surveys of Institute Fellows from 2003 to 2008 supports many of the pathways identified in the theory of change. The surveys show that teachers in all sites were motivated to participate in the Institute by the opportunity to develop materials to motivate their students, to develop curriculum fitted to their needs, to increase their content knowledge, and to exercise intellectual independence (Smith 2009). After participating in the Institute, teachers overwhelmingly agreed or strongly agreed that the seminars provided them with professionally useful new knowledge (87-94%) and that the seminars raised their expectations of their students (87-95%).

Periodic surveys of participating and nonparticipating teachers about their use of curriculum units developed by Teachers Institute Fellows show that virtually all Fellows go on to teach the units they prepared in their seminar (87% taught their units in 2 to 5 classes). Most teachers reported presenting their units in teacher-led discussions and stressed writing exercises and activities designed to strengthen speaking, listening, vocabulary, and reasoning skills. Two-thirds of all Fellows rated the units written by themselves or other Fellows as superior to all other types of curriculum they had used. Many reported that their units were written to be interdisciplinary and supported successful team teaching (Smith 2009).

Smith (2009) reports that Teachers Institutes are influential in retaining existing teachers because participating teachers find the seminars stimulating and feel "respected and acknowledged as creative, caring educated colleagues." Quantitative analyses of data in New Haven confirm this. Of those teachers who had been Institute Fellows by the end of the 2000-2001 school year, 63% were still teaching in New Haven in 2004-2005, compared with 43% of other teachers. Fellows were almost twice as likely as non-Fellows to remain teaching in the district five years later, controlling for differences in race, sex, and years of teaching experience, a significant difference (Smith 2009).

The majority of participating teachers rate student attention, motivation, interest, and content mastery as higher during Institute-prepared curriculum units compared with other curriculum units (Smith 2009). A retrospective analysis of student achievement outcomes conducted during the same 5-year period, however, did not find significant effects of the Teachers Institutes on student achievement test scores or course grades. This was not unexpected, because the curriculum units were not aligned with achievement tests. Smith (2009) concluded that student outcomes data more closely tied to the goals of the Institute-prepared curriculum units, or more extensive and reliable data on student outcomes more generally, is required to demonstrate Institute impacts on students.

What Other Research Says

Strong causal research on the effectiveness of teacher professional development is still limited, but it provides evidence that teacher professional development can improve the intermediate and longer-term outcomes that the Teachers Institutes are designed to influence, especially when the professional development has features of the Teachers Institutes. Other correlational research also suggests that teacher professional development can improve intermediate and longer-term outcomes.

A number of studies suggest that professional development can increase teacher content knowledge. For example, Weiss and Miller (2006) identified six pretest/-post-test studies of mathematics professional development programs and found positive gains in content knowledge. Hill and Ball (2004) found that teachers participating in the California Professional Development Institutes in mathematics made significant gains in their mathematics content knowledge; institutes with greater duration and that focused on analysis, reasoning, and communication demonstrated larger gains. Of the 25 evaluation studies reviewed in Blank, de las Alas, and Smith (2008), 10 reported evidence of measurable effects on teacher content knowledge.

Research has also demonstrated that increased content knowledge can influence classroom practices. For example, Hill et al. (2008) examined associations between mathematical knowledge for teaching and the mathematical quality of instruction and found a significant, strong association between them. Supovitz and Turner (2000) found that teachers who felt more well-prepared to teach science topics in elementary school were more likely to engage in reform-based teaching practices and create a classroom culture of investigation.

Studies have also shown that teacher professional development programs can have a positive impact on classroom practices. Scher and O'Reilly (2009) conducted a meta-analysis of strong causal studies and found that the pooled effect size1 of math and/or science professional development on teacher practice was .63 and highly significant. Porter et al. (2000) analyzed longitudinal data from science and mathematics teachers in 30 schools across 10 districts and found that professional development programs focused on specific, higher-order teaching strategies were associated with teachers' use of those strategies in the classroom, and the associations were even stronger when the professional development was reform-type, involved active learning, was coherent, and involved collective participation.

Several studies point to the value of professional development that supports teachers in developing their own curriculum. Carpenter et al. (1989) evaluated a professional development program that made teachers aware of research findings, then supported them in developing curriculum units. The evaluation documented positive effects on teacher knowledge, improvements in observed teacher practices, and higher student achievement. McCutchen et al. (2002) provided an instructional institute for teachers focused on increasing teacher knowledge and supporting teachers in developing their own curriculum around what they learned. The evaluation documented positive effects on teacher knowledge and improvements in observed teaching practices.

Teacher professional development can have a positive impact on student attitudes and student perceptions. Scher and O'Reilly (2009) found significant pooled effect sizes of math and/or science professional development on student attitudes (.42) and student perceptions (.57).

The ultimate goal of teacher professional development is to increase student learning and achievement. Yoon et al. (2007) identified nine studies of professional development that met What Works Clearinghouse evidence standards. All nine studies employed workshops or summer institutes for elementary school teachers and focused on a range of content areas. Most reported effects on student achievement were positive; 8 were statistically significant, and 9 of the remaining 12 were substantively important, with effect sizes of at least .25. The average effect size was .54. Studies of professional development that was more than 14 hours long showed positive effects while studies of professional development that was shorter did not. The meta-analysis conducted by Scher and O'Reilly (2009) also found positive effects on student math and science achievement, with pooled effect sizes ranging from .12 to .38. Subgroup analyses showed that impacts were concentrated in programs that took place over at least one academic year, focused on both content and pedagogy, and included both a workshop and coaching or another component. Blank, de las Alas, and Smith (2008) found that one third of the evaluation studies they reviewed reported measurable effects of teacher professional development in math and science.

What Experts Say

To help states and school districts making decisions about teacher learning and development, organizations providing technical assistance have synthesized research results and advice of experts to identify features and practices that make it more likely that a teacher professional development program will be effective. The National Comprehensive Center for Teacher Quality, for example, recently identified five features of high-quality professional development: (1) alignment with school goals,

district standards and assessments, and other professional learning activities; (2) focus on core content and modeling of teaching strategies for the content; (3) inclusion of opportunities for active learning of new teaching strategies; (4) provision of opportunities for collaboration among teachers; and (5) inclusion of embedded follow-up and continuous feedback (Archibald et al. 2011).

The Teachers Institute approach encompasses many of these recommended best practices: (1) each Teachers Institute is aligned with school reform goals and is designed to support a district's strategic plan, and the curriculum unit each teacher develops is aligned with state and local standards; (2) Institute seminars deepen teachers' knowledge of core subjects and assist them in developing strategies to teach their own students what they have learned; ­(3) teachers are active learners in Institute seminars, receiving feedback from their peers and often trying out the units with their students as they prepare them; (4) the collegial exchange of ideas among school teachers and university faculty members lies at the very center of Institute seminars and is a tenet of the Institute approach; and (5) Institute seminars are of substantial duration, involving a minimum of 26 hours in session plus substantially more for meeting with seminar leaders, researching seminar topics, and writing curriculum units. Teachers Institutes are planned, implemented, and sustained by teachers. Each Institute seminar topic is suggested by teachers based on what they think will enrich their classroom instruction. Teachers recruit their colleagues to participate, and one teacher in each seminar plays a coordinating role to handle administrative details, help establish collegiality, and act as a resource for other teachers.

The Theory of Change Needs to Be Rigorously Tested Using a Strong Evaluation Design.

The Teachers Institute theory of change has a solid foundation in experience and research, but it is still a theory that needs more testing with research designed to assess the causal relationships in the theory. Evaluation of the Teachers Institute approach employing a strong causal research design to explore the pathways and measure the magnitude of Institute impacts on intermediate and longer-term outcomes is needed to confirm that the theory of change provides an accurate map from Institute participation to outcomes.

Planning for a strong evaluation of the Teachers Institute approach is under way. The Yale-New Haven Teachers Institute is undertaking a major redesign of its Management Information System (MIS) to better support program activities and to capture data needed for research and evaluation. To the extent possible, the new MIS will be designed to accommodate variations among local Teachers Institutes and support their operations, as well as promote the collection of consistent data across locations.

Evaluation design activities are also under way. In the coming year, the Yale National Initiative will work with local Teachers Institute staff to prioritize research questions and identify Institutes where a strong evaluation is feasible. Building on past work to identify evaluation design options, the Initiative will develop a detailed evaluation design, sample design, and data collection and analysis plan for producing strong evidence of the effectiveness of the Teachers Institute approach. The Yale National Initiative will seek funding and contract with an independent evaluation firm to carry out the evaluation plan.

With stronger evaluation research based on the theory of change, school district decision-makers can have more confidence that implementing a local Teachers Institute will take them where they want to go and understand how it will do so.

More broadly, lessons from the Teachers Institute evaluation will contribute to the learning and understanding of researchers and policymakers about how best to support teacher learning and development and thereby promote student learning and achievement.

References

Archibald, S., Coggshall, J. G., Croft, A., and Goe, L. (2011). High-Quality Professional Development for all Teachers: Effective Allocating Resources. Chicago: National Comprehensive Center for Teacher Quality.

Arnold, T. B. (2010). Modeling the Number of Students Reached by the Teachers Institute. New Haven, CT: Yale University.

Blank, R. K., de las Alas, N., and Smith, C. (2008). Does Teacher Professional Development Have Effects on Teaching and Learning? Washington, DC: Council of Chief State School Officers.

Carpenter, T. P., Fennema, E., Peterson, P.L., Chiang, C. P., and Loef, M. (1989). Using Knowledge of Children's Mathematics Thinking in Classroom Teaching: An Experimental Study. American Educational Research Journal, vol. 26, no. 4, pp. 499-531.

Hill, H. C., and Ball, D. L. (2004). Learning Mathematics for Teaching: Results from California's Mathematics Professional Development Institutes. Journal of Research in Science Teaching, vol. 36, no. 8, pp. 916-29.

Hill, H.C., Blunk, M. L., Charalambous, C. Y., Lewis, J. M., Phelps, G. C., and Sleep, L. (2008). Mathematical Knowledge for Teaching and the Mathematical Quality of Instruction: An Exploratory Study. Cognition and Instruction, vol. 26, no. 4, pp. 530-511.

McCutchen, D., Abbott, R. D., Green, L. B., Beretvas, S. N., Cox, S., Potter, N. S., Quiroga, T., et al. (2002). Beginning Literacy. Journal of Learning Disabilities, vol. 35, no. 1, pp. 69-86.

Porter, A. C., Garet, M. S., Desimone, L., Yoon, K.S., and Birman, B. F. (2000). Does Professional Development Change Teaching Practice? Results from a Three-Year Study. Washington, DC: American Institutes for Research.

Reisman, J., and Gienapp, A. (2004). Theory of Change: A Practical Tool for Action, Results, and Learning. Seattle, WA: Organizational Research Services.

Scher, L. and O'Reilly, F. (2009). Professional Development for K-12 Math and Science: What do We Really Know? Journal of Research on Educational Effectiveness, vol. 2, no. 3, pp. 209-249.

Smith, R. M. (2004). To Motivate My Students. New Haven, CT: Yale-New Haven Teachers Institute, Yale University.

Smith, R. M. (2009). To Strengthen Teaching: An Evaluation of Teachers Institute Experiences. New Haven, CT: Yale-New Haven Teachers Institute, Yale University.

Supovitz, J. A., and Turner, H. M. (2000). The Effects of Professional Development on Science Teaching Practices and Classroom Culture. Journal of Research in Science Teaching, vol, 37, no. 9, pp. 963-980.

Weiss, I. R., and Miller, B. (2006). Deepening Teacher Content Knowledge for Teaching: A Review of the Evidence. Paper presented at the MSP Evaluation Summit II, Minneapolis, MN, October 4-5, 2006. Accessed at http://km.mspnet.org/index.cfm/14124, October 20, 2011.

Yale National Initiative (2007). The Teachers Institute Approach. New Haven, CT: Yale-New Haven Teachers Institute, Yale University.

Yoon, K. S., Duncan, T., Lee, S. W-Y., Scarloss, B., and Shapley, K. L. (2007). Reviewing the Evidence on How Teacher Professional Development Affects Student Achievement (Issues & Answers Report, REL 2007-No. 033). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southwest.