We conducted classroom observations from October to April, observing a total of 55 classes one or two times each. In addition, we observed lunchrooms and hallways at two sites. These observations lasted between 30 and 90 minutes, for a total of about 73 hours of observation data collected. We followed a non intrusive, hands-off, eyes-on approach and generally did not participate in classroom activities. We took field notes during observations to describe the classroom environment; classroom procedures; the teachers’ instruction; learning activities; materials used; and interactive patterns among students and between students and teachers.
Part of Research
Moreover, we also took note of interactions between teachers, since teachers co-taught some of the integrated classes common in the NT model. We wrote as much as possible of what we saw and heard during observations and included some of our own reflections or interpretations as memos written during or shortly after observations. We also met weekly to share our notes and memos so that all team members had a more complete view of what was happening at each school.
We conducted formal interviews with 16 teachers and 7 directors (i.e., principals). We recruited teachers for interviews through snowball sampling, whereby we asked directors to provide the names of two or three teachers they thought should be interviewed. Directors did not always suggest teachers they expected to say complimentary things about the model or who were implementing the model with high fidelity.
Instead, most were interested in learning from teachers they believed had not bought into the model or were not implementing the model fully. Because the data was collected in the context of an evaluation, the directors had an interest in learning how they might modify their practice and/or provide further supports and professional development to better meet teachers’ implementation needs.
How we do it?
We then invited the teacher’s directors recommended to participate in an interview, although not all consented. Therefore, the directors did not know exactly who participated among those they suggested. Next, we asked all the teachers that the directors had recommended for an interview to provide the names of additional teachers they thought we should speak with in order to gain an understanding of implementation at that school. The teachers who participated in interviews represented a sample of different content areas: two science teachers, five English teachers, four mathematics teachers, three modern languages teachers, and two business teachers. Almost half of the teachers we interviewed were mathematics or modern languages teachers, which was the result of a focused recruitment effort in response to specific partner needs as described above.
The number of interviews conducted was also limited by the time frame and budget for the evaluation. We interviewed two to three teachers from each school over the phone or at school. Each interview lasted approximately 30 to 45 minutes, for a total of about 10 hours of interview data. We followed a semi-structured protocol that enabled the evaluation team to compare similarities and differences between stakeholder expectations of the NT model and their experiences in it. Sample interview items included “Describe teacher collaboration at your school” and “Describe the leadership structure at your school.” We audio-recorded the interviews and transcribed them verbatim.
In order to analyze the data that we had collected for the New Tech implementation evaluation, we gathered all of the data documents, including observation field notes and interview transcripts. We read through all of these in order to obtain an overall understanding of what we had collected. After this preliminary reading, we reviewed the Degrees of Democracy Framework and began creating a list of possible codes, including the code examples.
Next, we utilized the NVivo data analysis software program to assign specific codes to data excerpts within the observation field notes and interview transcripts. After completing initial coding, we pulled the data we assigned to each code, and read through it, comparing the data to the descriptions of holistic democracy embedded in the Degrees of Democracy Framework. Once this reading was complete, we recorded some data in order to refine our analysis.
To check the validity of our analysis, we shared the analysis documents with the evaluation team members for peer editing because they were most familiar with the NT model, the data collection methods, the school sites, and the participants. We also shared my analysis with Philip Woods, one of the authors of the Degrees of Democracy Framework, for peer editing.