Research / Program Evaluation


*If you would also like to download this document, please click the link.

The Battle Creek Area Mathematics and Science Center has developed a Kindergarten through Seventh grade inquiry-based science curriculum that is aligned with the Michigan Grade Level Content Expectations. The purpose of the program is to provide good, inquiry-based science instruction in the classroom with opportunities for students to engage in all four strands of science proficiency (physical science, life science, earth science, and inquiry and technology). The curriculum provides teachers with classroom instruction that includes opportunities for interaction in the classroom, where students carry out investigations, talk and write about their observations and emerging understandings, and discuss ways to test them. Each unit undergoes an extensive development and evaluation process.

For over a decade, the BCAMSC has been evaluating and collecting data on the performance of the science units across the participating districts:


1997 | Science and Mathematics Program Improvement (SAMPI) Evaluation of BCAMSC Districts

“Calhoun County students continue to lag state norms in key measures of academic performance in science. However, we are beginning to see indications of significant progress, particularly in those school districts participating in the BCAMSC Unit/Kit program.

It is interesting to note that since 1990 we have seen a significant divergence of performance between local area BCAMSC member districts and non-participating districts. The ability to adjust to the new demands of the state test has been dramatically better in member districts than non-participating districts. The difference is significant and suggests that real progress has been made.”

Mark Jenness Ed.D
Director, Science and Mathematics Program Improvement (SAMPI)
Western Michigan University, 1997


1999 | Analysis and Review of the Battle Creek Area Mathematics and Science Center “How Things Move-Second Grade Unit/Kit.

“Using seven criteria from the American Association for the Advancement of Science Project 2061 Curriculum Analysis Method the Unit ‘How things Move’ fairs very well. This unit with the accompanying kit of supplies and equipment would be a strong component of any physical science curriculum. You should be pleased with the quality of the materials you have produced.”

Mark Jenness Ed.D
Director, Science and Mathematics Program Improvement (SAMPI)
Western Michigan University, 11/1999


2003 | Assessment of K-5 Improved MEAP Performance, within BCAMSC Member Districts

“Fifth Grade science MEAP scores were compared across 15 elementary schools (Battle Creek Public schools) relative to state and district average performance (1996-2002). An analysis was conducted comparing building performance and BCAMSC involvement against district and state averages. Science teacher leadership as a variable explains nearly 50% of the variations in MEAP performance across all fifteen elementary school buildings included in the analysis. This is strong evidence of the value and worth of the services contributed by the BCAMSC Outreach.

Summary: From this preliminary analysis it is reasonable to assert that the BCAMSC science outreach program activities that build science infrastructure and capacity among elementary staff have had a strong influence on the extent to which low, moderate, and high performing buildings have been able to demonstrate progress toward levels equivalent to that for the district and the state. In particular this study provides evidence of the strong influence sustained science leadership at the building level has on building performance over time.”

Decision Catalyst, Inc. 2/6/03


2004 | BCAMSC Program Viability Study

The BCAMSC offers a high quality product at a competitive price. We interviewed more than 30 users and administrators of the kit. The following is a summary of those interviews:

BC kits were the only kits that align flawlessly with the State of Michigan standards for science.

    30 of 30 persons interviewed said this or something similar

Science needs to be taught with hands on experiments and the kits allow teachers to do this.

    30 of 30 persons interviewed said this or something similar

Cost was very competitive with other national kit providers and the product quality is very good.

    25 of 30 persons interviewed said this or something similar

BCAMSC has a distinct advantage in the state of Michigan. Not only do they offer a product that is highly competitive with other kit providers, but it is tied to the Michigan benchmarks. Grand Rapids Public Schools invited 6 competitors to give a presentation and BCAMSC was ranked first. We believe this independent evaluation by a major Michigan School district is representative of the attributes of the BCAMSC product.

table_1

In speaking with representatives from the school district the program was most attractive given:

    Inquiry based program
    Alignment with state of Michigan criteria
    Ease of use
    Potential for ongoing training and support

Plante Moran
CPA and Management Consultant Firm, 1/2004


2005 | Impact of the Science Kit Program on Student Achievement as Measured by the 5th Grade Michigan Educational Assessment Program (MEAP) Test.

“In 2004, the schools with the BCAMSC Kit-based programs are slightly above state average. This includes several urban schools with high minority populations, showing that the proportion of students in these low-performing schools doing well on the test increased. This can likely be attributed to the use of the kit-based curriculum and associated assistance, since it was the primary science program intervention in these schools.

The proportion of students in the lowest proficiency category has declined since 2002 for all districts. Among those districts using the Battle Creek science kit program, a slightly greater proportion of students have moved out of the lowest proficiency category compared with overall state scores.

In comparison of two urban districts, the district receiving kit program services has a higher proportion of students in the ‘met’ or ‘exceeded’ proficiency categories.”

Mark Jenness Ed.D
Director, Science and Mathematics Program Improvement (SAMPI)
Western Michigan University, 1/2005


2007 | Battle Creek Area Mathematics and Science Center Unit/Kit In-Service 2006-2007 Teacher PD Evaluation Data Analysis.

In the Fall of 2006, 332 teachers from 61 buildings participated in science kit workshops. At the beginning and conclusion of each workshop the participants were asked to respond to a seven-item assessment instrument. First they rated themselves on the items according to their level at the beginning of the workshop. Then they responded to the same items according to their current perception at the end of the workshop. This procedure of responding to the pre/post items during the same sitting may yield a measure of their perceptions of changes in themselves in their knowledge, understanding and familiarity rather than independent assessments at each time point. The participants’ average change scores appear to be greater than zero for each item at a high level of significance. The table below presents the averages for each item ordered according increasing “Before” average scores from left to right.

Overall Changes from Pre-program to Post-program

In general, the lower the initial rating, the greater the increase in the rating from “before” to “after,” so that all but one item was rated between 4.2 and 4.4 on the average at the end.

Survey Item Average Ratings Before and After the 2006-07 In-Service Workshop

table_2

Mark Jenness Ed.D
Director, Science and Mathematics Program Improvement (SAMPI)
Western Michigan University, 8/2007


2008 | Revision Process for the Science Grade Level Content Expectations and the BCAMSC Unit/Kits.

The Battle Creek Area Mathematics and Science Center is keeping pace with the changing Michigan Curriculum, best inquiry practices, and the integration of reading and writing in the content. In response to the No Child Left Behind Act and with the intent of providing a more rigorous, yet coherent, teacher friendly science curriculum, the Michigan Department of Education has adoped Grade Level Content Expectations for Science. It is the mission of the BCAMSC to respond to the changes and revisions in the science curriculum for the State of Michigan, providing school districts with the most current, state aligned resources.

The revision process for the current BCAMSC Unit/Kit program will involve a small committee of educators, writers, scientists, and curriculum specialists, a science content review, and a pilot program with data collection analysis. The revised Unit/Kits will not only align the current science units with the new Grade Level Content Expectations in Science, but also integrate the Grade Level Content Expectations for English Language Arts and Mathematics within the instructional component of inquiry-based science. The revision committee began in the Summer of 2007 and implementation begins in the Fall of 2008.

In comparison of two urban districts, the district receiving kit program services has a higher proportion of students in the ‘met’ or ‘exceeded’ proficiency categories.”

BCAMSC Outreach


2010 | The Effects of BCAMSC Science Kits on Student Achievement in Science

Executive Summary

The Effects of BCAMSC Science Kits on Student Achievement in Science
Evaluation & Measurement Specialist, Inc | May, 2010

Introduction
Previous investigators have assessed the quality of the units contained in the BCAMSC science kits by comparing their content to accepted instructional standards and gauging their alignment with Michigan’s grade level content expectations for science. However, there has been no investigation of the extent to which a district’s use of the kits actually helps to increase student achievement in science. As part of its effort to continually improve its products, BCAMSC retained an independent evaluator to conduct an analysis of student performance among kit users.

Evaluation Question
What evidence is there that faithful use of BCAMSC’s science kits results in increased student achievement in science?

Methods of Study
The basic procedure for the study was to compare MEAP science scores from 31 school districts that implemented the kits over the past three years with the scores of a comparison group of districts that have never used the kits. Prior to the analysis, the evaluator ensured that the two groups of districts were comparable demographically and academically in order to minimize any internal biases that could confound the results. MEAP data for the study was obtained from the Michigan Department of Education. The Mann-Whitney U Test was the inferential statistical test used to detect any differences in student scores.

Results
The results of the study conducted on the data for all students showed a statistically significant difference between the scores of districts that use the kits and the scores of the districts that do not use the kits (p < .001). When the scores of African American students were studied by themselves, there was also a statistically significant difference between the two groups (p = .04). There were no significant differences between the study group and the comparison group for any other demographic variable.

Conclusions and Discussion
It is clear that the results of this study provide support for the effectiveness of the kits; however, no scientific study is perfect. It is important to consider things other than the kits that could have influenced the observed differences in scores. For example, teachers in the study could have received training outside of BCAMSC that improved their science teaching skills. The MEAP science assessment may have some undiscovered bias that colored the results we observed. Lastly, the study had no way to control for differences in teacher quality, experience, education, or how faithfully they implemented the kit program. Despite these qualifications, the findings of the study provide compelling evidence that use of the science kits contributes to a significant increase in student achievement in science.

Battle Creek Area Mathematics and Science Center
765 Upton Avenue | Battle Creek, MI 49037


Supporting Data from MEAP

The Michigan Educational Assessment Program (MEAP) for Science is given annually to all public school 5th and 8th grade students in Michigan. This comprehensive exam utilizes constructed response and multiple-choice assessments. Scores are calculated on the basis of proficiency. A student is considered proficient if he or she obtains a satisfactory score in the five measures of scientific literacy as defined in the Michigan Standards. The score of the students receiving instruction from the BCAMSC Science Units were compared to the mean percentage of students deemed proficient throughout the State of Michigan, with over 180,000 fifth grade students taking the exam.


State Comparison: BCAMSC Full Implementation Results

table_3

“This Center (BCAMSC) is the developer and distributor of the statewide known, “Battle Creek Science Kits.” These kits contain in one package, all the materials needed to address an identified area of science content. They are aligned completely with the Michigan Curriculum Frame works for science and the National Science Education Standards and address all components, including inquiry, assessment, and literacy pieces. Teacher and student workbooks are included with every kit. The kits are continually being improved as new ideas or developments occur. The center has looked at statewide MEAP data to evaluate and modify kits where performance seemed lower than expected. It is always looking at cost effectiveness, keeping its kits at a price that districts can afford. These kits have more impact on Michigan science classroom practice than most any other product. This is a fine example of where one Center in the Math/ Science Network focusing on a specific aspect for the benefit of the other Centers.”

Kevin J. Richard
Michigan Department of Education


District Comparisons | Vandercook Lake Schools, Bedford Public Schools, and Bloomingdale Public Schools

MEAP proficiency comparison of urban, rural, and suburban districts with full implementation of BCAMSC Science Units and varying degrees of poverty:


State Proficiency | Vandercook Lake Schools

Demographics: Urban, K-12 Population: 1370, 32% Poverty

table_4

“This school went from 74% passing on the MEAP to 89% this year! The teachers are very excited about the improvement, and know that it’s because of the Battle Creek program.”

Theron Blakesly
Director, Jackson County Mathematics and Science Center (Aug. 2003)


State Proficiency | Bedford Public Schools

Demographics: Suburban, K-12 Population: 5500, 8% Poverty

table_5

“The Bedford Public Schools utilizes the BCAMSC Science kit program. The inquiry approach to science is highly effective and is supported by research and results.”

Debby Kuhl
Assistant Superintendent of Instruction and Student Services, Bedford Public Schools


State Proficiency | Bloomingdale Public Schools

Demographics: Rural, K-12 Population: 1675, 65% Poverty

table_6

“Bloomingdale Public Schools has seen significant improvement in its standardized achievement scores since the implementation of the Battle Creek Science Kits. As a principal and superintendent, I have been very pleased with the support provided by the Battle Creek Math and Science Center to ensure the hands-on approach is consistently aligned with curricula developed by the state.”

Mr. Brett Geie
Superintendent Bloomingdale Public Schools


Focus on “High Priority” Schools

The BCAMSC has targeted a group of underachieving schools for intensive interventions for the past five years. Prior to their interventions, the percent of students reaching proficiency levels was 51% of the state average. Percent proficient is currently at 90% of the state average.

Michigan Mathematics and Science Network Final Report, 2006


For further information regarding the evaluation and efficacy of the BCAMSC Science Units, please contact the BCAMSC Outreach office @ 269-965-9440.