Introduction
The mobiLLab science education program was developed by faculty at the University of Teacher Education in St.Gallen (Pädagogische Hochschule St. Gallen (PHSG)) to spark young people’s interest in science and technology (S&T). During the last 30 years, science centers and mobile laboratories have been developed in response to a decreasing interest in S&T careers among young people (Huber, 2014), something critical to our Digital Age society and workforce (Sjøberg & Schreiner, 2010). Since its inception in 2008, the mobiLLab program has provided secondary school pupils and their teachers in Eastern Switzerland with hands-on training in science experimentation using industry and laboratory equipments. The program also serves as on-the-job training for pre-service science and math teachers at the PHSG, who participate in several school visits as pupil coaches. For the first four years of operation, mobiLLab program staff had regularly received positive verbal feedback from participating teachers and pupils and had more requests from schools than they could visit. Even so, before making changes to the program, mobiLLab leaders wanted to elicit more specific, candid feedback from program participants that could inform program development.
Starting in 2012, mobiLLab team members worked with researchers to better understand the effectiveness of their program. This paper begins with a description of the mobiLLab program and provides detail about a typical visit. The next section describes how, through an exploratory background investigation, program priorities and indicators of effectiveness were identified (Cors, 2013). We then explain how, through a mixed-methods pilot study (Cors et al., 2015), researchers examined these indicators. We further describe findings about pupil and teacher satisfaction with the program; teacher ratings of classroom preparation materials offered on the mobiLLab website; pupil educational outcomes related to S&T interest; and which factors affect these educational outcomes. A final section discusses the advantages and limitations of the methods used, and offers recommendations for future research.
Background: A Program for Inspiring Tomorrow’s Science and Industry Workers
The mobiLLab program was developed to support the PHSG strategy to promote interest in S&T topics and careers among Switzerland’s youth. School visits are designed for pupils aged 14 to 16 who attend secondary school level 1 (German: Oberstufenschule). These pupils generally either pursue further vocational training or transfer into the university preparatory secondary school level II (German: Gymnasium).
A typical visit
A mobiLLab school visit begins at the PHSG building, where the deployment team loads the experimental equipment into a van, shown in Figure 1, and drives to the school. Typically, a class visit with mobiLLab lasts a half a day. The mobiLLab usually stays at a school for one or several days, during which it holds two class visits per day.
MobiLLab brings with it 12 experimental posts, listed in Table 1. During classroom prepration, teachers work with pupils to choose four experimental posts at which they will work and to prepare for the visit by reviewing the corresponding E-Learning tutorial for each post. The E-Learning tutorials are 10- to 13-minute video sequences that introduce the theory, equipment and sometimes applications for each experimental post. The last part of each E-Learning tutorial is a quiz consisting of about 10 questions. Most teachers print out worksheets (Journalblätter) from the mobiLLab website for pupil use. The worksheets include blank spaces for pupils to write down their own hypotheses and questions about the experiments before the visit as well as blank spaces for them to record the results of their experimentation.
A day for tinkering
The mobiLLab offers pupils an opportunity to work independently and in an evaluation-free environment, something thought to promote engagement in activities and interest development (Boekaerts & Minnaert, 1999; Ryan & Deci, 2000). In contrast to most classroom experiences, where pupils regularly encounter goals, deadlines, tests and other directives, teachers and the mobiLLab team present the mobiLLab visit as a day for trying things out and working in a self-directed manner. Pupils work in pairs, as shown in Figure 2 and, in addition to following step-by-step directions at each post, are encouraged to play around and ‘tinker’ with the equipment. Pupils are asked to try to deal with unexpected results on their own before turning to a mobiLLab coach. This independent problem-solving is supported by inquiry-based responses from mobiLLab coaches, who offer comments and questions (and no direct answers) to support pupils in exploring their own explanations for their observations. Pupils are also encouraged to bring items from home to test. At the Food Analysis Post, for example, pupils test the sugar content of soft drinks and homemade jam. Sometimes pupils bring tap or pond water to analyze via ion chromatography or metal objects to analyze with x-ray fluorescence.
A desire to move forward strategically
In 2011, the mobiLLab operations team made some changes in response to teacher comments. Specifically, the mobiLLab team developed two new items for each post: the E-Learning tutorials that pupils review online before the visit and the laminated step-by-step procedural guides (Kurzanleitung) for each post. Before making other changes to the program, the mobiLLab team wanted to evaluate the effectiveness of the program and to identify factors that promote this effectiveness. What did it mean to be effective? What factors influence mobiLLab’s effectiveness?
Phase I: A Background Investigation
A first step in the evaluation was to identify exactly what it meant for the mobiLLab program to be effective. Researchers worked with mobiLLab faculty and staff to organize a background investigation to explore the program’s priorities and identify measures of effectiveness.
Methods
The background investigation took place between October 2012 and April 2013. This discovery work took place in part during mobiLLab visits through observations of and informal conversational interviews with teachers and pupils. In addition, informal interviews conducted using an interview guide, were held in person or over the telephone with mobiLLab team members and representatives from similar programs worldwide. Interview guides were developed based on guidelines from Patton (2002) to conduct informal (not taped), non-structured, open-ended discussions. Interviewees received a list of questions before the interview, in order to encourage reflection and well-thought-out responses. Both informal conversational interviews and script-guided interviews were conducted in German or English, depending upon the preference of the in¬terviewee. Activities also included reviewing mobiLLab program materials, relevant economic trend reports for Switzerland and Europe, and relevant research studies. Figure 3 shows the scope of investigation activities.
This exploratory inquiry provided information that the mobiLLab team used to sketch a ‘logic model,’ or map, showing the logical relationships among the resources invested in the program, the activities that take place, and the benefits or changes that result from them. The mobiLLab team developed the logic model according to a process designed for educational program planning that was developed at the University of Wisconsin-Madison (Taylor-Powell et al., 2003). The logic model expresses the mobiLLab team’s theory of change, which illustrates how the program is supposed to work. Taylor-Powell described how the logic model helps groups use evaluation resources effectively by explicitly describing how program resources and activities are meant to be linked to desired outcomes:
”A logic model is the first step in evaluation. It helps determine when and what to evaluate so that evaluation resources are used effectively and efficiently. Through evaluation, we test and verify the reality of the program theory – how we believe the program will work. A logic model helps us focus on appropriate process and outcome measures” (p. 3).
Results: Articulating their situation
A first step in creating the logic model was for the mobiLLab team to define the situation, or the environment in which the program exists, which is a complex of sociopolitical, environmental, and economic conditions. An accurate understanding of the situation is a foundational part of logic model development in that it identifies forces driving the need for strategic planning and describes the people, resources, and activities related to program challenges. The mobiLLab team formulated the following situation statement:
“In spite of good science and math scores in secondary school (Eichenberger, 2010), young people in Switzer¬land, as in many other developed countries, show low interest in these subjects (Sjøberg & Schreiner, 2010). Moreover, too few young Swiss who show talent in science and math are completing univer¬sity degrees in these disci¬plines and they are choosing non-tech professions or professions outside of industry (MINT-Meter, 2012; Vogel-Misicka, 2012). This trend comes at a time when demand for science and technology graduates is growing, making it necessary for Switzerland to import high-tech and industry workers to remain competitive (High Level Group on Increasing Human Resources for Science and Technology in Europe, 2004; PresenceSwitzerland, 2012). To address the lack of “home-grown” industry and technology workers, mobile laboratory programs have started operating in countries including Germany and Switzerland.
By bringing laboratory experiments, scientists and science coaches into secondary school class-rooms in the German-speaking part of Switzerland, mobiLLab gives pupils an opportunity to expe¬rience inquiry-based science experimentation. Studies conducted in Europe and the US show that visits with mobile laboratories and science centers sometimes result in the development of pupils’ science interest attitude and knowledge immediately after a visit and that any changes tend to fade over a matter of one or two months (Barmby et al., 2005; Brandt et al., 2008; Dowell, 2011; Gassmann, 2012; Jarvis & Pell, 2005; Pawek, 2009).
Now in its fourth year of operation, the mobiLLab team would like to evaluate the program’s effec¬tiveness to inform further development. Specifically, we want to better understand how mobiLLab affects pupils’ science and technology interest, attitudes and knowledge development, and how positive changes can be sustained” (Cors, 2013, p. 4).
Results: The core program aim is to awaken youth interest in S&T
In the logic model, or theory of change shown in Figure 4, outcomes and assumptions that the mobiLLab team believed to be most influential to program success are shown in bold. The logic model shows, for example, how classroom preparation, shown as an Action Outcome, is important for helping the pupils know what to expect, reducing anxiety and promoting curiosity about the mobiLLab visit, a Learning Outcome. This enables pupils to better engage in activities at experimental posts, which causes them to ‘become more technophillic,’ a Condition Outcome. Becoming more technophillic is an important step towards maintaining interest in S&T, which the mobiLLab team believes makes it more likely that pupils will later choose related careers, another Condition Outcome.
Through this logic modeling process, mobiLLab team members confirmed that awakening pupils’ interest in doing science with technology, or pupils becoming more technophillic, is the core goal of the program. During background investigation interviews, stakeholders including teachers, industry representatives who visited classes during mobiLLab events, and mobiLLab team members, explained that awakening pupils’ interest involves promoting multiple views of the relevance of S&T in their lives. They spoke about fostering development of pupils’ basic interest in S&T; their awareness of it in the world around them; how it is useful in society; and their comfort level with it. Similarly, studies often couple measures of science interest with measures of attitude and self-concept of ability related to S&T (Denissen et al., 2007; Dowell, 2011; Dresel & Lämmle, 2011; Potvin & Abdelkrim, 2014). Even though some elements of the logic model were somewhat roughly expressed, it provided a centerpiece for discussion among mobiLLab staff and researchers that informed identification of program effectiveness measures.
Results: Indicators of program effectiveness
Based on several logic model elements, the mobiLLab team identified a list of indicators of program effectiveness, which would be practical to measure during a research investigation. One indicator, participant satisfaction, is reflected in several logic model outcomes, such as ‘pupils feel engaged…’ and ‘teachers continue to request mobiLLab.’ A second indicator, usefulness of classroom preparation materials, comes from the logic model assumption that the level of the E-Learning tutorials is appropriate for pupil learning and from the Learning Outcome that ‘teachers learn how to prepare for the mobiLLab visit.’ A final indicator was change in pupils’ affective educational outcomes, to be measured as S&T interest, attitude and self-concept of ability. The literature review conducted in conjunction with the background investigation helped researchers identify existing instruments that could be adapted for use in the mobiLLab pilot study to measure these aspects of program effectiveness. The specific instruments that were used to measure each indicator, along with the source for each, are listed in Table 2.
Results: Factors thought to affect pupils S&T interest development
Drawing on results from the logic model and the literature review, the mobiLLab research-faculty team identified factors they thought had the greatest influence on pupils’ affective educational outcomes. These factors included pre-visit classroom preparation activities; pupils’ feelings of novelty, or unfamiliarity; and teachers’ attitudes about learning approaches. The process for identifying these factors is shown in Figure 5 and each factor is described below.
Classroom preparation. Classroom preparation was seen as critical to program effectiveness. The mobiLLab team’s hypothesis, shown in the logic model, was that a more complete classroom preparation would better help pupils know what to expect at the visit and therefore support their engagement in visit activities, which would, in turn, improve their S&T interest. Similarly, several studies of informal science learning programs provide evidence for a link between a more complete classroom preparation and development of educational outcomes, which sometimes related to pupils’ exploratory behavior at the program visit (Anderson & Lucas, 1997; Cotton & Cotton, 2009; Jarvis & Pell, 2005; Kubota & Olstad, 1991; Orion & Hofstein, 1994).
Novelty. The mobiLLab theory of change emphasizes classroom preparation because it increases pupils’ familiarity with the schedule and objects they will encounter at the visit, which should lower their anxiety and heighten their curiosity about the visit. This improved familiarity should, in turn, enable pupils to better engage in at-visit activities, which should promote development of more positive interest in S&T. Evidence for such a link between increased familiarity, or reduced novelty, and the effectiveness of informal science education program has been produced by several studies. These studies indicate that pupils’ individual novelty factors, such as relevant content knowledge or familiarity with the informal learning setting, related significantly to more on-task behavior at the visit and to the development of more positive educational outcomes (Anderson & Lucas, 1997; Falk & Balling, 1982; Falk et al., 1978; Jarvis & Pell, 2005). Based on this, mobiLLab investigators identified three novelty impact factors thought to most influence how novel pupils found the mobiLLab experience: 1) a cognitive factor, measured as pupils’ grades, 2) a setting orientation factor, measured as frequency of pupils visits to informal learning venues, such as museums and science centers; and 3) a technological capability factor that reflects whether pupils’ explore and tinker, or to seek direction and support, when interacting with technology. As already mentioned, the first two factors have been examined in previous studies of informal science learning. The capabilities impact factor became part of the research design in response to interviews and conversations with mobiLLab program faculty and staff during and after the background investigation. They explained that they wanted to have a better understand how pupils feel about working with technology. The technological capability construct was chosen because it is an indicator of how capable people feel interacting with technology. It was developed as part of the Technological Profile Inventory (TPI), which supports a South African university admissions process that sought to admit engineering students with the best chance of success (Luckay & Collier-Reed, 2011).
Teacher Attitude. As an assumption in the mobiLLab logic model shows, mobiLLab team members also thought that teacher attitude influenced pupils’ interest. The great influence of teacher attitude on what pupils gain from an informal learning experience was also emphasized in interview responses from leaders of similar programs worldwide (Cors, 2013). Teacher attitude and teaching approach as a key to improving educational outcomes is not a new idea and, as early as the 1960s, the United National Educational, Scientific, and Cultural Organization recognized the “essential role of teachers in educational advancement and the importance of their contribution to the development of man and modern society” (ILO & UNESCO, 1966, p. 20). Based on a previous study of how teacher attitudes are linked to how pupils learn physics, investigators selected two factors to examine: teachers’ attitude to situational learning and to constructivist learning (Kuhn, 2010).
Phase II – Pilot Study
A mixed-methods pilot research investigation was developed to examine how classroom preparation, pupil novelty factors, and teacher attitude related to pupils’ affective educational outcomes. Affective educational outcomes, called ‘core S&T outcomes,’ were measured as interest in, attitude to, and self-concept to S&T. The investigation was designed to explore the questions, ‘How do differences in classroom preparation and in pupils’ novelty factors relate to changes pupils’ core S&T educational outcomes from before to after a mobiLLab visit?’ and ‘What moderating role do teachers’ attitudes play?’ The study also presented an opportunity to examine measures of program effectiveness.
Methods
The mobiLLab pilot study took place in Spring 2014. Data collection involved 9 teachers and 15 of their class groups who experienced a mobiLLab visit. All 9 teachers and 208 pupils completed pre- and post-visit surveys, which occurred in January and one week after their mobiLLab visit, which occurred from February to May, respectively. Teachers also participated in post-visit interviews.
Pupil survey. Pupils responded to survey items about their core S&T outcomes, their individual novelty factors, and opinions about the mobiLLab program. They rated these items using a 4-point Likert scale: “1”=completely untrue (“stimmt gar nicht”), “2” = somewhat true (“stimmt wenig”), “3” = very/quite true (“stimmt sehr”), “4” completely true (“stimmt völlig”). Examples of survey items are shown in Table 3. All of these items were borrowed from other pupil surveys and were adapted to the mobiLLab pilot surveys through a review process that involved mobiLLab program leaders, to ensure that the language would be appropriate for participating pupils. A group of eight testers, including four teens who attended school in the same provincial areas as mobiLLab pupils, completed a draft of the survey and provided feedback to improve understandability.
Paired t-tests were employed to assess whether pupils’ interest, attitude and self-concept regarding both science and technology changed significantly between pre- and post- visit surveys (when p<0.05). For significant changes, Cohen’s d was calculated to indicate the magnitude of the change, called effect size, which can be interpreted based on guidelines from Cohen (1998): small d=0.2, medium d=0.5, large d=0.8. Relations between impact factors, such as technological capability, and educational outcomes were explored through multivariate analysis of regression (MANOVA). Results are reported as F values, a comparison of group means for tinkers and direction seekers. For significant relations, an effect size is given as partial eta squared, Ƞp2, which can be interpreted from guidelines from Cohen (1988): small Ƞp2=0.01, medium Ƞp2=0.06, large Ƞp2=0.14.
Teacher interviews and survey. Teacher interviews took place at schools where teachers worked and lasted 30 to 40 minutes. The aims were to characterize classroom preparation and better understand teachers’ experiences with the mobiLLab program. Interviews were developed and conducted according to guidelines from Patton (2002) in a semi-structured manner. This involved following a scripted list of questions and sometimes diverging from the script when opportunities arose to talk with teachers about suggestions for program improvement. It was clear beforehand that there would not be enough time during the interviews for teachers to comment on each of the classroom preparation resources available on the mobiLLab website. In anticipation of the limited time, the interviewer (first author) asked teachers about the four resources thought to be most used by teachers: the introduction to mobiLLab PowerPoint presentation (Einführung ins mobiLLab), E-Learning, the worksheets (Journalblätter), and the step-by-step instructions for working at each post (Kurzanleitung). If other resources were discussed, these conversations were generally initiated by the teacher.
Through an online survey, teachers responded to questions about their preparation and rated four of the materials thought to be most frequently downloaded from the mobiLLab website and used for classroom preparation. These materials were the introduction to mobiLLab PowerPoint presentation (Einführung ins mobiLLab), E-Learning, the worksheets (Journalblätter), and the step-by-step instructions for working at each post (Kurzanleitung). Teachers rated these website materials based on four criteria: appropriate level and language for pupils; clarity and understandability for pupils; whether the material was edited by the teacher prior to use in class; and appeal to pupils. They rated the materials using a four-point Likert scale to indicate for example, how clear and understandable each item was: 1= not at all (stimmt gar nicht), 2 = somewhat (stimmt wenig), 3= quite (stimmt ziemlich), 4= completely (stimmt völlig).
Teacher sample and intervention
It was expected that survey responses from teachers about classroom preparation would differ between treatment teachers, who received additional preparation materials, and control teachers, who received no additional preparation materials. However, teachers’ accounts of their preparation did not vary significantly for most factors, such as which mobiLLab website materials they used during classroom preparation and their attitudes toward situated learning. In fact, classroom preparation time was the only aspect from which a preparation typology could be created. Four preparation types, shown in Figure 6, were defined based on duration and number of classroom lesson-hours (45 minutes each) devoted to preparation.
One explanation for the low variability of materials used by teachers for classroom preparation could be the small sample of (9) teachers. An even more likely reason was that, even though the mobiLLab manager shared new preparation materials with only five of the teachers (treatment group), other teachers (control group) sometimes gained access to these same resources. This sample ‘contamination’ is illustrated in Table 4, which shows how control group teachers used most of the new preparation materials, likely acquired from colleagues in the treatment group who worked at the same school. All new resources except for the novelty space triangle were used by at least one control group teacher. Because of the popularity of some of the new resources with teachers, such as the Planning Guide, they became a permanent part of website materials the mobiLLab program offers for classroom preparation.
Pupil sample
Responses from pupils (108 male; 97 female; 3 no response) about core S&T outcomes from the pre- and post-visit survey are shown in Figure 7. Pupil ratings of technology-related core S&T outcomes showed slight or insignificant changes. That is, pupils’ interest in technology was moderate and decreased significantly from pre- to post-survey, with small effect, (M=2.55->2.43, p<.001, Cohen’s d=0.18). In contrast, their attitude was somewhat positive and showed no significant change (M=3.04->3.07; p=.284), and their somewhat positive self-concept decreased significantly, with small effect, (M=2.86->2.80, p=.006, Cohen’s d=0.10). Responses about natural science were similar: pupils indicated a moderate interest in natural science that decreased significantly, with small effect, (M=2.52->2.44, p=.005, Cohen’s d=0.13); a somewhat positive attitude that showed no significant change (M=2.94->2.97; p=.384), and a somewhat positive self-concept that decreased significantly, with small effect, (M=2.87->2.82, p=.046, Cohen’s d=0.09). These results reflect the collective results from other studies of science learning at mobile laboratories and science centers, which show that pupils interest sometimes decreases and sometimes increases, and that these changes often fade over time (Barmby et al., 2005; Brandt et al., 2008; Dowell, 2011; Guderian, 2007; Jarvis & Pell, 2005; Pawek, 2009; Sasson, 2014).
Results: program satisfaction and classroom preparation materials
This section begins with a description of findings from an improved version of the pupil pre- and post- surveys (N=215) completed in 2015. Qualitative data from the pilot study provides further insights into participants’ program satisfaction. Also described are teachers’ ratings about the usefulness of frequently used preparation materials.
Program satisfaction
Pupils’ program satisfaction was measured through a grade they gave the mobiLLab program and through several question about how they liked the mobiLLab visit. Pupils gave their mobiLLab experience an average grade of 4.8 (SD=0.9), which is more than a full grade higher than the grade they gave the mobiLLab in 2010 (M=3.2, SD=1.5), as shown in Figure 8. MobiLLab team members attributed this improved grade from mobiLLab pupils largely to the addition of the E-Learning and step-by-step procedural guides (Kurzanleitung) for each post in 2011. MobiLLab team leaders also point to other factors that could have contributed to this increased program satisfaction: there are more teachers each year who have worked with mobiLLab in a previous year and can therefore better support pupils to prepare for the visit; new posts, such as Food Analysis, are easier to operate and can test more items from home; and recent additions of new objects, such as a prism to the Visible Light post, which appear to be popular with pupils.
Pupils’ responses to three questions about their satisfaction, shown in Figure 9, indicate that their mobiLLab experiences were fairly positive. On average, pupils gave a positive rating for liking the visit (M=3.1, SD=0.8). Responses about whether they would like to participate in another mobiLLab visit were slightly better than neutral (M=2.7, SD=1.0). Finally, pupils thought they had to work at least as hard during the mobiLLab visit as they usually do during science class (M=2.4, SD=0.8). These findings suggest that pupils liked their mobiLLab experience, even though it involved some work. This supports the assertion by researchers that informal learning is more than just play, offering an environment where learners work but also enjoy themselves (Rennie, 2007).
During interviews, several teachers explained that during classroom discussions after the mobiLLab visit, pupils also voiced positive feedback about the program. Teachers themselves also indicated that they were satisfied with the program, with about half of teachers expressing (unsolicited) interest in another mobiLLab visit. Teachers said mobiLLab is valuable to them because it offers pupils a chance to work with equipment and materials the schools do not have and because pupils can develop and implement their own ideas.
Results: Teacher ratings of classroom preparation materials
Teacher responses about classroom materials, shown in Figure 10, were relatively encouraging. They found the Introduction to mobiLLab to be at a good content level (M= 3.4, SD=0.7), clear and understandable (M= 3.4, SD=0.7), interesting and exciting for pupils (M= 3.2, SD=0.7), and did not need much changing before use (M= 1.6, SD=1.1). Similarly, teachers rated E-Learning online tutorials as having a good content level (M= 3.3, SD=0.9), as clear and understandable (M= 3.1, SD=1.0), interesting and exciting for pupils (M= 3.4 SD=0.7), but indicated they could use some modification (M= 2.2, SD=1.0). The post step-by-step instructions received ratings that were almost as good, with teachers indicating a good content level (M= 2.9, SD=0.9); clarity and understandability (M= 2.9, SD=0.7); that they were interesting and exciting for pupils (M= 3.0, SD=0.6); but that they could use some adjustment before use (M= 2.1, SD=1.2). The Journalblätter worksheets received more moderate reviews. Teachers indicated that they had a good content level (M= 3.1, SD=1.1); were clear and understandable (M= 2.9, SD=1.1); were somewhat interesting and exciting for pupils (M= 2.7, SD=0.5); but indicated they needed some adjustment before use (M= 2.3, SD=1.2). During interviews, teachers offered specific suggestions about modifying the Journalblätter worksheets that led to a major shortening and revision of this resource. Also in response to specific suggestions offered during interviews, the mobiLLab team managers revised several other online resources, reorganized the website, and added some information to teachers’ orientation materials.
Results: Factors that affect pupils’ educational outcomes
Results of a multivariate analysis of regression (MANOVA) indicated that two factors affected how pupils’ core S&T outcomes changed from before to after the mobiLLab visit. These two factors were preparation time and pupils’ perception of their technologically capability.
Preparation time. Preparation time had an overall small effect on core S&T outcomes, ɳp2 =.03. A closer look through post-hoc tests suggest that pupils who experienced a preparation that started closer to the mobiLLab visit and involved more classroom time (’duration short, lesson time high’), showed significantly greater interest (science, p=.049; technology, p=.012) and had a more positive attitude (science, p=.011; technology, p=.010). These results could suggest that when preparation starts too early, pupils have difficulty recalling preparation lessons and feel unprepared for the visit. Also, more classroom time may simply give pupils more opportunity to become familiar with relevant content, equipment, and the schedule for the visit. Qualitative data provide insights into how prepration time can be used most effectively. That is, during pilot study interviews, teachers emphasized that an effective preparation 1) involves pupils reviewing the E-Learning tutorials; 2) relates mobiLLab to pupils’ interest; 3) relates classroom activities and assignments to mobiLLab; and 4) orients pupils to the plan for the day.
NOTE: Data about teacher pre-visit attitudes about the importance of situated (M=2.9, SD=.06) and constructivist (M=2.8, SD=.04) learning did not change significantly from before to after the mobiLLab visit. Also, teacher attitudes were not found to be significant moderators of the relation between impact factors and pupils’ core S&T outcomes.
Comfort with technology. Pilot study findings suggest a link between pupils’ comfort with mobiLLab equipment and engagement with mobiLLab visit activities. That is, during interviews, several teachers talked about how pupils’ engagement with mobiLLab activities depended upon them becoming comfortable with the idea of handling equipment, without for example breaking something.
Quantitative data also suggest that pupils’ comfort interacting with technology is linked to their program experience. Specifically, findings showed that more technologically capable pupils (tinkerers), reported significantly different changes in their core S&T outcomes, from before to after the mobiLLab visit, than direction seekers (medium effect: ɳp2 =.05). Follow-up ANOVA tests for individual outcomes revealed significant relations between technological capability and changes in pupils’ interest in and self-concept to technology. That is, pupils’ technological capability accounted for differences in how their interest in technology changed from before to after the mobiLLab visit with small effect, F(1,199)=5.69, (p=.018), Ƞp2=.028. As illustrated in Figure 11, tinkerers had more positive interest in technology than their direction-seeking peers. However, tinkerers’ interest in technology decreased slightly from pre- to post-visit, while direction-seekers’ interest remained unchanged. This could mean that tinkerers were bored or that somehow their expectations for the visit were not met. These results could also reflect a trend of decreased interest in science with age, a phenomenon identified in other studies of similar programs (Barmby et al., 2005; Guderian, 2007). This interpretation is based on the fact that there was a timespan of five to twenty weeks between pre- and post-visit surveys, depending upon when the mobiLLab visit for a given class took place.
ANOVA results also show that pupils’ technologically capability accounted for small but significant differences in changes in pupils’ self-concept to technology, F(1,199)=3.90, (p=.050), Ƞp2=.019. This effect is illustrated in Figure 12, which shows how tinkerers started with greater self-concept to technology than direction-seekers. However, direction-seekers’ self-concept decreased significantly more than than tinkerers’ self-concept to technology, which remained virtually unchanged. One could imagine that direction seekers felt less comfortable with the equipment at the mobiLLab visit and/or perhaps overwhelmed by the challenge of using it, which caused them to feel frustrated and therefore disengage with the activity.
There was no significant relationship between pupils’ technological capability and changes in their attitude to technology, F(1,199)=.147, (p=.702). Also technological capability did not account for differences in changes in pupils’ natural science educational outcomes F(3,195)=.41, (p=.746). Likewise, pupils’ science grades, math grades, and how often they visited other informal learning programs were not significantly linked to how their core S&T outcomes changed.
Summary and outlook
The mobiLLab team sought to identify and measure indicators of program effectiveness that were meaningful for their work in the field. Through an exploratory background investigation, indicators of program effectiveness were identified: participant satisfaction, usefulness of mobiLLab website materials for classroom preparation, and changes in pupils’ S&T interest, attitude and self-concept from pre- to post-visit. These indicators were examined through pupil and teacher surveys and teacher interviews during a pilot research investigation. Findings about program satisfaction produced encouraging results, indicating that pupils and teachers were satisfied overall with their mobiLLab program experience and that pupil satisfaction improved over time. Results also showed that teachers are generally pleased with classroom preparation materials provided on the mobiLLab website and offered useful input for improving these resources.
Like results from other studies about informal science education programs, our results showed that pupil’ S&T interest, attitude and self-concept changed slightly or not at all significantly from before to after the visit. Fortunately, the investigation also explored factors that influence these outcomes. Results show that classroom preparations that began less than 15 days before the mobiLLab visit and lasted longer than eight lesson hours were linked to more positive pupil S&T interest and self-concept. This offers evidence that the time invested by the mobiLLab team in developing preparation materials and by teachers in conducting classroom preparation activities is worthwhile.
A second factor that predicted how pupils’ S&T outcomes changed from before to after the mobiLLab visit was their comfort interacting with technology. That is, findings provide evidence that interest in and self-concept to technology for tinkerers, or pupils who see themselves as more technologically capable, changed differently than their direction-seeking peers. The nature of these differences was unexpected. It was perhaps no surprise that tinkerers had greater S&T interest, attitude, and self-concept than direction seekers. However, tinkerers’ interest in technology decreased slightly, while direction seekers interest remained unchanged. In contrast, tinkerers’ self-concept of ability with technology remained steady, while direction seekers developed a slightly lower self-concept of ability with technology. Qualitative data also point to how pupils’ comfort with technology affects their mobiLLab experience. That is, during pilot study interviews, teachers asserted that the more comfortable pupils feel with the mobiLLab equipment, the better they engage in activities and profit from the visit. Through the lens of novelty, this could mean that the mobiLLab needs to offer more appealing novelty, such as a more authentic laboratory environment with lab coats and clipboards, to attract pupils’ interest. Perhaps additional ‘whacky challenges’ need to be added to maintain the interest of tinkerers and keep them from being bored. In contrast, for direction-seekers, mobiLLab could try to reduce unfamiliarity by offering more opportunities for pre-visit practice with equipment and/or some simple tasks at the mobiLLab visit with which they can succeed. These activities could reduce how overwhelmed and intimidated direction-seekers feel and boost their self-concept. Future studies should examine how these approaches affect learners’ S&T interest and self-concept of ability.
The mobiLLab pilot study was designed based on the idea that classroom preparation reduces unfamiliarity and promotes at-visit engagement, which, in turn, promotes the development of S&T interest. This link between learners’ novelty factors, at-visit experience, and their educational outcomes has been put forth by several models for informal learning research, yet few studies have measured learners’ at-visit experiences. Future studies should examine how individual novelty factors, such as technological capability, relate to how learners perceive novelty during a visit, measured through indicators such as exploratory behavior, oriented feeling, cognitive load and curiosity feeling. Such studies would demonstrate, for example, whether a novelty-reducing preparation indeed improves how oriented pupils feel at a science center visit. By examining relations among learner novelty impact factors, at-visit novelty factors, and educational outcomes, studies can deepen our understanding of the role of novelty in informal learning.
The investigation process and findings offer a model that can inform other informal learning programs about evaluating their own programs. The investigation followed many of the criteria that are part of quality research to evaluate science education (Bennett et al., 2006). For example, by exploring mobiLLab program priorities and goals with team members and other program stakeholders, researchers developed measures of program effectiveness that represent the real-world challenges of an informal science education program. And by involving mobiLLab team members and local youth in developing and testing measurement instruments for the pupil survey, they felt relatively confident that they collected responses from pupils that match what they aimed to measure. Collecting teacher responses through both interviews and an on-line survey is a form of data triangulation that contributed to the validity of some variables. Moreover, data was collected both before and after the mobiLLab visit and the pilot study involved a moderately large pupil sample.
Investigators also learned about the challenges of studying informal learning programs, including some factors that commonly limit such studies, which must often organize research activities to conform with classroom pupil groupings and course schedules (Bell et al., 2009; Brownell et al., 2013). For example, the pilot study did not include a control or comparison group, so conclusions cannot be made about whether educational outcomes from a mobiLLab visit are different from outcomes that result from classroom learning. Also, pupil and teacher samples were not chosen randomly, but consisted of those classes whose teachers made the extra effort to request a mobiLLab visit. It is also worth noting that the one-time, transient nature of informal learning programs like mobiLLab introduces questions about whether findings are the result of the program experience or of other factors in learners’ lives. That is, during the pre- and post-visit surveys, pupils could have encountered other classroom lessons or out-of-school experiences that influenced their science interest, attitude and self-concept.
A final note is about the high-technology nature of some informal learning programs, which reflects how technology has become more prevalent in our lives. The mobiLLab evaluation reveals that pupils’ interest in and self-concept to technology are different from how they view science, depending upon their perception of how technologically capable they are. By recognizing these links, studies about mobile laboratories and science centers can help us better understand how we are preparing young people for life in the 21st century, much of which is high-tech. How technologically capable pupils see themselves as has been recognized as an important skill for thriving and contributing to Digital Age societies. The National Academy of Engineering (NAE) and US National Research Council (NRC) described technological capability as one of three dimensions of technological literacy, which consists of dimensions of capabilities, knowledge, and critical thinking. They explain how technological literacy has become a critical aspect of how people function in and support today’s economy and society (Garmire & Pearson, 2006):
“There are a number {of benefits of technological literacy} … some of the most important relate to improving how people—from consumers to policy makers— think and make decisions about technology; increasing citizen participation in discussion of technological developments; supporting a modern workforce, which requires workers with significant technological savvy; and ensuring equal opportunity in such areas as education and employment for people with differing social, cultural, educational, and work backgrounds” (p. 22).
Authors
Rebecca Cors*, University of Teacher Education, Institute for Teaching Natural Science, Switzerland, rebecca.cors(at)phsg.ch
Nicolas Robin, University of Teacher Education, Institute for Teaching Natural Science, Switzerland, nicolas.robin(at)phsg.ch*corresponding author
[vc_tta_accordion active_section=”0″ no_fill=”true” el_class=”lahteet”][vc_tta_section title=”References” tab_id=”1458134585005-b3f22396-5506″]
Anderson, D., & Lucas, K. (1997). The Effectiveness of Orienting Students to the Physical Features of a Science Museum Prior to Visitation. Research in Science Education, 27(4), 485-495.
Barmby, P., Kind, P. M., Jones, K., & Bush, N. (2005). Evaluation of Lab in a Lorry (Vol. Final Report): CEM Center and School of Education, Durham University.
Bell, P., Lewenstein, B., Shouse, A. W., & Feder, M. A. (2009). Learning Science in Informal Environments: People, Places, and Pursuits. PDF is available at: http://www.nap.edu/catalog/12190.html: National Academies Press.
Bennett, J., Lubben, F., & Hogarth, S. (2006). Bringing Science to Life: A Synthesis of the Research Evidence on the Effects of Context-Based and STS Approaches to Science Teaching. Wiley Interscience. Retrieved from doi:10.1002/sce.20186
Boekaerts, M., & Minnaert, A. (1999). Self-regulation with respect to informal learning. International Journal of Educational Research, 31, 533-544.
Branch, R., Kim, D., & Koenecke, L. (1999). Evaluating Online Educational Materials For Use In Instruction. Retrieved from http://www.kidsource.com/index.html
Brandt, A., Möller, J., & Kohse-Höinghaus, K. (2008). Was bewirken außerschulische Experimentierlabors? Ein Kontrollgruppenexperiment mit Follow up- Erhebung zu Effekten auf Selbstkonzept und Interesse (What’s the Effect of Science Laboratories? A Control Group Experiment with Follow-up Data on Self-Concept and Interest). Zeitschrift für Pädagogische Psychologie, 22(1), 5-12.
Brownell, S. E., Kloser, M. J., Fukami, T., & Shavelson, S. (2013). Context Matters: Volunteer Bias, Small Sample Size, and the Value of Comparison Groups in the Assessment of Research-Based Undergraduate Introductory Biology Lab Courses. Journal of Microbiology Biology Education, 14(2), 176–182.
Cohen, J. W. (Ed.). (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
Cors, R. (2013). MobiLLab Program Background Investigation: Directions for Program Improvement and Evaluation Research (Verschaffen eines Überblicks des mobiLLabs) (http://phsg.contentdm.oclc.org/cdm/ref/collection/p15782coll3/id/94, Trans.) (pp. 32). St. Gallen, Switzerland: University of Teacher Education (Pädagogische Hochschule St. Gallen).
Cors, R., Müller, A., & Robin, N. (2015). Advancing Informal MINT Learning: Preparation and Novelty at a Mobile Laboratory. New Perspectives in Science Education, 53-58.
Cotton, D. R. E., & Cotton, P. (2009). Field biology experiences of undergraduate students: the impact of novelty space. Journal of Biology Education, 43(4), 169-174.
Denissen, J., Zarrett, N., & Eccles, J. (2007). I Like to Do It, I’m Able, and I Know I Am: Longitudinal Couplings Between Domain-Specific Achievement, Self-Concept, and Interest Child Development, 78(2), 430-447.
Dowell, K. (2011). 2010-2011 Evaluation of the MdBioLab Program. In EvalSolutions (Ed.), (pp. 33): Prepared for: MdBio Foundation, Rockville, Maryland.
Dresel, M., & Lämmle, L. (2011). Motivation. In T. Goetz (Ed.), Emotion, Motivation, und selbsreguliertes Lernen. Schöning/UTB: Paderborn.
Eichenberger, I. (2010, December 8). Swiss students get an encouraging report, Swissinfo.
Engel, K. (2004). Schülerlabors: authentische, aktivierende Lernumgebungen als Möglichkeit, Interesse an Naturwissenschaften und Technik zu wecken. (Disseration), Christian-Albrechts-Universität, Kiel, Germany.
Falk, J. H., & Balling, J. D. (1982). The Field Trip Milieu: Learning and Behavior as a Function of Contextual Events. Journal of Educational Research, 76(1), 22-28.
Falk, J. H., Martin, W. M., & Balling, J. D. (1978). The Novel Field-Trip Phenomenon: Adjustment to Novel Settings Interferes with Task Learning. Journal of Research in Science Teaching, 15(2), 127-134.
Garmire, E. M., & Pearson, G. (2006). Tech tally: Approaches to assessing technological literacy. Washington, DC: National Academies Press.
Gassmann, F. (2012). Das Schülerlabor iLab des Paul Scherrer Instituts (pp. 2). Switzerland: Paul Scherrer Institute.
Guderian, P. (2007). Wirksamkeitsanalyse außerschulischer Lernorte – der Einfluss mehrmaliger Besuche eines Schülerlabors auf die Entwicklung des Interesses an Physik. (Disseration), Mathematisch-Naturwissenschaftliche Fakultät I. Retrieved from http://edoc.hu-berlin.de/docviews/abstract.php?id=27927
urn:nbn:de:kobv:11-10077545
High Level Group on Increasing Human Resources for Science and Technology in Europe. (2004). Europe Needs More Scientists. Retrieved on 9. January 2013 from http://ec.europa.eu/research/ conferences/2004/sciprof/pdf/final_en.pdf: European Commission.
Huber, B. (2014). Swiss Barometer of Young Talent in STEM subjects (MINT-Nachwuchsbarometer Schweiz): Swiss Academy of Engineering Sciences.
ILO, & UNESCO. (1966). Recommendation Concerning the Status of Teachers. Retrieved from http://www.ilo.org/public/english/dialogue/sector/techmeet/ceart/rec66i.htm
Jarvis, T., & Pell, A. (2005). Factors Influencing Elementary School Children’s Attitudes toward Science before, during, and after a Visit to the UK National Space Centre. Journal of Research in Science Teaching, 42(1), 53-83.
Kubota, C. A., & Olstad, R. G. (1991). Effects of Novelty-Reducing Preparation on Exploratory Behavior and Cognitive Learning in a Science Museum Setting. Journal of Research in Science Teaching, 28(3), 225–234.
Kuhn, J. (2010). Authentische Aufgaben im theoretischen Rahmen von Instruktions- und Lehr-Lern-Forschung: Effektivität und Optimierung von Ankermedien für eine neue Aufgabenkultur im Physikunterricht (Habilitationsschrift ed.). Wiesbaden: Vieweg &Teubner.
Luckay, M., & Collier-Reed, B. (2011, 10-12 August). Admitting Engineering Students with the Best Chance of Success- Technological Literacy and the Technological Profile Inventory (TPI). Paper presented at the 1st Biennial Conference of the South African Society for Engineering, Sellenbosch.
MINT-Meter. (2012, May 23). Neuer Höchststand der MINT-Fachkräftelücke.
OECD. (2006). Test PISA 2006 Schweiz und Liechtenstein; Deutsch: OECD Programme for International Student Assessment:.
Orion, N., & Hofstein, A. (1994). Factors that Influence Learning during a Scientific Field Trip in a Natural Environment. Journal of Research in Science Teaching, 31(10), 1097-1119.
Patton, M. Q. (2002). Qualitative Research and Evaluation Methods (3rd Edition ed.). Thousand Oaks, California: Sage Publications.
Pawek, C. (2009). Schülerlabore als interessefördernde ausserschulische Lernumgebuhngen für Schülerinnen und Schüler aus der Mittel- und Oberstufe. (Dissertation), Christian-Albrechts-Universität, Kiel, Germany.
Potvin, P., & Abdelkrim, H. (2014). Interest, Motivation and Attitude Towards Science and Technology at K-12 Levels: a Systematic Review of 12 Years of Educational Research. Studies in Science Education, 50(1), 85–129.
PresenceSwitzerland. (2012). Foreign workers. swissworld.org: Federal Department of Foreign Affairs, Switzerland.
Rennie, L. J. (1994). Measuring Affective Outcomes from a Visit to a Science Education Centre. Research in Science Education, 24, 261-269.
Rennie, L. J. (2007). Learning Science Outside of School. In S. K. Abell, N.G. Lederman (Ed.), Handbook of Research on Science Education (pp. 125-167). Mahwah, New Jersey: Lawrence Erlbaum Associates.
Ryan, R., & Deci, E. (2000). Self-Determination Theory and the Facilitation of Intrinsic Motivation, Social Development, and Well-Being. American Psychologist, 55(1), 68-78.
Sasson, I. (2014). The Role of Informal Science Centers in Science Education: Attitudes, Skills, and Self-efficacy. Journal of Technology and Science Education, 4(3).
Sjøberg, S., & Schreiner, C. (2010). The ROSE project: An overview and key findings (pp. 31): University of Oslo.
Taylor-Powell, E., Jones, L., & Henert, E. (2003). Enhancing Program Performance with Logic Models. Retrieved January 7, 2013, from the University of Wisconsin-Extension web site: http://www.uwex.edu/ces/lmcourse/.
Vogel-Misicka, S. (2012, October 7). Switzerland faces a lack of IT professionals. swissinfo.ch.
Willig, C. (2013). Introducing Qualitative Research in Psychology (Third Edition ed.). Berkshire, England: Open Universtiy Press.
[/vc_tta_section][/vc_tta_accordion]
