METHODS

This study was part of a comprehensive evaluation of the BYSC that employed mixed methods to evaluate various aspects of the project.⁴⁷ It was reviewed and approved by the Institutional Review Board for research ethics at our university. Our evaluation approach aligned with the notion of utilitarian pragmatism⁴⁸ as it assessed the education programme relative to its stated goals as evidenced by concrete and practical outcomes. Our approach was also influenced by realistic inquiry^49,50 because it was designed to meet the needs of a particular programme in a contextualised manner with the data that were available and/or feasible to attain. With this approach we tried not only to determine if the programme was “working”, but for whom and under what circumstances.

Multiple methods were used to assess the coaches’ satisfaction with the training as well as its impact on their knowledge, attitudes and beliefs. Data were collected before, during, and after the training programme. At a four-month follow up, data were gathered to assess how well the education programme had prepared coaches to implement the strategies and best practices it promoted. The evaluation plan and procedures were shared with the ICC and cultural advisor prior to the training so they could comment on instruments and data collection procedures to ensure there was no unintended bias related to personal characteristics such as gender, political affiliation, or local issues unknown to the American team.

Participants and Setting

Participants included 33 Belizean coaches (23 male, 10 female) from organisations that joined the BYSC in the first year. Their average age was 30.52 years old (SD 6.10). Most participants worked in or near Belize City, which is the country’s largest city and is located on the coast. However, some participants travelled from more rural regions farther inland (e.g., bordering Guatemala). All participants spoke English but were of African, Creole, and/or Mayan descent. Participants attended a four-day BYSC coach education programme delivered in March 2014 at a continuing education centre in Belize City. The centre provided a large conference room sufficient for lecture, discussion and demonstration activities. Outdoor space, including a basketball court and open field, were available for demonstration lessons and practical sport activities. These lessons and activities included basketball, soccer, martial arts, physical fitness, and team building. Of these, basketball and soccer were the most commonly played sports in BYSC organisations. However, we believe there was sufficient variety in our examples to demonstrate the concepts and strategies from the training could be applied with any physical activity.

Data collection related to implementation at the four-month follow-up occurred in Belize City in July 2014 when many summer camps were operating throughout the city. Due to the high temperatures, most camp activities took place outdoors in parks, courtyards, playing fields, and a hotel swimming pool. Across multiple site visits and meetings, the first author was able to follow up with 11 of the coaches trained in March, one-third of entire group. Eight of these participants were observed coaching or directing youth programmes.

Overview of Coach Education Programme

Following the recommendations of SDP scholars,³⁶ we endeavoured to shape the coach education programme with local input. In setting aims for the BYSC at the grant proposal stage, we integrated input from the Public Affairs Section at the U.S. Embassy as well as conversations with several Belizeans involved in youth sport programming. At this stage, all parties agreed that it was important to make the training relevant to issues of inter-personal violence. The focus of the project, including the coach education programme, was further refined with input from the first five Belizean delegates who came to the U.S. on an exchange visit in January 2014. For example, these delegates explained that there was virtually no professional development or certification available for youth sport coaches in Belize. Therefore, they encouraged us to integrate fundamental material including how to create a positive motivational climate, coaching for skill development, injury prevention, and effective planning. Even during the four-day training programme, minor adjustments to the planned topics and schedule were made based on participant input. For example, during daily debriefing sessions, participants gave feedback on which topics or modes of practice they thought would be more effective or useful for the following day, such as replacing trainer-led presentations with participant-led action planning at the end of the week.

The training team for the March 2014 coach education included four male professors specialised in physical education and sport pedagogy, the TPSR model, sport-based youth development, and athletic training. The training team also included two consultants (one male, one female) with expertise in the TPSR model, sport psychology, community sport programmes, and sport for development. All members of the training team had extensive practical experience working with youth in their respective fields as well as university and/or professional development teaching experience. The team met prior to the course and debriefed daily throughout to coordinate their efforts, avoid gaps and redundancy across their sessions, and ensure that the curriculum was relevant and responsive to their audience.

The educational programme lasted approximately eight hours per day. The daily routine involved registration and light breakfast followed by the first morning session, break, second morning session, lunch, first afternoon session, and finally a second afternoon session. Two educational sessions were always run concurrently so the participants could be divided into smaller groups to maximise interaction. In line with the aims of the project, several sessions were devoted to the principles of youth development and specific coaching strategies from the TPSR model. Related to social change, sessions were devoted to action planning, using sport to reclaim public space, and community organizing. Some topic areas were integrated into the curriculum based on input from the first BYSC delegates who visited the U.S. in January 2014.

Some earlier sessions relied more on lectures to introduce terms and concepts related to sport-based youth development and the TPSR model. However, these sessions maximised participant-driven discussion after the initial presentation. Other sessions were more practical in nature, such as a trainer leading a model lesson in order to demonstrate how to integrate leadership roles into a basketball practice. These practical sessions engaged participants actively and culminated in debriefing sessions that provided them with opportunities to ask questions, share insights, and make connections to their own work. By the last two days of the training, sessions became more participant-centred. For example, on the third day, several participants were invited to lead practical sessions to demonstrate their ability to implement specific coaching strategies. On the fourth day, based on participant input, some of the practical sessions were reduced in order to devote more time to action planning. This involved small group discussions and sharing among participants about their hopes for the BYSC initiative, personal goals they wanted to set for applying these concepts, and obstacles they anticipated encountering. One of the final sessions involved a process of peer election among the participants to identify leaders from within the group who would be charged with helping to organise and facilitate communication among participants after the training week.¹¹

Data Collection

Training Feedback Surveys

Participants completed customised feedback surveys after the second day of training and again at the end of the course. The mid-training version consisted of seven forced-choice items and three open response items. The forced choice items were rated from 1=very dissatisfied to 5=very satisfied and had to do with quality of the training (e.g., quality of training materials and balance of lecture vs. active learning). The open response items asked participants to identify strengths and weaknesses in the training including examples of content they found most applicable to their work. The final version of the survey also contained a section on training quality and another forced choice section in which participants rated the extent to which they felt the training helped them understand key topics. The 19 items in that section were rated from 1=very ineffective to 5=very effective. The final feedback survey ended with three open response items relating to the most and least useful aspects of the training and content that participants were willing to commit to applying in their work. These customised surveys were created jointly by the American training team, the ICC and the cultural advisor. Customised surveys of this type are recognised as an important data source in formative programme evaluations.⁴⁷ To maximize face validity and content validity, the American team vetted items and crossed them with the training curriculum to ensure that all key learning objectives were represented. To ensure that the required reading level, wording and cultural framing did not present obstacles or biases given the intended audience, the ICC and cultural advisor gave constructive feedback. The finalized surveys were approved by the aforementioned parties and have proven useful in terms of program development and research activities.^11,12

Self-Efficacy for Teaching Personal and Social Skills

This validated survey was administered pre- and post-training. Prior to the first session on the first day and after the last formal session of the course, participants completed a self-efficacy survey that asked them to report their feelings of confidence for teaching personal and social skills through sport (e.g., cooperation). This eight-item scale is rated from 1=not confident at all through 10=extremely confident. This scale was drawn from the Exemplary Physical Education Curriculum (EPEC) Self-Efficacy survey.⁵¹ This survey’s reliability has been demonstrated by showing satisfactory levels of internal consistency and convergent and content validity.

Interviews

In July 2014, the first author returned to Belize to conduct a four-month follow up. The ICC helped to identify BYSC trainees who were implementing summer programmes at that time. Based on access and availability, ¹¹ (8 male, 3 female) participants working in eight different programmes were recruited for interviews. This subsample constituted one-third of the original trainee group. These interviews were semi-structured and conversational in tone, typically lasting 30 minutes. Main topics addressed included recollections of the March 2014 training experience, description of the current sport programme being delivered, examples of integrating training content into coaching, barriers and facilitators to implementation, as well as recommendations to improve the training programme and ongoing support. Interviews were audio recorded and transcribed verbatim.

TPSR Implementation Checklist

This checklist¹⁵ has proven useful in documenting which elements of TPSR implementation are seen in practice. In the current study, it was applied in assessing demonstration lessons delivered by the training team and/or participant volunteers in March 2014. The checklist was also used on site visits in July 2014. The checklist is comprised of four sections relating to the core TPSR goals and lesson format as well as common teaching strategies and student behaviours. The teaching strategies include modelling respect, setting expectations, providing opportunities for success, fostering social interaction, assigning management tasks, promoting leadership, giving choices and voices, involving students in assessment, and addressing the transfer of life skills. Key student behaviours include participating, engaging, showing respect, encouraging others, cooperating, helping others, leading, expressing voice, and asking for help. Some of these strategies and behaviours are more common than others so together they represent a range of indicators to describe and assess the extent to which a lesson aligns with the TPSR model.^19,34

Field Journals

The first, third and fourth authors were part of the training team that delivered the coach education programme in March 2014. All three recorded their observations and reflections during the training week. Feedback elicited during informal conversations were also frequently documented in these journals. Generally, entries were made in the evening or morning hours (i.e., before or after training sessions). Entries were made directly into electronic files called field journals. The first author used the same approach when conducting site visits during the summer.

Artefacts and Documents

Artefacts and documents were retained throughout the study. Planning documents, correspondence, the original project proposal, as well as handouts and PowerPoint presentations are materials generated prior to the delivery of the coach education programme. Registration and attendance records, video footage and photo documentation are materials that documented the delivery of the training programme. We also retained correspondence with the ICC, reports to the funding agency, and media coverage were generated between March and July. During July, additional video footage and photo documentation was gathered.

Data Analysis and Trustworthiness

Regarding quantitative data, analysis began with standard procedures for the initial cleaning and screening of survey data.⁵² Through this process, it was found that the data were appropriate for inferential statistics. Study constructs scores were created by averaging the items related to the subscale. Descriptive statistics (means and standard deviations) were then calculated to assess both the general effectiveness of the training and how well the training helped the participants understand the topics covered during the training (Table 1). Next, descriptive statistics were calculated and a paired-samples t-test was conducted to examine changes in participants’ self-efficacy from pre- to post-training (Table 2). Cohen’s d is presented as a measure of effective size for the t-test conducted. A Cohen’s d value between .15 and .40 is associated with a small effect, between .40 and .75 with a medium effect, and above .75 with a large effect.⁵³

Qualitative data were analysed with a combination of deductive and inductive analytic strategies.⁵⁴ The value of this approach has been promoted in sport-related research⁵⁵ and has proven useful in previous TPSR programme evaluations.^11,15,26 Specifically, data were organised using a priori codes that attributed units of meaning to key concepts of interest such as satisfaction, learning, attitudes and beliefs, transfer of learning, relevance, application, and the various aspects of the TPSR model. Once data were coded and organised, inductive analysis was undertaken to identify emergent themes and patterns within these broader categories. Higher order themes and subthemes were determined through consensus with all authors. Illustrative quotes were selected to integrate into the narrative based on how well they characterised the meaning of the various subthemes.

Trustworthiness in this evaluation was enhanced using several well established techniques.^47,56 Chief among these were the triangulation of multiple data sources and methods. Member check procedures consisted of debriefing interviews with several leaders in the BYSC to share and discuss the initial interpretations of the research team.

Extensive peer debriefing occurred among the members of the evaluation team throughout the process. Credibility is also enhanced due to our detailed knowledge and direct experience with the project over an extended period of time.

Regarding reflexivity, we were sensitive to imbalances in power. Because members of the American training team were both delivering the program and also conducting the evaluation, they made efforts to create a climate of partnership and minimise this power differential. Specifically, the local voice was prioritised through encouraging honest feedback during structured feedback sessions within the training program as well as through member checking during data collection and analysis. Efforts were also made to establish meaningful relationships with participants in order to foster a sense of of mutual respect and openness. The ICC, cultural advisor and other BYSC directors were helpful in assuring participants that their voices were valued and should be heard. Despite these proactive attempts, some participants may have been uncomfortable sharing negative or critical perspectives due to the inherent power differential. Awareness of this potential tension informed our approach to gathering, analysing, and reporting data. The cultural advisor, in particular, was helpful as a critical friend as we discussed our findings and interpretations.

Attitudes and Beliefs

Table 2 displays the mean variables for all items and the self-efficacy for teaching personal and social skills construct administered at pre- and post-training, as well as change in (Δ) scores. The participants who completed both the pre- and post-training survey (n = 18), on average, perceived their self-efficacy levels for the construct and all items to be above 8.6 (out of 10) upon completion of the training programme, with many of the items above 9.1 (out of 10). All items and the self-efficacy construct showed increases from pre- to post-survey administration. Results of the paired-samples t-test are included in Table 2 along with the related Cohen’s d value. The increase across the self-efficacy construct from pre- to post-survey administration was found to be significant (p < .001, t = 6.191, d = 1.46). Moreover, the two items that saw the largest increase after the orientation training related to constructive competition (Δ = 2.16) and best effort (Δ = 1.34); while the two factors that saw the smallest increase related to respect (Δ = 0.73) and responsibility (Δ = 1.00).

Consistent with increased feelings of self-efficacy, open responses in feedback surveys indicated varied impacts on participants’ beliefs and attitudes. Many demonstrated critical reflection as well as motivation and commitment to the BYSC project. One female participant mentioned increased confidence directly and how that helped her feel empowered as a change agent. She wrote, “One of the most valuable thing that I learnt in this training thus far is building self-confidence within myself. Am sure I can go back within my community to execute what I learned for the past days”. She and many of her counterparts were reflective in their responses. One male indicated the training influenced the way he would view his athletes and his role as a coach, writing:

The most useful aspect of the training were the tools and knowledge given to me to actually connect with the kids we work with on a personal level and understand their personal situations. This will further help me know them better as a person and not just as an athlete who can help me win games.

Regarding the belief that a coalition of youth sport organisations and coaches could make a difference, one male participant wrote:

In Belize, our youth of today and the future needs those strategies to help us as a nation to become better in sports, social life and even on a daily basis. These strategies will help our youth/children to be better learners, respect others people as an individual as well as a culture, which will end up in positive result for us as a small country.

Implementation

During the July 2014 follow-up visit, the first author visited multiple summer camps and conducted interviews with BYSC administrators and coaches. At that point, 20 of the coaches trained in March had the opportunity to apply the content of the training in youth programmes, which included sport-specific camps (e.g., volleyball, basketball, and soccer), swimming lessons, recreational summer camps, a city-wide basketball tournament, an outdoor education camp, and visits to primary school physical education programmes across the country. In total, after four months, trained BYSC coaches had worked directly with 1,184 youth participants. Within their organisations, the BYSC trained coaches had been able to mentor at least 25 new coaches and eight youth leaders. Of course, these outputs do not reflect the extent to which material from the training was actually applied. In an interim report on the project submitted after the July 2014 visit, the first author opined:

My impression is that almost all of the coaches we trained value and support the principles of youth development…Based on my observations and interviews, the most commonly used teaching strategies in BYSC programmes are modelling respect, creating a positive and inclusive environment, planning activities that allow all participants to feel successful, and discussing character or life skills. I saw some brilliant examples of fostering social interaction among students, involving youth leaders in management and instruction tasks, but those were isolated examples and need to be developed more across programmes.

In the following paragraphs, three examples are provided that establish links between the coach education programme and changes in practice.

Example One

Two of the coaches that were interviewed and observed were Gabriele and Dale, who both worked for a government-operated sport organisation. In this capacity, they supported sport programmes and to a lesser extent physical education in primary schools during the academic year. Regarding the primary schools, Gabriele reported, “Well, we did most of the primary schools in the district, in the entire [Belize City]; roughly about 10 primary schools”.

She and Dale reported that they tried to use strategies from the trainings in these lessons, but struggled because they typically did not know the students and had only one opportunity to teach them. The intent of these visits was to model best practices for classroom teachers charged with teaching physical education. However, as Dale explained, “Most of the time, the teacher leaves and I just be the teacher of the class”. Both reported they were able to implement more of the training content in their summer camps. According to Dale, “We had been working with primary school programmes, but now that the summer programmes have started, we now have the opportunity to work with some of the skills we did in the workshop”. Regarding their coaching in the summer camps, Dale reported, “I know we work on the respect aspect. You know, like, ‘your friend wants to learn too and it’s not just about you’. We try to make sure we get everybody involved”. The first author observed a soccer lesson taught by these two coaches that was consistent with their reports. In fact, during the observed lesson, they integrated an intentional activity to promote communication among the players. While this activity was well-implemented and highlighted new structures the coaches were using to foster life skills, there was little discussion or debriefing with the players about these life skills. In an interview after the lesson, when this point was raised, Dale reflected:

In the camp, when we were doing the specific things to achieve the particular goal of the respect, the leadership, team building, we weren’t doin’ the talkin’ part enough. I gotta be honest, that part we wasn’t stressing a lot, but that’s really what we should be stressing that part because that’s really the important part.

Example Two

Another example of the training content being applied was seen in the context of a non-for-profit organisation. This organisation had a strong set of core values and commitment to youth development. Interviews with the programme director and coaches as well as observations made it clear that empowering strategies such as youth leadership were part of the culture of the organisation. In fact, two separate programmes operated by this organisation were observed and in both cases, teenagers who had previously participated in the programme were now serving as assistant coaches. Although such strategies pre-dated the March 2014 training, coaches reported the training had validated their approach and helped them become more intentional and explicit in integrating their core values into activities. One coach, Manny, explained:

Basically, the way the actual training was, was basically hand in hand with what we do. The way we incorporate the actual values into the actual programme is something we were lacking. [The training] gave us the main idea of how we could actual do everything smooth, including the physical and the mental…Sometimes we were doing the physical only, and then the mental afterward, but now I’m at the point where I teach both the physical and mental.

As a senior coach in the programme, Manny explained that he had been stressing this new approach with his peers and the youth coaches. He reported, “telling others they are not just going to teach the sport this year but the core values too”, then added, “It was my main idea for this training for this year…not to only teach the sport but to teach and learn the values of the actual programme”.

Example Three

In another government-sponsored programme focused on youth services, a model lesson was observed that integrated life skills education with basketball practice. Through observation and interviewing, it was clear that the coach,

Elijah, was directly applying strategies from the March 2014 training. In a single lesson, Elijah demonstrated all nine of the teaching strategies on the TPSR checklist. As quoted in a previous publication,11 (p.538) Elijah explained:

In the previous year, our life skills [instruction] was very poor because we used it in the classroom setting and their attention span is very short. So we started using the TPSR model last year and our life skills turnout was tremendous.

This success was not just a matter of changes in pedagogy, it also required organisational support to rearrange the schedule of the programme and alter the curriculum. As summarised in a field journal:

This year, [Elijah’s programme] are experimenting with teaching life skill lessons through the sport activity (i.e., purposeful drills and activities to highlight conflict resolution and communication skills). All parties involved agree the youth find this approach much more engaging and observe that attendance and retention seem to be improved over previous years.

The examples provided above demonstrate the linkages between training and implementation. These are selected examples of stronger implementation shared for that purpose. It must be noted that other programmes were observed that showed little or no evidence of implementation. Virtually all coaches had positive recollections of the training, but only some had made systematic efforts to apply the material. Some coaches would speak enthusiastically about how much they had changed their approach, but upon direct observation only the most fundamental coaching strategies covered could be seen (e.g., modelling respectful behaviour, including all students, creating a positive climate). It was noted in a field journal in July 2014 how organisational factors were influencing the degree of implementation:

I saw a wide range in terms of the quality of implementation of our training strategies and principles. At the organisational level, I have the impression that some have taken little action to make changes whereas others have taken concrete steps in a relatively short period of time.

Consistent with this last point, some coaches received support to make changes related to scheduling, venue, curriculum, and coaching style. Other coaches met resistance to change in various forms. It is beyond the scope of this study to fully analyse organisational barriers and facilitators, but they did appear to influence implementation.