Justin Richards1, Emma Sherry2, Olivia Philpott3, Lewis Keane4, Nico Schulenkorf5, Adrian Bauman1
1 University of Sydney, School of Public Health & Charles Perkins Centre, Australia
2 La Trobe University, Centre for Sport and Social Impact, Australia
3 Netball Australia, Australia
4 Cricket Fiji, Fiji
5University of Technology Sydney, Department of Sport Management (Australia)
Citation: Richards, J., Sherry, E., Philpott, O., Keane, L., Schulenkorf, N., Bauman, A. Evaluation protocol: Netball to promote physical and mental health in Samoa and Tonga. Journal of Sport for Development. 2016;4(7):1-11
It is widely accepted that physical activity has health benefits and that it is a critical component of addressing the global emergence of non-communicable diseases.1 This may be particularly pertinent in Samoa and Tonga where almost 50% of adults are insufficiently active,2 the prevalence of overweight is among the highest in the world (i.e. >80%)3,4 and the costs of related non-communicable diseases are escalating.5 However, the promotion of physical activity may also have broader implications in Samoa and Tonga. Specifically, there is growing evidence that physical activity participation can prevent mental illness (e.g. depression) and is associated with mental well-being (e.g. happiness).6,7 It is also evident that mental health may be an important mediating factor for preventing early mortality due to non-communicable diseases.8 Despite a paucity of data on the mental health needs of Pacific Island Countries, suicide rates are higher than the global average, suggesting that mental illnesses may be prevalent.9,10 Consequently, addressing this apparent mental health need through physical activity interventions may also play an important role in reducing the burden of non-communicable diseases in Samoa and Tonga.
Increasing physical activity and preventing non-communicable diseases are stated objectives of the One Netball Pacific program in Samoa and Tonga.11 These programs commenced in 2009 in Samoa and 2011 in Tonga and have received ongoing strategic and resource support from Netball Australia, enabled through Pacific Sports Partnership funding.11 Despite concurrent qualitative evaluation of program delivery, there has been limited quantitative evaluation of intervention processes or its impact on netball participation, physical activity levels and other health-related indicators.11 Furthermore, we are not aware of any published evaluations assessing the impact of netball participation on body composition, mental well-being or the subsequent reduced risk of other non-communicable diseases.
The purpose of this evaluation is to address a gap between current practice and existing evidence in the sport-for-development sector. Despite pervasive positive rhetoric about the health implications of sport programs that continues to attract ongoing international investment, the evidence base remains limited.12
Therefore, we aim to conduct a process and impact evaluation of the One Netball Pacific program in Samoa and Tonga. Specifically, we aim to assess the organisational objectives of “creating more opportunities for women and girls to take part in physical activity through netball” to “improve health-related behaviours” and “reduce the impact of non-communicable diseases in these communities”.11
To assess the delivery and reach of One Netball Pacific programs and its impact on program registrant: netball access; netball participation; recreational physical activity; body composition; mental well-being.
The overall objectives of the netball programs in Samoa and Tonga are to build local governance capacity, provide leadership opportunities for women and sustainably increase recreational physical activity to improve health across all age-groups. We propose to focus on the health-related objectives and evaluate two components of the programs being delivered for adults:
a) Social Mixed Netball
The Social Mixed Netball League is known as “Business House” in Samoa and the “Corporate Cup” or “Fiefia” in Tonga. Despite minor differences between countries, it typically comprises two or three playing “seasons” each year that go for six to ten weeks and are conducted in the major towns of Samoa (Apia) and Tonga (Nuku’alofa). Groups of individuals representing local organisations self-select 12 team members and then voluntarily register to participate in a netball league that welcomes all genders. Previous iterations have often been over-subscribed with approximately twice as many teams registering as places available in the competition. Selection of the participating teams in each “season” is decided on a “first-come, first-served” basis and the number of teams included is determined by locally available infrastructure and resources. Participation in the Social Mixed Netball League typically involves:
- Separating the selected teams into two equal-sized pools;
- One or two games of organised netball per week for each team at a central location (30minutes/game);
- Additional voluntary training sessions that are self-organised.
(NOTE: Social Mixed teams typically comprise people with mixed abilities representing local workplaces).
b) Grassroots Netball Development
The local netball federations have adopted a multi-faceted program approach for promoting netball participation by adults throughout Samoa and Tonga. Grassroots Netball Development typically includes:
- Providing equipment that enables communities to participate in recreational netball;
- Delivering coach, player and administrator development workshops designed to build capacity and encourage local autonomy for promoting sustained netball participation;
- Organising mass participation short tournaments/ carnivals (e.g. one-day competition) that attract participants from multiple villages to a central location for playing and promoting netball;
- Facilitating a home & away netball series where teams from several neighbouring villages play one game against each other at home and one game away;
- Supporting club netball series where registered clubs are graded and compete against each other.
(NOTE: The tournaments / carnivals, home & away series and club netball series typically cater for all genders and include both open and senior age categories. In Tonga, the senior category is known as “Over 35” and only includes participants whose age or body mass index are higher than 35).
The proposed actions, outputs, impacts and outcomes of the Social Mixed Netball League and the Grassroots Netball Development program are consistent with a published theoretical framework for sport-for-development programs.13 Although there is some overlap with the stated objectives of governance and empowerment, we have developed a logic model that focuses on the health-related outcomes (Figure 1).14
a) Social Mixed Netball League (Figure 2)
The oversubscription of registered teams provides an opportunity to conduct a natural experiment. The first teams to register will be allocated to the intervention group until local infrastructure and resource capacity is reached. All subsequent teams that register will be wait-listed and given priority entry to the subsequent “season”. All adults in the registered teams will have their netball access, netball participation, recreational physical activity, body composition and mental well-being assessed before and after the program. Further follow-up assessment will occur after a six-week “wash-out” period. We will also conduct a parallel process evaluation for intervention delivery and reach in the participating teams.
b) Grassroots Netball Development (Figure 3)
We will conduct periodic monitoring in an evaluation of multiple intervention communities. Six communities targeted by the Grassroots Netball Development program in each of Samoa and Tonga will be identified. People from each of these communities will have their netball access, netball participation, recreational physical activity, body composition and mental well-being assessed at 12-month intervals. We will also conduct a parallel process evaluation of intervention delivery and reach in each of the communities. Small focus group discussions with community representatives and semi-structured interviews with program personnel will be conducted in parallel at 12 monthly intervals to further substantiate these process indicators and triangulate the quantitative data.
Despite different intervention recruitment procedures the target population is consistent for both the Social Mixed Netball League and the Grassroots Netball Development program.
- Adults (aged older than 18 years);
- Eligible to participate in the netball programs;
- (NOTE: Participation in the “Over 35” programs requires an age or body mass index higher than 35).
- Professional advice against vigorous-intensity physical activity due to health contraindications;
- Diagnosed severe mental illness.
Sample Size and Selection
All sample size and selection procedures will be determined by pragmatic considerations of program structure and capacity. The following calculations provide guidance and a recruitment target.
a) Social Mixed Netball League
The required sample size (n=392 per country) was calculated using relevant physical activity data from a previous validation study of the selected metric and by applying a standard formula based on: a normal distribution (z); probability of type I error (α=0.05); probability of type II error (β=0.20); estimated standard deviation (σ=52.96 mins/day)15; expected effect size (δ=15.00 mins/day). To allow for approximately 15% attrition, all members of first 20 teams in the intervention group (n=240 per country) and all members of the first 20 wait-listed teams (n=240 per country) will be included in the evaluation.
b) Grassroots Netball Development
The required sample size (n=146 per country) was calculated using relevant physical activity data from a previous validation study of the selected metric and applying a standard formula based on the following factors: a normal distribution (z); probability of type I error (α=0.05); probability of type II error (β=0.20); estimated standard deviation of within-group change (σ=64.60 mins/day)16 ; expected effect size (δ=15.00 mins/day). To allow for approximately 15% attrition, a convenience sample of 30 people will be selected from each of the six participating communities (n=180 per country) for quantitative data collection. Study participants will be the same people at each assessment period. The participating communities will be selected based on previous engagement with the national netball federations and agreement from village elders and leaders to participate in the evaluation. Additionally, qualitative assessment of a purposeful sample of program participants, coaches, umpires, staff and other stakeholders from the national netball federations will be conducted until data saturation is reached.
The primary and secondary outcomes are consistent for both the Social Mixed Netball League and the Grassroots Netball Development program. The primary outcomes are netball access and netball participation. The secondary outcomes are recreational physical activity, body composition and mental well-being.
Data Collection Procedures (Quantitative)
Researchers from the University of Sydney will collaborate with health workers and train a local evaluation team in both Samoa and Tonga to collect the necessary quantitative data. These results will be recorded using iPads in the order outlined below and automatically uploaded to a central database. The survey items will be translated into the local language to facilitate communication and delivered in the form of an interview. Prior to commencing data collection, the interviewer will explain the purpose of the evaluation to the participant and then answer any questions before obtaining their informed consent to proceed.
The local evaluation team will use the same measurement tools to collect demographic data and results for the primary and secondary outcomes at all time points for both the Social Mixed Netball League and the Grassroots Netball Development program.
Data will be collected individually and commence with the following identification and demographic information:
- Full name (anonymised to program implementers and data analysts);
- Gender (male / female / other Samoa: fa’afafine, Tonga: leities);
- Age (years);
- Village of residence (including postcode if available);
- Highest completed education level (none / primary school / secondary school / university degree);
- Main work status in past 12 months (government / private sector / non-government organisation / self-employed / non-paid volunteer / student / homemaker / retired / unemployed).
The survey items used to collect data on each of the outcome measures have been derived from various sources (Table 1).
Netball access will be assessed using a series of novel survey items about local capacity to engage in ongoing netball activities. All questions have been developed and piloted in the local communities and are required to be answered (Table 1: survey items 1.1-1.6).
Netball participation will be assessed using a series of novel survey items about playing frequency and comparisons to pre-intervention. The questions have been developed and piloted in the local communities and will vary according to the assessment timing and program component (Table 1: survey items 2.1-2.4).
Mental well-being will be evaluated using the WHO-5 Well-Being Index, which is a globally validated self-report measure.17 All questions are required to be answered (Table 1: survey items 3.1-3.5).
We will also apply a globally validated one-item happiness measure, which has been modified to include a visual analogue scale to improve discriminating power and reduce gender distortion (Table 1: survey item 4.1).18 This will subsequently be converted to a score on a linear 100 point scale for analysis.
Recreational physical activity will be assessed using the relevant section of the Global Physical Activity Questionnaire (GPAQ), which is a globally validated self-report measure that has been adapted to the local context in Samoa and Tonga.15,16 The questions asked will vary according to the responses provided and minutes of moderate and vigorous intensity recreational physical activity will be calculated for analysis (Table 1: survey items 5.1-5.2).
Body height and weight will be assessed objectively and used to calculate Body Mass Index (BMI) by applying standardised methods previously used in low- and middle-income settings.19 All measures will be performed by locally trained assessors using consistent apparatus for all participants (i.e electronic scales, telescopic ruler).
The timing / location of data collection and process indicators will vary according to program component:
a) Social Mixed Netball League
All data will be collected face-to-face by a member of the evaluation team at a location that is convenient for the registered participants (e.g. workplace, training venue, village hall). Arrangements to complete baseline measurements will be made immediately after a team registers for the program. Follow-up measurements can commence immediately after the “season” is complete and will take place at the same location as the baseline assessments.
Process indicators for program delivery and reach will include a weekly record of player participation in games and training. Each team captain will be provided with a weekly template to complete and submit to a member of the local evaluation team who will collate these in a previously established excel spreadsheet.
b) Grassroots Netball Development
All data will be collected face-to-face by a member of the evaluation team at a convenient time and location for the participating members from each community. Arrangements to complete baseline measurements will be made as soon as the evaluation team has been trained. Follow-up measurements will be scheduled at a time that is convenient for the participants and will take place at the same location as the baseline assessments.
Process indicators for program delivery will be recorded throughout the year using previously established excel spreadsheets and collated annually by a member of the local evaluation team. It will include the collection of the following data for each participating community across all age groups:
- Equipment provision (number of netballs and other equipment provided);
- Training workshops (number of participants and types of netball workshops);
- Mass participation tournaments / carnivals (number of participants and tournaments /carnivals);
- Home and away series (number of participants, games and training sessions);
- Club Netball Series (number of registered clubs / players and series / games played).
Data Collection Procedures (Qualitative)
A researcher from La Trobe University will lead the semi-structured interviews and small focus group discussions for the Grassroots Netball Development program. All data will be collected face-to-face at a convenient time and location for the participants. Prior to commencing data collection, the interviewer will explain the purpose of the evaluation to the participant and then answer any questions before obtaining their informed consent to proceed. The question framework addresses issues related to program format, the participating population, delivery successes and implementation challenges.
Data Analysis and Presentation
All quantitative data will be scored and cleaned using the same procedures as outlined in relevant previous studies.15-19 The unit of analysis will be the individual. Due to contextual differences and anticipated heterogeneity in program implementation, we will initially conduct separate analyses for each country.
(NOTE: If our process indicators indicate that program implementation is consistent in both Samoa and Tonga we will consider conducting subsequent between-country comparisons and pooled analyses that are not described in this protocol).
a) Descriptive Statistics
Descriptive statistics for the Social Mixed Netball League will be grouped according to intervention and wait-list at each time point. For the Grassroots Netball Development program, descriptive statistics will be presented for each community and also pooled at each time point. The sample proportions for gender, village of residence, education level and work status will be calculated and tabulated. The means and 95% confidence intervals for age and each outcome variable will also be calculated and tabulated.
b) Baseline Data Analyses
The means and 95% confidence intervals for each outcome variable at baseline will be used to assess if there are any differences between groups in the Social Mixed Netball League and between communities in the Grassroots Netball Development program. We will also assess if there are any differences at baseline between study completers and those lost to follow-up. Finally, the baseline GPAQ data will be compared to the most recent WHO STEPS survey data collected in Samoa (2013) and Tonga (2012) to assess who the netball program is reaching (i.e. recreational physical activity: evaluation sample vs. country norms).
c) Impact assessment analyses
All participants who complete baseline measurements will be included in an intention-to-treat analysis for each outcome. For participants lost to follow-up, we will assume no change from their most recent measurement. All results for the within-group and between-group analyses will be tabulated and the threshold for statistical significance will be p<0.05.
The within-group and between-group analyses for the Social Mixed Netball League will be conducted according to intervention vs. control group. We will also conduct further stratified analyses if the process indicators suggest heterogeneity in program delivery and reach for different teams. Stratified analyses may also be conducted according to differences in baseline characteristics (e.g. BMI). The analyses for the Grassroots Netball Development program will be completed for each separate community. We will also pool the results into high and low uptake communities according to the distribution of the collected process indicators and qualitative data for program delivery and reach. Provided the relevant statistical assumptions are met, all within-group changes will be assessed using a paired t-test and between-group analyses will apply a univariate ANOVA. Crude and standardised effect sizes will also be calculated using a pooled standard deviation. Analyses will be completed for the crude data initially and subsequently adjusted for baseline measures, gender, age, education and/or work status. The adjusted analyses will also allow for clustering according to team and/or community.
The qualitative data will be analysed using a hybrid model of inductive and deductive thematic analysis. Theory from sport-for-development, specifically focussing on the key concepts within a program logic framework will provide the deductive framework for interpreting the collected data and identifying pre-established themes.20 Themes identified through the inductive qualitative data analysis will be undertaken through a systematic open coding process, including an initial broad read through each interview transcript, a search for new recurring concepts and patterns not previously identified via the deductive coding underpinned by a program logic framework, and then grouping together of these new recurring concepts and themes. Data will be organised in NVivo 10 for both inductive and deductive systematic coding.
This evaluation has been approved by the La Trobe University Human Research Ethics Committee (13-073). Approval to evaluate the One Netball Pacific programs has also been granted by Netball Australia, the national netball federations of Tonga and Samoa, as well as village elders and community leaders of villages/towns included in this evaluation.
The proposed evaluation has several strengths and will introduce a new level of rigour to assessing sport-for-development programs that use netball to promote health-related outcomes.12 It is guided by a clearly developed logic model that is consistent with an existing theoretical framework for intervention implementation.13 The proposed methods minimally disrupt program development and are designed to capture quantitative and qualitative data that explains delivery, reach and impact. This mixed methods approach and the use of previously validated measures will improve the external validity of the results.15-19The proposed analyses will identify the critical program components for promoting netball and recreational physical activity participation in Samoa and Tonga. Consequently, the results of this evaluation will inform best practice for future investment of netball resources locally, but may also have more global implications in the sport-for-development sector.
However, the practical reality of building evaluation into an already existing program introduces methodological limitations. Firstly, training a local data collection team means that the interviewers will not always be blinded to group allocation. Although this builds local capacity and improves feasibility, it may also introduce a reporting bias. Secondly, there is a risk of group contamination as people move between local communities and workplaces. This may dilute the potential program response and compromise participant tracking. Thirdly, the recreational physical activity section of GPAQ has not been validated in isolation from the transport and vocational domains that are part of the complete measurement tool. Although our approach focusses on the survey components relevant to the netball intervention and improves evaluation feasibility by reducing participant burden, it is not known how this will affect comparisons with existing WHO STEPS data. Fourthly, there are several limitations in the sampling procedures. The “first-come, first-served” allocation process for the intervention and wait-list groups in the Social Mixed Netball League may introduce a selection bias that we will attempt to adjust for in the analyses. Furthermore, the absence of a control group for the Grassroots Netball Development program compromises the attribution of causation and this is only partially addressed by our proposed triangulated and stratified analyses. Finally, the self-selection of participants in both program components and the convenience sampling methods utilised for the grassroots netball development evaluation limit the applicability of the evaluation findings to the broader community, but may also be critical for retaining an adequate sample size. Despite these limitations, we have attempted to maintain the most rigorous methods possible within the pragmatic constraints of a real-world program and contribute to the evidence base for evaluation methodology in the sport-for-development sector.
JR, ES, NS, LK and AB declare no financial, personal or professional competing interests. OP currently works for Netball Australia and has provided critical insight into the delivery of One Netball Pacific programs in Samoa and Tonga, but will be excluded from the proposed data collection and analyses.
The organisational stakeholders in this evaluation and their primary roles are listed below:
- Netball Australia
Secure resources and provide strategic guidance for One Netball Pacific programs with the support of Pacific Sports Partnership funding.
- Netball Samoa
Deliver the One Netball Pacific programs in Samoa.
- Tonga Netball Association
Deliver the One Netball Pacific programs in Tonga.
- La Trobe University / University of Sydney / University of Technology Sydney
Complete an independent evaluation of program delivery and impact.
1. Lee IM, Shiroma EJ, Lobelo F, Puska P, Blair SN, Katzmarzyk PT, Lancet Physical Activity Series Working G. Effect of physical inactivity on major non-communicable diseases worldwide: an analysis of burden of disease and life expectancy. Lancet, 2012. 380(9838): p. 219-29.
2. Hallal PC, Andersen LB, Bull FC, Guthold R, Haskell W, Ekelund U, Lancet Physical Activity Series Working G. Global physical activity levels: surveillance progress, pitfalls, and prospects. Lancet, 2012. 380(9838): p. 247-57.
3. Ng M, Fleming T, Robinson M, Thomson B, Graetz N, Margono C, Mullany, E, Biryukov S, Abbafati C, Abera S, Abraham J, Abu-Rmeileh N, Achoki T, AlBuhairan F, Alemu Z, Alfonso R, Ali M, Ali R, Guzman N, Ammar W, Anwari P, Banerjee A, Barquera S, Basu S, Bennett D, Bhutta Z, Blore J, Cabral N, Nonato I, Chang J, Chowdhury R, Courville K, Criqui M, Cundiff D, Dabhadkar K, Dandona L, Davis A, Dayama A, Dharmaratne S, Ding E, Durrani A, Esteghamati A, Farzadfar F, Fay D, Feigin V, Flaxman A, Forouzanfar M, Goto A, Green M, Gupta R, Hafezi-Nejad N, Hankey G, Harewood H, Havmoeller R, Hay S, Hernandez L, Husseini A, Idrisov B, Ikeda N, Islami F, Jahangir E, Jassal S, Jee S, Jeffreys M, Jonas J, Kabagambe E, Khalifa S, Kengne A, Khader Y, Khang Y, Kim D, Kimokoti R, Kinge J, Kokubo Y, Kosen S, Kwan G, Lai T, Leinsalu M, Li Y, Liang X, Liu S, Logroscino G, Lotufo P, Lu Y, Ma J, Mainoo N, Mensah G, Merriman T, Mokdad A, Moschandreas J, Naghavi M, Naheed A, Nand D, Narayan K, Nelson E, Neuhouser M, Nisar M, Ohkubo T, Oti S, Pedroza A, Prabhakaran D, Roy N, Sampson U, Seo H, Sepanlou S, Shibuya K, Shiri R, Shiue I, Singh G, Singh J, Skirbekk V, Stapelberg N, Sturua L, Sykes B, Tobias M, Tran B, Trasande L, Toyoshima H, van de Vijver S, Vasankari T, Veerman J, Velasquez-Melendez G, Vlassov V, Vollset S, Vos T, Wang C, Wang X, Weiderpass E, Werdecker A, Wright J, Yang Y, Yatsuya H, Yoon J, Yoon S, Zhao Y, Zhou M, Zhu S, Lopez A, Murray C, Gakidou E. Global, regional, and national prevalence of overweight and obesity in children and adults during 1980-2013: a systematic analysis for the Global Burden of Disease Study 2013. Lancet, 2014. 384(9945): p. 766-81.
4. Hawley NL, McGarvey ST. Obesity and diabetes in Pacific Islanders: the current burden and the need for urgent action. Curr Diab Rep, 2015. 15(5): p. 29.
5. Anderson I. The economic costs of non-communicable diseases in the Pacific Islands. 2012, The World Bank: Washington.
6. Cooney GM, Dwan K, Greig CA, Lawlor DA, Rimer J, Waugh FR, McMurdo M, Mead GE. Exercise for depression. Cochrane Database Syst Rev, 2013. 9: p. CD004366.
7. Richards J, Jiang X, Kelly P, Chau J, Bauman A, Ding D. Don’t worry, be happy: cross-sectional associations between physical activity and happiness in 15 European countries. BMC Public Health, 2015. 15: p. 53.
8. Thornicroft G. Premature death among people with mental illness. BMJ, 2013. 346: p. f2969.
9. Hughes F. Mental health in the Pacific: the role of the Pacific Island Mental Health Network. Pac Health Dialog, 2009. 15(1): p. 177-80.
10. World Health Organisation (WHO), Secretariat of the Pacific Community (SPC). Towards Healthy Islands: Pacific Mental Health Response. 2013, Tenth Pacific Health Ministers Meeting: Apia, Samoa.
11. Sherry E, Schulenkorf N, Nicholson M, Hoye R. Pacific Netball Partnerships: Summary Research Report (Samoa and Tonga). 2015, La Trobe University: Melbourne, Australia.
12. Cavill N, Richardson D, Foster C. Improving Health Through Participation in Sport: a review of research and practice. 2012, British Heart Foundation Health Promotion Research Group: Oxford, UK.
13. Richards J, Foster C. Sport-for-development program objectives and delivery: a mismatch in gulu, northern Uganda, in Global Sport-for-Development: Critical Perspectives. N. Schulenkorf and D. Adair, Editors. 2013, Palgrave MacMillan: Melbourne.
14. Bauman A, Nutbeam D. Evaluation in a Nutshell. 2014, Sydney: McGraw Hill.
15. Herrmann SD, Heumann KJ, DerAnanian CA, Ainsworth BE. Validity and Reliability of the Global Physical Activity Questionnaire (GPAQ). Measurement in Physical Education and Exercise Science, 2013. 17(3): p. 221-235.
16. Cleland CL, Hunter RF, Kee F, Cupples ME, Sallis JF, Tully MA. Validity of the global physical activity questionnaire (GPAQ) in assessing levels and change in moderate-vigorous physical activity and sedentary behaviour. BMC Public Health, 2014. 14: p. 1255.
17. Topp CW, Ostergaard SD, Sondergaard S, Bech P. The WHO-5 Well-Being Index: a systematic review of the literature. Psychother Psychosom, 2015. 84(3): p. 167-76.
18. Studer R. Does it matter how happiness is measured? Evidence from a randomized controlled experiment. 2011, University of Zurich: Department of Economics.
19. Richards J, Foster C, Townsend N, Bauman A. Physical fitness and mental health impact of a sport-for-development intervention in a post-conflict setting: randomised controlled trial nested within an observational study of adolescents in Gulu, Uganda. BMC Public Health, 2014. 14: p. 619.
20. Coalter F. Sport-for-development: What game are we playing? 2013, Abingdon, UK: Routlege.