Methods: recruiting to internet-based research studies

The submission deadline approaches, and students send out desperate e-mails requesting participants. 'Please, please help me! 20 participants needed by Friday!' Many of these requests seem to relate to internet-based studies. This is surprising, because internet-mediated research (IMR) is designed to be more accessible and less effortful for participants, while potentially attracting large amounts of data quickly at little cost (K. Gardner, 2007; Hewson, 2003). One of the purported advantages of IMR is that, once a study website is active, often 'all the researcher needs to do is sit and wait for data to come in' (Hewson, 2003, p.290). Clearly, this is not always the case.

Some IMR researchers seek to recruit by e-mailing details to friends and colleagues and requesting that recipients forward these to others, thereby creating a 'snowball' effect. The snowball can fail to materialise though, presumably because the researcher's impassioned pleas are not persuasive to those with no personal or professional connection to the researcher.During my doctoral studies, I used financial incentives to sustain the snowball effect. This 'incentivised snowballing' technique involves paying (or otherwise rewarding) those who, by forwarding study information via e-mail, enrol most participants (see Table 1).

Table 1. How to use incentivised snowballing

Study phase: Preparation of data collection materials
Action: 1. Incorporate a compulsory item in the data collection form requesting e-mail details of the person from whom study information was received.
-----------------------------------------------------------------
Study phase: Advertising study via e-mail
Action: 2. Send an e-mail appealing for participants, outlining inclusion criteria, incentives for participation, weblink, researcher contact details, etc.
Emphasise that a (financial) reward will be given to the person(s) who recruit most others by forwarding this e-mail.
--------------------------------------------------------------
Study Phase:Data collection (if slow)
Action:3a. Resend the advertisement e-mail to original recipients.3b. E-mail leading recruiters to inform them that further recruitment could determine whether they are awarded the prize.

----------------------------------------------------------------
This entails circulating an advertisement e-mail detailing both the study (inclusion criteria, incentives for participants, link to study website, etc.) and the incentives available for recipients who recruit others. When completing the study, participants indicate who alerted them to the research, allowing the most successful 'recruiters' to be identified and rewarded. A social incentive is introduced, because participation will assist the recruiter (i.e. a relative, friend or colleague) in winning the recruitment prize. Incentivised snowballing should therefore result in studies being publicised more widely than where no real recruitment incentives are in place.

My experiences suggest that incentivised snowballing promotes rapid, low-cost recruitment to IMR. In one study, non-diabetic adult participants with no eating disorder history who completed a 64-item baseline questionnaire and nine further items a week later were entered into a £25 cash prize draw (B. Gardner & Abraham, 2009). A separate £25 cash prize was offered to the person who enrolled most study completers. Two researchers e-mailed details to friends, family and colleagues, and in the first week 94 participants completed the baseline questionnaire. In total 266 baseline participants (202 of whom completed the study) were recruited over 40 days. This period included 17 days on which data collection slowed to three or fewer baseline responses; we subsequently re-sent the advert to original recipients, which stimulated renewed recruiter activity. The best recruiter enlisted 16 participants.

I have also used incentivised snowballing in parallel with traditional recruitment methods (B. Gardner & Abraham, in press). A paper questionnaire mailed with a prepaid reply envelope to 1200 car drivers in Brighton and Hove prompted only 85 responses over four months, despite three £50 cash prizes on offer to participants. Incentivised snowballing recruited 105 local drivers in 21 days to an online version of the questionnaire, and the leading recruiter received £25 for enrolling seven participants.

Of course, data collection speed using incentivised snowballing will depend on the inclusiveness of participant criteria. Also, reliance on online social networks makes the technique unsuitable for recruiting from certain populations (e.g. those with limited internet access, though the principle of incentivised recruitment could be applied to non-IMR). Additionally, growing membership of web-based networks makes it difficult to anticipate biases arising from sampling exclusively from these groupings, and potential ethical issues surrounding rewarding members of larger social systems should also be considered. Ethical problems associated with e-mailing unsolicited study advertisements (Hewson, 2003) can be minimised by requesting that study details are forwarded only to known others, and unethical recruiter activity (e.g. pressuring others to participate, spam mails to unknown recipients) should be explicitly discouraged in a code of recruiter practice attached to the e-mail. The researcher however retains responsibility for recruiter behaviour and must provide contact details so as to accommodate potential complaints about unscrupulous recruiters. Potential pros and cons should be considered carefully when deciding whether to use incentivised snowballing (see Table 2).

Table 2. Some possible advantages and disadvantages of using incentivised snowballing for internet-based research
----------------------------------------------------------------- 

Advantages: May reach outside of usual mailing list networks, engaging those otherwise unlikely to participate
Disadvantages: Sampling exclusively from online social networks gives rise to biases not yet properly understood
------------------------------------------------------------------
Advantages: Can promote rapid data collection
Disadvantages: Success depends on inclusiveness of participant criteria and whether target populations are accessible online
-----------------------------------------------------------------
Advantages: Little recruiter effort required
Disadvantages: Limited researcher control over who is recruited, or recruiter behaviour (e.g. coercing others into participation, spam mailing)
-------------------------------------------------------------------
Advantages: Ensures recruitment e-mails are not sent by an unknown source, so minimising the likelihood of offending recipients
Disadvantages: May be unethical to reward members for size of online social networks
------------------------------------------------------------------- 

Further work is needed to rigorously assess the usefulness and limitations of incentivised snowballing, but my experiences and preliminary feedback from colleagues suggest that it can be effective for recruiting to IMR studies. The technique should be thought of as compatible with and complementary to existing IMR recruitment methods (see K. Gardner, 2007).

Benjamin Gardner is with the Health Behaviour Research Centre at University College London [email protected]
 

A new power calculator

Rory Allen and Ian Hannent on a new computerised tool

Power is an important concept in statistics, if only for the very practical reason that many grant-giving bodies now require a minimum power of 80 per cent to be built into the design of any study eligible for funding. At the same time, though the idea is essentially quite simple, standard textbooks often make the calculations appear mysterious. The provision of computer programs or tables to enable the necessary calculations, though functional, provides no insight into what is going on.

The development of a visual aid or 'nomogram' by Douglas Altman was a major step forward, making the calculation – at least for a typical independent groups scenario – almost instantaneous. But Altman's version used only two alpha levels (both 2-tailed), and given the design, the use of even two levels made the diagram somewhat overcrowded. Moreover, it provided no insights into the underlying principles on which it operated.

It seemed to us that it might be useful to develop an approach to power calculations that overcame these difficulties. It should be sufficiently versatile to accommodate some flexibility over alpha values, but should also be of value heuristically.

It would need to be self-explanatory enough to enable researchers to use it as a quick and easy sample size calculator, but would also allow students to get to grips with the power concept by experimenting with different combinations of the four variables alpha, effect size, sample size and power, thus gaining the confidence that can only be acquired by hands-on familiarity. This program is now available on www.paidresearch.org/powercalculator. We hope that the program, along with the accompanying quick guide and more detailed explanation of principle, will be helpful to researchers, and useful as a means of familiarising students with a concept that is as theoretically interesting as it is practically useful.

Rory Allen is with the Department of Psychology, Goldsmiths, University of [email protected]

Ian Hannent is with the Department of Psychology, Goldsmiths, University of London