Qualitative research - emerging from the cocoon of science

Qualitative research is widespread within the private and public sectors. It informs most commercial marketing campaigns, public service communications and much organisational change. So why is psychology not at the cutting edge of qualitative research? Why does it still have such a low profile within academic psychology?

In the October 2005 edition of The Psychologist, Anna Madill and colleagues highlighted the dearth of highly skilled qualitative researchers within psychology and the growing demand for supervision of student qualitative projects. Little qualitative research is published in leading work psychology journals (Cassell et al., 2006). Qualitative research has roots in the social sciences (Gordon, 1999), so why has it been sidelined by psychologists?

Commercial vs. academic

As a psychologist and qualitative researcher working in the commercial arena for thirty years, I am struck by the differences between academic and commercial qualitative practice. Qualitative research can be traced back to the beginning of the 20th century, possibly earlier. Anthropologists such as Boas, Mead, Malinowski, Bateson and Evans-Pritchard developed a fieldwork method whereby observers immersed themselves within another culture to study the customs, habits, beliefs and behaviours of that society. Subsequently, 'the Chicago School' of the 1920s established the importance of qualitative research in understanding the group life of human beings. These approaches enabled researchers to develop understanding from the perspective of the 'researched'. You could say they represent the beginnings of a social constructionist epistemology.

Over recent decades commercial qualitative research has thrived, due to its proven usefulness for business clients' strategic and tactical decision making. Simultaneously, within some academic circles, qualitative research has been largely ignored because of its perceived lack of 'objectivity' and scientific discipline. Only fairly recently, nudged by a postmodern world view, is it assuming new found respectability within academia. As a result of these different perspectives and usages, coupled with little or no cross fertilisation of thinking, academic and commercial qualitative research practices have evolved independently and divergently. Key areas of difference arise:
-    Whilst commercial qualitative research has seen exponential growth in the last half century – estimates suggest an industry of $2-3 billion per annum worldwide (Imms & Ereaut 2002) – its academic relation has not.
-   Commercial and academic practitioners have little shared understanding; they barely speak the same language. Unlike academia, commercial research is less constrained by a classical science paradigm, so 'subjectivity' (as demonstrated through experience, interpretation, knowledge of market dynamics) is regarded as a strength rather than a weakness. Commercial research is essentially driven by what is 'useful' rather than what is 'true' (although these terms deserve deeper discussion). Meanwhile, in academia, the scientific paradigm prevails and qualitative is the poor relation to quantitative research.
-    Current concerns within the two spheres are very different. Academic research tends to highlight 'validity' and differences between methodological approaches; commercial research is, arguably, moving towards a position of research-based consultancy.

What do commercial qualitative researchers do?

Commercial qualitative research explores people's needs, attitudes, beliefs, fears, behaviour and how these shape and are shaped in relation to others. Although there are many qualitative methodologies, for years focus groups have dominated the commercial arena. Increasingly approaches such as ethnography, semiotics and Neuro Linguistic Programming are also employed.

Typically, projects are commissioned by commercial organisations or government departments who generally define the research objectives, often with researcher input. The research agency prepares a proposal outlining a suggested approach. This is discussed and agreed with the 'client'. The research team then embarks on the fieldwork. Depending on the scale of the research, commissioning to debrief is unlikely to take longer than two months. This may come as a shock to academic researchers, but do not assume that 'fast' necessarily means lack of intellectual rigour!

Commercial researchers are not wedded to particular methodologies. Methodology is a means to an end; the 'end' being actionable knowledge. A mix of (accelerated) grounded theory, discourse (and narrative) analyses and alethic heuristics (a kind of holistic understanding which does not attempt to separate 'objective' from 'subjective' or understanding and explanation: Alvesson & Skoldberg 2000), is typically adopted – although commercial researchers would not use these terms. Where 'grounded theory' is employed, there is rarely the luxury of 'staying in the field until no further evidence emerges' (Goulding, 2002); hypotheses can be held 'lightly', and the point at which a 'good enough' understanding is reached can be sooner in commercial than academic research. If judged to develop or sharpen understanding, a researcher may switch happily between methodologies without too much concern for the purity of the approach. Different methodologies may be adopted at different stages: during the early stages of the research process, there is greater emphasis on capturing and making sense of individual perspectives on an issue. Later, emphasis shifts to making connections, highlighting patterns, creating a broader understanding which will allow the client to make strategic decisions.  

Many commercial qualitative researchers, myself included, favour a social constructionist approach: we argue that data is never without interpretation, that we have to be the eternal sense-makers (Weick, 1995). Analysis of qualitative data is an intensive sense-making process; narrative analysis of transcripts, generating possible understandings, testing hypotheses with colleagues, making creative leaps, challenging thinking through reflectivity, reflexivity, re-interpreting and so on. This is always driven by the key question, 'How can this best help our client make appropriate decisions and implement them?' In this sense, qualitative practitioners are unavoidably consultants as well as researchers; research and its application cannot be separated. When it comes to debriefing research outcomes with the client team, this process of sense-making continues; shaping and developing knowledge to help achieve stated objectives.

Projects are very diverse. I have worked with groups of teenagers to develop an advertising strategy for a government teenage road safety campaign; with staff and customers in a project to create a new shareholder magazine; with a major cosmetic company to develop products and advertising which are relevant and appealing; and recently within a major government department as part of a process of defining and implementing a new internal communications strategy.

Limitations of the current qualitative research paradigm

The differences between commercial and academic qualitative research are significant and should not be underestimated. However, there are also underlying similarities which constrain the growth and full utilisation of qualitative research practice in both worlds.

Qualitative research within both arenas are – to different degrees – still constrained by a traditional scientific paradigm which struggles to make sense of and validate qualitative approaches. In academia it feels as if a lot of effort has gone into trying to justify the qualitative approach's existence, rather than into trying to develop a new paradigm which makes sense of this type of knowledge. In commerce, qualitative researchers are adept at 'sitting on the fence' and juggling two conflicting epistemologies – 'realism' and a variety of 'interpretivist' approaches – at the same time. For instance, clients may insist that the research sample and topic guide are slavishly adhered to (to avoid accusations of limited validity), although it is simultaneously accepted that the researcher is providing an informed 'consultancy' opinion, based on training and past experience as well as present learning.

When crudely interpreted, this traditional scientific model regards research as data gathering (rather than actionable learning) and the researcher as data gatherer (rather than an active sense maker). Often this leads to research being relegated to a back room function, and in both commercial and academic settings there is an under-acknowledgement of its input to decision making. Furthermore, researchers may be excluded from implementation of the strategy that arises from their research, because this is perceived to compromise their 'objectivity'.

In fact the term 'qualitative research' itself suggests a methodological focus, rather than a way of thinking and a route to understanding and decision making. Spotlighting methodology detracts from the (less visible) painstaking and skilled thinking and analysis that is knowledge generation – the purpose of research. Consequently, it is easy for those unfamiliar with the rigour of qualitative practice to assume that it lacks discipline; that 'anyone can do it'. One result has been the frequent denigration of qualitative research in academia and the media.

Within commercial research, there has been growing interest in developing a new paradigm which more accurately reflects the activity we call qualitative research (Ereaut, 2002; Valentine, 2002; Keegan 2005). In parallel, academic research has also been developing new ways of understanding qualitative research (Alvesson & Skolberg, 2000; Shotter, 1993; Marshall, 1999). These approaches attempt to liberate qualitative thinking from the traditional scientific paradigm, but without losing the discipline and rigour associated with the scientific method.

What might a new qualitative research paradigm look like?

A new research paradigm for qualitative research needs to offer an appropriate epistemological understanding whilst simultaneously reflecting the way in which contemporary practitioners work. I will start by briefly exploring some themes that are currently shaping the commercial qualitative research environment.

Research in a complex world
Social constructionism and the complexity sciences (which challenge Newtonian science and emphasise inter-connection, non-linear systems, unpredictability: Lewin, 1993) are increasingly influencing commercial research practice. Researchers inevitably reflect the prevailing world view as well as personal experience (Weick 1995). But it is precisely the rigorous, reflective, reflexive, intuitive, contextualised subjectivity, embodied in excellent qualitative research, which is its greatest strength (Ereaut, 2002).

There is a move towards greater inclusivity; adopting a genuinely cooperative process of knowledge generation including clients, research participants, and researchers. This sidesteps a 'linear' model of research and is conducive to harnessing diverse thinking and different types of knowledge (Stacey, 2003). For instance a researcher may participate in client workshops as 'the voice of the consumer', or clients may participate in consumer focus groups as consumers.

Researching the future
Research is often criticised for being backward focused rather than helping us deal with future issues. Increasingly research needs to throw light on situations that we cannot adequately predict. How can we prepare people for a flu epidemic that may never arrive? How can we anticipate next year's fickle teenage fashions? Addressing these issues demands creativity, intuition, insight which goes far beyond simple data gathering. Just as Alvin Toffler, in his classic book Future Shock, talked about 'education in the future tense' (Toffler 1970), so we need to think about 'research in the future tense'. What does this mean?

In attempting to address these questions, commercial qualitative research is seeing a resurgence of interest in creative research techniques. Approaches such as Creative Workshops (Holmes & Keegan, 1983) and Breakthrough Events (Langmaid & Andrews, 2003) are increasingly being used to position 're-search' as 'future-search'. In these approaches, researchers work with people in groups to re-define problems, generate new ideas, create alternative interpretations. These are approaches which explore possible futures that enable clients to make educated guesses about trends and cultural shifts. For instance, we are currently working with a major insurance company and their customers to anticipate future financial trends. This knowledge will feed into the development of new products and services.

Working at the 'edge of chaos'
'Edge of chaos', simply stated, is the point in a system where there is stability and instability at the same time. There is balance; neither too little nor too much structure (Stacey, 2003). Research situations which we cannot totally control, but which at the same time are contained, may allow new thinking to emerge. Sometimes this can be simply achieved by allowing research participants to set the agenda; issues may emerge which the researcher has not anticipated. Alternatively it may involve some disruption in habitual patterns; asking people to draw their feelings, role play, deferring judgement, or redefining strange ideas into useful ones (Parnes, 1967).

Research as leadership
Life is increasingly about acting authoritatively, making a 'good enough' decision in an uncertain situation, in which there is little knowledge and little time to assimilate it (Griffin, 2002). Leadership and spontaneity are not traditionally associated with research but they are essential if researchers are to contribute to informed decision making, strategic recommendations and creative leaps with limited time and partial resources. Leadership skills need greater emphasis within the qualitative mix.So how might these themes inform a new qualitative research paradigm?

'Emergent Inquiry': A new qualitative perspective?

'Emergent Inquiry' is an understanding of qualitative research, fed by the complexity sciences, social constructionism and current commercial qualitative practice. It aims to push back the boundaries of research, making it more congruent with 'everyday life'; richer, messier, more contradictory. It views scientific method as a useful discipline, not a set of rules. It regards knowledge as emerging from a broad pool of experience and accepts that research is, by definition, a creative and collaborative process.

'Emergent Inquiry' broadly fits within the 'participative inquiry' arena (Reason & Bradbury, 2001), but with no assumption of an 'objective reality' which we are trying to access through a veil of cultural obfuscation, and no claims of being 'emancipatory' or a force for 'good'. It encompasses three interwoven strands:

Emotion as valid research input
The separation of mind, body and emotion, introduced by Descartes in the mid-17th century and now largely discredited by neuroscience, is still embedded within the prevailing research paradigm. Consequently, our opinions, feelings and emotion are concealed inputs to research; we act as if they do not influence us, whilst they unavoidably impact on our research thinking (Damasio, 2000). Instead of attempting to 'stand outside' and minimise emotional 'biases', this approach understands emotion as valid research input. What we feel in a particular research situation is as valid as what we hear. (Gordon & Langmaid 1988). However, discipline and rigour are just as important when analysing 'emotional' content as they are with 'intellectual' content. Research and consultancy are as much experiential and emotional as cerebral. By openly acknowledging the importance of the researcher's emotional input, we broaden the scope of the research process; it becomes closer to 'real life' and thereby enables more relevant knowledge generation.

Generating knowledge as a way of life
We have grown to view 'research' as something separate from 'life' when, in practice, we experience the day-to-day world through research; we observe, make meaning, create connections, test hypotheses, and experiment. I am suggesting that we need to consciously lower the barrier between 'research' and 'life'. This would mean the researcher – along with everyone else involved in the project – keeping their eyes and ears open and gathering clues and inspiration wherever it is to be found. The more we can get out of role – client, employee, researcher, consumer – whilst still using the experiences, knowledge, skills and rigour that are associated with that role, the more connections we can make and the more creative the outcome.  

Research as ongoing process
Where does research begin and end? From an emergent inquiry perspective, qualitative research needs to encompass problem definition as part of the research process.  Academics and process consultants (Schein, 1999) may take this as given, but it is not traditionally part of the commercial researcher role. Process consultants (Schein, 1999) would take this as given but it is not traditionally part of the researcher role. If, however, the role is redefined as 'research consultant' then it is essential. From a process consultancy perspective, the client 'often does not know what she is looking for and indeed should not really be expected to know'. A change of role expectations between clients and researchers is needed before problem definition can be openly acknowledged as a valid research area.

At the end of the project, the traditional research model decrees that the researcher hands over the 'findings' and walks away. To be involved in research implementation is deemed inappropriate; it is in conflict with the 'objectivity' of the researcher. As a result, the researcher, by now a repository of knowledge which is not easily transferred as 'findings', is dismissed and a valuable resource is lost. By contrast, a 'process consultancy' model implies developing research knowledge within the context in which it is relevant, so that learning from the research can be disseminated and developed. For example the commercial researcher may continue to work in the client company to help influence and develop corporate thinking. By contrast, the model for academic research currently views research more as ongoing process than 'event'.

Time to move on

'Emergent Inquiry' challenges current thinking in both academic and commercial qualitative practice. It aims to develop an understanding of qualitative knowledge generation on its own terms rather than shoehorning it into an existing scientific paradigm. The approach may sit uncomfortably with 'research'. Whilst it may be easier to rely on traditional research boundaries, in doing so, we undervalue the considerable strengths of qualitative practice.

Qualitative research – or more specifically, qualitative thinking – has never been more relevant in a business and academic context. We need to develop and make better use of the wide range of skills and abilities inherent in qualitative practice. If we still cling – overtly or covertly – to the coat tails of traditional scientific method, even though science itself has moved on, then we lose the opportunity to broaden our understanding of knowledge generation and its practical application.

- Dr Sheila Keegan is a business psychologist and co-founder of Campbell Keegan Ltd, based in London. E-mail: [email protected].

Weblinks
Association for Qualitative Research: www.aqr.org.uk
Social Research Association: www.the-sra.org.uk
The Market Research Society: www.mrs.org.uk
Independent Consultants Group: www.indepconsultants.org

Discuss and debate
How can we train qualitative research practitioners from the perspective of emergent inquiry?
How do we maintain research rigour and discipline without clear external standards?
Where does this leave validity and reliability? Isn't this just sloppy research?
Is there such a thing as 'truth' in research outcomes?
Where is the distinction between research and opinion?
Why would commercial approaches to qualitative research have any relevance in an academic environment?
Have your say on these or other issues this article raises. E-mail 'Letters' on [email protected] or contribute to our forum via www.thepsychologist.org.uk.

References

Alvesson, M. & Skoldberg, K. (2000). Reflexive methodology: New vistas for qualitative research.  London: Sage.
Cassell, C., Bishop, V., Symon, G., Johnson, P. & Buehring, A. (2006). Enhancing the quality of qualitative research in the work psychology field. Paper presented at the Division of Occupational Psychology Annual Conference, January, Glasgow.
Damasio, A. (2000). The feeling of what happens. London: Vintage.
Earls, M. (2002). Welcome to the creative age: Bananas, business and the death of marketing. Chichester: Wiley.
Ereaut, G. (2002). Analysis and interpretation in qualitative market research. London: Sage.
Gordon, W. (1999). Goodthinking: A guide to qualitative research. Henley-on-Thames: Admap.
Gordon, W. & Langmaid, R. (1988). Qualitative market research: A practitioner and buyers guide, London: Gower.
Goulding, C. (Ed.) (2002). Grounded theory: A practical guide for management, business and market researchers.  London: Sage.
Griffin, D. (2002). The emergence of leadership: Linking self organization and ethics. London: Routledge.
Holmes, C. & Keegan, S. (1983). Current and developing creative research methods in new product development. Paper given at the MRS Annual Conference, Brighton.
Imms, M. & Ereaut, G. (2002). An introduction to qualitative market research. London: Sage.
Keegan, S. (2005). Emergent inquiry: The 'new' qualitative research Paper given at the AQR/QRCA BiAnnual Conference, Dublin.
Keegan, S. (2006). Science vs imagination: A house divided against itself cannot stand. Keynote paper given at the Australian Market and Social Research Society.
Langmaid, R. & Andrews, M. (2003). Breakthrough zone: Harnessing consumer creativity for business innovation. Chichester: Wiley.
Lewin, R. (1993). Complexity: Life at the edge of chaos. London: Phoenix.
Marshall, J. (1999). Living life as inquiry. In Systemic Practice and Action Research, Vol 12, pp155–171.
Parnes, S.J. (1967). Creative behaviour guidebook. New York: Charles Scribner's Sons.
Reason, P. & Bradbury, H. (2001).The handbook of action research: Participative inquiry and practice. London: Sage.
Schein, E. (1999). Process consultation revisited. Boston, MA: Addison-Wesley.
Shotter, J. (1993). Conversational realities: Constructing life through language. London: Sage.
Stacey, R. (2003). Complexity and group processes: A radically social understanding of individuals. New York: Brunner-Routledge.
Tofler, A. (1970). Future shock. London: Pan.
Valentine, V. (2002). Repositioning research: A new MR language model. International Journal of Market Research Quarter, 2.
Weick, K.E. (1995). Sensemaking in organisations. London: Sage.