Systematic reviews 101: How to phrase your research question

As promised, and first thing’s first, when writing a systematic review, how should we phrase our research question? This is useful when phrasing questions for individual studies too.

PICO is a useful mnemonic for building research questions in clinical science:

• Patient group
• Intervention
• Comparison/Control group
• Outcome measures

How does this look in practice?

What is the effect of [intervention] on [outcome measure] in [patient group] (compared to [control group])?

How can we make this more applicable for language evolution?

I guess we can change the mnemonic:

Population (either whole language populations in large scale studies, small sample populations either in the real world or under a certain condition in a laboratory experiment, or a population of computational or mathematical agents or population proxy)

Intervention
Comparison/Control group
Outcome measures

Here are some examples of what this might look like using language evolution research:

What is the effect of [L2 speakers] on [morphological complexity] in [large language populations] compared to [small language populations]?

What is the effect of [speed of cultural evolution] on [the baldwin effect] in [a population of baysian agents]?

What is the effect of [iterated learning] on [the morphosyntactic structure in an artificial language] in [experimental participants]?

What is the effect of [communication] on the [distribution of vowels] in [a population of computational agents]?

All of the above are good research questions for individual studies, but I’m not sure it would be possible to do a review on any of the above research questions simply because there is not enough studies, and even when studies have investigated the same intervention and outcome measure, they haven’t used the same type of population.

In clinical research the same studies are done again and again, with the same disease, intervention and population. This makes sense as one study does not necessarily create enough evidence to risk people’s lives on the results. We don’t have this problem in language evolution (thank god), however I feel we may suffer from a lack of replication of  studies. There has been quite a lot of movement recently (see here) to make replication of psychological experiments encouraged, worthwhile and publishable. It is also relatively easy to replicate computational modelling work, but the tendency is to change the parameters or interventions to generate new (and therefore publishable) findings. And real world data is a problem because we end up analysing the same database of languages over and over again. However, I suppose controlling for things like linguistic family, and therefore treating each language family as its own study, in a way, is a sort of meta-analysis of natural replications.

I’m not sure there’s an immediate solution to the problems I’ve identified above, and I’m certainly not the first person to point them out, but thinking carefully about your research question before starting to conduct a review is very useful and excellent practice, and you should remember that when doing a systematic review, the narrower your research question, the easier, more thorough and complete your review will be.