Fun Language Experiment: Results

Two days ago I ran a pilot experiment online from Replicated Typo.  Thanks to all who took part. It’s a bit cheeky to exploit our readers, but it’s all in the name of science.  Unfortunately, the pilot was a complete failure. Suggestions and comments are welcome.

The experiment was into the role of variation in language learning.  Here’s what I was up to (plus the source code for running similar experiments):

Language learners must learn to detect conditioned variation at several levels of abstraction in their language input (phonetic, phonological, lexical, syntactic etc.).  Although there may be much variation between speakers at lower levels (e.g. accent) the more abstract levels (e.g. lexical, syntactic) are set by consensus between speakers.  Receiving input from multiple speakers may help learners by hi-lighting relevant variation.  For example, receiving input from one speaker, it may not be clear which variation is idiosyncratic to the speaker and which is systematic for the population.  By comparing variation between speakers, variation which is conditioned by the context, and not the speaker, is made more obvious.  Specifically, higher-order conditioned variation should be easier to infer from multiple speakers.  This might also be related to the reported bilingual advantage in metalinguistic awareness (e.g. Bialystok, 1991).

Gomez (2002) shows that increased variation in adjacent dependencies (probabilities of words occurring next to each other) can cause learners to pick up on long-distance dependencies (probabilities of words occurring in the same sentence, but with other words in between).  I hypothesise that input from multiple speakers gives better cues to higher-order dependencies than input from one speaker alone.

Materials

The pilot was based on Gomez (2002).  Participants listened to strings from an artificial language with consistent long-range dependencies spoken by either one or two speakers.  They were then tested on strings they had previously heard and novel strings that violated the dependencies.  The prediction was that listeners exposed to two speakers would learn the long-range dependencies better, and therefore score higher in the test phase.

Two artificial languages were created using the following grammar:

L1
S -> aXg
S -> bXh
S -> dXi
S -> eXj
S -> fXk
L2
S -> aXh
S -> bXi
S -> dXj
S -> eXk
S -> fXg

Where a = ‘pel’, b=’rot’, c =’dak’, d = ‘rud’, e = ‘jic’, f = ‘tood’, g = ‘chila’, h = ‘plizet, i= ‘feenam’, j = ‘deecha’.  X was drawn from five two-syllable words (‘wadim’, ‘fengle’, ‘coomo’, ‘loga’, ‘gople’).  These words come from Gomez (2002).  This gives a total of 25 strings in each langauge.  The languages have the same adjacent dependencies, but different long-distance dependencies.

Two speakers (one male, one female with different regional accents) were recorded saying each of the strings, with a short pause between each word and a one second pause between each string.

Procedure

20 Participants were recruited from the web.  Participants were told that they would be listening to sentences from an artificial language.  They were told that they would be tested on the language, but no specifics were given.

There were two training conditions.  In the one-speaker condition, participants heard the 25 strings from L1 spoken by the same speaker (different recordings of the same string for each repetition).  In the two-speaker condition, participants heard the same sequence of strings, but each alternative sentence was spoken by a different speaker.

Participants then listened to twelve strings from the same speaker (in both conditions), six of which came from L1 and six of which came from L2.  The order of strings was randomised.  Participants indicated after each string whether they had heard it before or not.  The number of correct answers was recorded.

Results

Participants in the two conditions did not score significantly differently from each other or from chance (mean for one-person condition = 53%, mean for two-person condition = 52%, t = 0.2, p = 0.8).  In both conditions, participants scored higher on true positives than true negatives.  This suggests that they thought all the presented sentences were familiar.  Participants in the two-person condition did score marginally higher on false positives (mean for one-person = 23%, two-person = 35%, t = 1.5, p = 0.16).  That is, they were better at detecting sentences that they had not heard before.

Discussion

It’s embarrassing to put up negative results, but that’s what I get for doing it all in the public arena.

This pilot follows a previous one which had fewer ‘words’.  The results were similar, but a few participants scored perfectly, so I made this pilot more difficult.  It seems as if I made it too difficult.  The participants in Gomez (2002) had 18 minutes of training for only three long distance dependencies, while the ones in this experiment had about one and a half minutes of training for five long distance dependencies.  I decided that I couldn’t trust participants that were recruited online to listen to more than a few minutes of training data.

I’m not convinced that the experiment won’t work, but it’s back to the drawing board.  Any suggestions or comments are most welcome.

References

Gómez RL (2002). Variability and detection of invariant structure. Psychological science : a journal of the American Psychological Society / APS, 13 (5), 431-6 PMID: 12219809

Source Code for Experiment

The following php code creates a form with a hidden element which stores the progress of the participant (number of rounds completed, order of stimuli, time spent doing the experiment etc.).  The script then sends this information and the participant’s button-presses to itself. The results are written to a file on a serverAdd the html and php tags around this source code:

$fileRoot = 'http://myserver.com/Experiments/Pilot/';  # folder location of this file
 $prefix = 'sounds/Test'; # location and prefix of sound files e.g. ./sounds/Test1.mp3
 $nameOfThisFile = 'test3.php';  # name of this file (used for sending data to itself)
 $numTests = 12;
 $numConditions = 4;
 function player($fname){
 $playerA = '<embed type="application/x-shockwave-flash" flashvars="audioUrl=';

 $playerB = '" src="http://www.google.com/reader/ui/3523697345-audio-player.swf" width="400" height="27" quality="best"></embed>';

 echo $playerA.$fname.$playerB;
 }
 function getCondition(){
 global $numConditions;
 $file = fopen("results/log.txt",'r');
 $contents =fread($file,filesize("results/log.txt"));
 if($file){
 return substr_count($contents,"\n") % $numConditions ;
 }
 else{
 return 0;
 #return rand(0,$numConditions-1);  # random first condition
 }
 }

function getRealIpAddr()
{
 if (!empty($_SERVER['HTTP_CLIENT_IP']))   //check ip from share internet
 {
 $ip=$_SERVER['HTTP_CLIENT_IP'];
 }
 elseif (!empty($_SERVER['HTTP_X_FORWARDED_FOR']))   //to check ip is pass from proxy
 {
 $ip=$_SERVER['HTTP_X_FORWARDED_FOR'];
 }
 else
 {
 $ip=$_SERVER['REMOTE_ADDR'];
 }
 return $ip;
}

 $prev = "";
 $order = array();
 $firstTime = false;
 $pointer = 0;
 $result = "";
 $condition = 0;
 $ip=getRealIpAddr();
 if(isset($_POST['prev'])) {
 $prev = explode(";",$_POST['prev']);
 $pointer = (int) $prev[0];
 $order = explode(",",$prev[1]);
 $result = $prev[2];
 $condition = (int) $prev[3];
 $timeStamp = (int) $prev[4];
 $timer = ((int)$prev[5])+(time() - $timeStamp);

 if(isset($_POST['Yes'])) {
 $result = $result."Y";
 }
 if(isset($_POST['No'])) {
 $result = $result."N";
 }
 }
 else{
 $firstTime = true;
 $order = range(1,$numTests);
 shuffle($order);
 $condition = getCondition();

 }
 $timeStamp = strval(time());
 $pointer = strlen($result);
 $orderStr = implode(",",$order);
 $prev = strval($pointer).";".$orderStr.";".$result.";".$condition.";".$timeStamp.";".$timer.";".$ip;
 # echo '<br />'.$order[$pointer];

 if($pointer==$numTests){
 $file = fopen("results/log.txt",'a+');
 fwrite($file,$prev."\n");
 echo 'Thank you for taking part in my experiment!';
 echo '<br /><br />Back to <a href="http://www.replicatedtypo.com">Replicated Typo</a>.';
 $prev= '';
 }
 else{
 echo '<font size="14">Language Experiment</font> <br /><br />';

if($firstTime){

 $conditionFile = 'sounds/A.mp3';
 if($condition == 1){
 $conditionFile = 'sounds/B.mp3';
 }
 if($condition == 2){
 $conditionFile = 'sounds/C.mp3';
 }
 if($condition == 3){
 $conditionFile = 'sounds/D.mp3';
 }

 echo '<b>This Experiment uses sound - turn on your speakers or plug in your headphones!</b><br /><br />';

 echo "This experiment tests your memory.  Don't take notes!<br /><br />";
 echo 'First, you will listen to a number of sentences from an alien language. <br />Listen carefully because you will be tested on them. <br /> Press the play button below and listen <b>only once</b>.<br /> Listen <b>all the way through</b>.<br /><br />Then press "To test" to begin the test. <br />';
 echo '<br />';
 echo '<b>The alien language:</b>';
 echo '<br />';
 player($conditionFile);
}
else{
 echo 'TEST<br />';
 echo 'Recordings of sentences will appear below.  Each sentence may be one you have already heard, or it may be from a different language.<br />';
 echo 'For each sentence, press play to listen to it.  <br />Listen to each sentence <b>ONLY ONCE</b>.<br />';
 echo "Then indicate whether you think you've heard it before.<br />";
 echo '<br />There are 12 test sentences.';
 echo '<br /><br />';
 if($pointer % 2==($condition%2)){
 echo '<div style="background-color:#C0C0C0;width:500px;padding:10px;">';
 }
 else{
 echo '<div style="background-color:#8C8C8C;width:500px;padding:10px;">';
 }
 echo '<b>Sentence '.strval($pointer+1).'</b>';
 echo '<br />';

 player($prefix.strval($order[$pointer]).'.mp3');
 echo '<br /><br />Have you heard this sentence before?<br />';

}

 echo '      <form action="'.$nameOfThisFile.'" method="post">';  # action is name of this file

 echo '      <input type="hidden" name="prev" value="'.$prev.'">';
 echo '<br />';
 if($firstTime){
 echo '<input type="submit" name="OK" value="TO TEST ->">';
 }
 else{
 echo '      <input type="submit" name="Yes" value="Yes">';
 echo '   <input type="submit" name="No" value="No">';
 echo '</div>';
 }
 echo '</form>';
 }