Please use our GitHub Issues page for all new support inquiries. If you need to contact us directly, please e-mail [email protected]

Hi! I would like to check for interrater reliability between two independent raters on the same variable. Does anyone know what script I should use for this? (Including any preliminary steps, as I am completely new to writing scripts.) Thanks!

asked 21 Jun '13, 18:00

cap317's gravatar image

accept rate: 0%

edited 21 Jun '13, 18:02

To be more specific, a variable that I would like to code is nominal and ranges from 0 to 3. Observations will be coded in 15-s intervals.

(22 Jun '13, 09:23) cap317 cap317's gravatar image

Are both coders coding 100% of the data? If so you can match the two columns by ordinal number. If not, onsets and offsets for each cell should suffice. The latter option would work if they both coded 100% as well. The script will be slightly different depending on how the secondary coder coded. I can send you a example script depending on how much and how the secondary coder coded.

permanent link

answered 26 Jun '13, 11:06

david's gravatar image

david ♦
accept rate: 4%

Yes, 100% of the data will be coded by both raters. Thanks very much for offering to send an example script--that would be much appreciated!

(27 Jun '13, 22:45) cap317 cap317's gravatar image

One more question, so we have an idea of how your data is set up. Is there something in the primary coder's cell the you know is the same in the reliability coder's cell? For example, will the cells you want to compare (to see if primary and rel coders agreed) always have the same onset or the same trial number? Or will they always have the same ordinal (i.e., will the primary and reliability coders always have the same number of cells)?

If there is something that can be counted on to be the same in the primary coder's cell and the reliability coder's cell, I can send you an example script that uses that to identify which cells should be compared. If there's nothing to match the cells on like that, it's a different situation, and I'll need to know more about how you coded.

permanent link

answered 02 Jul '13, 17:27

WhitneyCole's gravatar image

WhitneyCole ♦
accept rate: 21%

Yes, the raters will be coding segments identified as the same. Also we are hoping to use the program to input our independent raters' codes--so we would be interested in both entering codes and calculating interrater reliability for two or more raters.

We would really appreciate if you could send us an example script to do these procedures. Thanks so much!

(09 Jul '13, 08:19) cap317 cap317's gravatar image

Below are links to 3 scripts that will create your primary and reliability columns and check the interrater reliability. The vast majority of each script is api that allows ruby to talk to openshapa, so you can ignore it entirely. At the very bottom is the "User Editable Section" where you can change the scripts to specifically fit your coding needs. The text adjacent to the #s is meant to help explain what the script is doing and how you can change it to suit your files.

The Create Primary Column Script does not add cells at 15 second intervals. That can be done manually and easily. For example if you have 45 seconds of audio for a given file you can create 3 cells and assign them onsets and offsets accordingly and use the snap to region tool that Gladys described in the other post. The Create Reliability Column Script should be run after the primary coder has finished coded a given file.

Create Primary Column Script

Create Reliability Column Script

Check Reliability Script

permanent link

answered 10 Jul '13, 15:30

david's gravatar image

david ♦
accept rate: 4%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here



Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text]( "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:


question asked: 21 Jun '13, 18:00

question was seen: 8,809 times

last updated: 10 Jul '13, 15:30

powered by OSQA