Announcing the new Datavyu v1.3.6 with enhanced features and transcription support!

Hi all,

This summer several students and interns in my lab have been using OpenSHAPA familiar with it. I think it is overall going quite well, but I wanted to write and ask about one particular feature that's been hindering adoption.

Basically, when we do data analysis we typically look for data in CSV format that is what you might think of as standard tabular form. E.g., it has a set of columns with headers like subject, trial, and response, and all of these columns have a consistent type. For analysis packages like R (our main one), but also Excel, SPSS and matlab, this makes reading in the data really easy.

Unfortunately, when we code in SHAPA and use the export functionality we typically get a CSV with a highly irregular format. For example, the tables are consistent but they don't have consistent headers (e.g. the header has lots of control characters in it about what variable is what and isn't comma separated), and there is often extra information above and below. It seems like we could create a custom ruby script to export each experiment, but this would be a big barrier to entry for most students.

Just to give you a sense of what this looks like, here's a sample:

#4
Posture (MATRIX,true,)-Type|NOMINAL,Orient|NOMINAL
00:00:31:076,00:03:39:600,(si,0)
00:03:39:600,00:03:49:320,(c,0)
00:03:58:969,00:04:06:015,(si,2)
00:04:06:015,00:04:19:605,(c,1)

You can see that there is a number above the table, then the column headers are complex, and then the fields are given in parentheses. All of this is easy to fix, but multiplied by many files it becomes a bit of a drag to edit manually.

What do you do to export data in tabular form? I searched the discussion group and found this:

https://openshapa.org/support/questions/37/how-do-i-export-my-data

but it looks like Clinton suggests CSV and the format (as I described above) seems to need considerable cleanup for each script. The scripts in that post look interesting but maybe more than we need in order to simply export what we have in our coding. Or maybe we are just doing something wrong on the SHAPA setup side.

Any help much appreciated!

Mike

asked 09 Aug '12, 03:30

mcfrank's gravatar image

mcfrank
1111
accept rate: 0%


Hello Michael,

That is just one of many ways to export your data. In the Adolph Lab we usually use Ruby or R scripts to do so. The R functionality is brand new, so it is not as well tested, but in the latest nightly at https://github.com/OpenSHAPA/openshapa/downloads it should be able to run R scripts if you'd like to test it. The data is loaded into a data.frame called "db", so you can just do write.csv(db, "~/Desktop/test.csv") in a .R file and run that from the scripts menu in OpenSHAPA with the current file open. This will print each argument/cell in its own column.

The other way to do it in a nice format is to use Ruby to loop over your data. This way you can aggregate data down/across rows by looping over a nested structure. I am currently finishing up the tutorial on how to do this on our wiki (https://github.com/OpenSHAPA/openshapa/wiki/Ruby-Scripting-Manual) and there is also a git repo for example scripts available here: https://github.com/OpenSHAPA/OpenSHAPA-Example-Scripts on how to print a single file or create a script that will loop over multiple files. All scripts must be run from inside of OpenSHAPA, and the interesting bits are at the bottom of these scripts (the top part is all the API... that will be gone in future versions but we're in the middle of an API update along with the new OpenSHAPA version).

Sorry this is kind of convoluted, we're currently trying a lot of things to streamline this process more (and at least make it so you can get a spreadsheet of your data super easy that is actually usable). In a very near future update this should be added to the nightlies.

Thanks, and let me know if you have any problems or other questions, and again, I apologize that this part of the OpenSHAPA process is currently so confusing.

Jesse Lingeman

OpenSHAPA Developer

permanent link

answered 09 Aug '12, 03:49

Jesse's gravatar image

Jesse ♦
11181015
accept rate: 53%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×24
×13
×10

question asked: 09 Aug '12, 03:30

question was seen: 3,505 times

last updated: 09 Aug '12, 03:49

powered by OSQA