-
Notifications
You must be signed in to change notification settings - Fork 4
R package to aid cleaning/checking/formatting data using Codebooks/Data Dictionaries #46
Comments
Hi Peter, this is a direct hit on my main problem/use case for a new R tool! On Wed, Apr 20, 2016 at 11:25 PM, Peter Baker [email protected]
|
Hi Ivan |
Yep. require(devtools)
source_gist(4497585)
parse.codebook
"
http://jason.bryer.org/codebooks/ccdCodebook.txt
The codebook provided is in plain text and required two modifications: One, general file information at the top of the file was deleted, and two, any descriptions that spanned lines need to be modified so the are on only one line.
"
setwd("~/tools/disentangle/codebooks_jbryer")
ccd.codebook <- parse.codebook(
file = 'ccdCodebook.txt'
,
var.names=c('variable','order','type','description')
,
level.names=c('level','label')
,
level.sep='='
,
var.widths=c(13, 7, 7, Inf)
)
head(ccd.codebook)
str(ccd.codebook)
ccd.var.levels <- attr(ccd.codebook, 'levels')
names(ccd.var.levels)
ccd.var.levels[['FIPST']]
|
Will explore after lunch - been exploring my codebook dryworkflow stuff on github which I haven't looked at for a year - too much on unfortunately but this is perfect excuse. Cheers |
Hi, Regarding my Excel reference: Not a fan of it for my solo work, but it's popular with research coordinators and serves as a common tool for many research teams. Could there be convergence between the data-cleaning systems of
In particular:
tidyverse/dplyr#631 (incidentally, Edwin de Jonge started this thread, but did not comment again - are they taking a different path with their Is this relevant? I look forward to following the developments on this topic. Cheers! |
Hi Tanya, good to hear from you.
Also check out the EML package http://ropensci.github.io/EML/ (on CRAN)
which allows define the attributes of data objects such as geographic
coverage, temporal coverage, type (eg. data.frame, image, spatialPolygon),
missing values, enumerated value lists and numeric ranges, along with
copious amounts of relevant metadata such as context of who, why, IP,
licencing etc etc.
…On Fri, Jan 20, 2017 at 2:56 AM, Tanya Murphy ***@***.***> wrote:
Hi,
Very excited to find this thread. I had thrown together a note for myself
a few days ago, which I included below. I will look at your posts more
closely in a couple of weeks and try to make connections. In the meantime,
you may find these resources interesting (if you haven't already seen them).
Regarding my Excel reference: Not a fan of it for my solo work, but it's
popular with research coordinators and serves as a common tool for many
research teams.
------------------------------
Could there be convergence between the data-cleaning systems of
-
Edwin de Jonge and Mark van der Loo <https://github.com/data-cleaning>
-
Win-Vector "parametrized" data cleaning e.g.
-
Using replyr::let to Parameterize dplyr Expressions – Win-Vector
Blog
<http://www.win-vector.com/blog/2016/12/using-replyrlet-to-parameterize-dplyr-expressions/>
#rlang #data-prep
-
variable treatment – Win-Vector Blog
<http://www.win-vector.com/blog/tag/variable-treatment/> #rlang
#data-prep
-
Parametric variable names and dplyr – Win-Vector Blog
<http://www.win-vector.com/blog/2016/12/parametric-variable-names-and-dplyr/>
-
tidyverse
In particular:
- facilitate using and enhancing Excel-based data dictionary/codebook
that contains (obviously) variable names, labels, description, units (if
applicable), flagging/cleaning rules, factoring/categorization rules for
individual variables. Flagging/cleaning and categorization rules for joint
distributions -- ideally be used as inputs to emerging functions that
replace cumbersome nested ifelse staements.
Rules-based programming - SE
<http://softwareengineering.stackexchange.com/questions/103659/how-can-one-manage-thousands-of-if-then-else-rules>
tidyverse/tidyr#183 <tidyverse/tidyr#183>
tidyverse/dplyr#631 <tidyverse/dplyr#631>
(incidentally, Edwin de Jonge started this thread, but did not comment
again - are they taking a different path with their deducorrect and Data
cleaning packages <https://github.com/data-cleaning>?)
Is this relevant?
ropenscilabs/assertr: Assertive programming for R analysis pipelines
<https://github.com/ropenscilabs/assertr>
------------------------------
I look forward to following the developments on this topic. Cheers!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#46 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABPsp2dHe2Zo31eoqW-0ZKhrTc34VF4Yks5rT4ergaJpZM4ILsFR>
.
|
Subject: R package to aid cleaning/checking/formatting data using Codebooks/Data Dictionaries
Statisticians and scientists producing primary data often have different needs to those scraping secondary and tertiary data off the web.
Often in medical, epidemiological, social, psych and other scientific studies, researchers use codebooks to document data formats, variable labels, factor labels, ranges for continuous variables, details of measuring instruments etc. Sometimes statisticians get a photocopied codebook or pdf but my preference (and that of aware researchers) is a spreadsheet so that these meta data can be used.
For small data sets its probably OK to manually set up factor labels and check for non-defined factor levels and identify out of range values for continuous variables. For data sets with hundreds of variables or when there are many data files with similar structure it is probably better to automate these procedures.
A package for extracting information from codebooks and using the meta data to assist labelling and data cleaning would prove useful to statisticians and scientists at the coal face of producing primary data.
The text was updated successfully, but these errors were encountered: