- \n
- \n
<\/a>Bad data is worse than no data at all.<\/li>\n
- What is \u201cgood\u201d data and where do you find it?<\/li>\n
- Best practices for data analysis.<\/li>\n<\/ul>\n
There\u2019s no such thing as perfect data, but there are several factors that qualify data as good [1]:<\/p>\n
- \n
- It\u2019s readable and well-documented,<\/li>\n
- It\u2019s readily available. For example, it\u2019s accessible through a trusted digital repository.<\/li>\n
- The data is tidy and re-usable by others with a focus on ease of (re-)executability and reliance on deterministically obtained results [2].<\/li>\n<\/ul>\n
Following a few best practices will ensure that any data you collect and analyze will be as good as it gets.<\/p>\n
1. Collect Data Carefully<\/strong><\/p>\n
Good data sets will come with flaws, and these flaws should be readily apparent. For example, an honest data set will have any errors or limitations clearly noted. However, it\u2019s really up to you, the analyst, to make an informed decision about the quality of data once you have it in hand. Use the same due diligence you would take in making a major purchase: once you\u2019ve found your \u201cperfect\u201d data set, perform more web-searches with the goal of uncovering any flaws.<\/p>\n
Some key questions to consider [3] :<\/p>\n
- \n
- Where did the numbers come from? What do they mean?<\/li>\n
- How was the data collected?<\/li>\n
- Is the data current?<\/li>\n
- How accurate is the data?<\/li>\n<\/ul>\n
Three great sources to collect data from<\/p>\n
US Census Bureau<\/strong><\/p>\n
U.S. Census Bureau data is available to anyone for free. To download a CSV file:<\/p>\n
- \n
- Go to data.census.gov[4]<\/li>\n
- Search for the topic you\u2019re interested in.\u00a0<\/li>\n
- Select the \u201cDownload\u201d button.<\/li>\n<\/ul>\n
The wide range of good data held by the Census Bureau is staggering. For example, I typed \u201cInstitutional\u201d to bring up the population in institutional facilities by sex and age, while data scientist Emily Kubiceka used U.S. Census Bureau data to compare hearing and deaf Americans [5].<\/p>\n
Data.gov<\/strong><\/p>\n
Data.gov [6] contains data from many different US government agencies including climate, food safety, and government budgets. There’s a staggering amount of information to be gleaned. As an example, I found\u00a040,261 datasets\u00a0 for “covid-19” including:<\/span><\/p>\n
- \n
- Louisville Metro Government estimated expenditures related to COVID-19.\u00a0<\/span>\n<\/li>\n
- State of Connecticut statistics for Connecticut correctional facilities.<\/li>\n
- Locations offering COVID-19 testing in Chicago.<\/li>\n<\/ul>\n
Kaggle<\/strong><\/p>\n
Kaggle [7] is a huge repository for public and private data. It\u2019s where you\u2019ll find data from The University of California, Irvine\u2019s Machine Learning Repository, data on the Zika virus outbreak, and even data on people attempting to buy firearms.\u00a0 Unlike the government websites listed above, you’ll need to check the license information for re-use of a particular dataset. Plus, not all data sets are wholly reliable: check your sources carefully before use.<\/p>\n
2. Analyze with Care<\/strong><\/p>\n
So, you\u2019ve found the ideal data set, and you\u2019ve checked it to make sure it\u2019s not riddled with flaws. Your analysis is going to be passed along to many people, most (or all) of whom aren\u2019t mind readers. They may not know what steps you took in analyzing your data, so make sure your steps are clear with the following best practices [3]:<\/p>\n
- \n
- \nDon\u2019t<\/strong> use X, Y or Z for variable names or units. Do<\/strong> use descriptive names like \u201c2020 prison population\u201d or \u201cNumber of ice creams sold.\u201d<\/li>\n
- \nDon\u2019t<\/strong> guess which models fit. Do<\/strong> perform exploratory data analysis, check residuals, and validate your results with out-of-sample testing when possible.<\/li>\n
- \nDon\u2019t<\/strong> create visual puzzles. Do<\/strong> create well-scaled and well-labeled graphs with appropriate titles and labels. Other tips [8]: Use readable fonts, small and neat legends and avoid overlapping text.<\/li>\n
- \nDon\u2019t<\/strong> assume that regression is a magic tool. Do test for linearity and normality, transforming variables if necessary.<\/li>\n
- \nDon\u2019t<\/strong> pass on a model unless you know exactly what it means. Do<\/strong> be prepared to explain the logic behind the model, including any assumptions made.\u00a0\u00a0<\/li>\n
- \nDon\u2019t<\/strong> leave out uncertainty. Do<\/strong> report your standard errors and confidence intervals.<\/li>\n
- \nDon\u2019t<\/strong> delete your modeling scratch paper. Do<\/strong> leave a paper trail, like annotated files, for others to follow. Your predecessor (when you\u2019ve moved along to better pastures) will thank you.<\/li>\n<\/ul>\n
3. Don\u2019t be the weak link in the chain<\/strong><\/p>\n
Bad data doesn\u2019t appear from nowhere. That data set you started with was created by someone, possibly several people, in several different stages. If they too have followed these best practices, then the result will be a helpful piece of data analysis. But if you introduce error, and fail to account for it, those errors are going to be compounded as the data gets passed along.\u00a0<\/p>\n
References<\/strong><\/p>\n
Data set image:\u00a0Pro8055, CC BY-SA 4.0 via Wikimedia Commons<\/p>\n
[1] Message of the day<\/a><\/p>\n
[2] Learning from reproducing computational results: introducing three principles and the Reproduction Package<\/a><\/p>\n
[3] How to avoid trouble:\u00a0\u00a0principles of good data analysis<\/a><\/p>\n
\u00a0[4] United States Census Bureau<\/a><\/p>\n
[5] Better data lead to better forecasts<\/a><\/p>\n
[6] Data.gov<\/a><\/p>\n
[7] Kaggle<\/a><\/p>\n
[8]Twenty rules for good graphics<\/a><\/p>\n<\/div>\n
Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
- \nDon\u2019t<\/strong> guess which models fit. Do<\/strong> perform exploratory data analysis, check residuals, and validate your results with out-of-sample testing when possible.<\/li>\n
- Louisville Metro Government estimated expenditures related to COVID-19.\u00a0<\/span>\n<\/li>\n