Data Exploration and Preparation

Data Exploration and Preparation

The Different Types of Data

Quantitative, qualitative, or textual; what the heck?

Quantitative is just a big word for measurable. Most numerical data falls in to this type of data; things like the age or income of customers and the weight of trucks going through a port of entry. Within this type there are two sub-types, discrete (integers) and continuous (infinite, or just very large real numbers)

Qualitative data is data this is finite. These can be numerical or alphanumeric, however, when numerical they are just codes and not quantities, like department numbers.

Text data is data in the form of natural language, like press releases or reports.

Examining the Distribution of Variables

My mind sort of works like a search engine. You ask me something, and I start seeing pictures.

Temple Grandin

Here we’re looking for anomalies in the distribution of the data. For example, in a study of the effects of medication on adolescents the distribution cure shows a number of participants in their 20s.

Get some simple orders of magnitude, like average age and income. Look at the significant quantiles, i.e. 1%, 10%, 25%, 50% (median), 75%, 90% and 99%. Look for data that is out-of-place.

We also need to look for incompatibilities between variables; studies on mice with a median income over $20,000 – maybe not.

Detection of Rare or Missing Values

Rare vales can cause a bias in the analysis. Just like a four-leaf clover is rare, it’s also not as important as it may seem to be. We’re also looking for missing values. These can cause problems because most statistical methods don’t know what to do with them. You could replace it with a similar variable, class it as its own ‘missing value’ class. Even boil em, mash em, stick em in a stew.

Detection of Aberrant Values

You don’t have to be a mathematician to have a feel for numbers.

John Forbes Nash, Jr.

Aberrant values are incorrect values corresponding to an incorrect measurement, a calculation error or other errors. These can be difficult to detect because extreme values are not always aberrant and aberrant values are not always extreme. One method of detecting these values is using frequency tables. For example, data where most ages are on the decade, 20, 30 and so on. This most likely wouldn’t cause variation in the distribution but is the mark of aberrant values.

Detection of Extreme Values

Extreme values are not necessarily aberrant; it may relate to a specific group or category of individuals which may or may not be important to the study. Use a case-by-case approach because an extreme value may relate to a sub-set which is worth analysis.

Tests of Normality

When we use some methods we must test for normality. Some methods only work when this assumption is made. Fun fact, the normal law, or Laplace–Gaussian law is just the jargon term for a bell-curve.

Homoscedasticity and Heteroscedasticity

Depending on the methods used we must check for homoscedasticity and heteroscedasticity. Clear as mud right. It’s just a way to describe the data. Homo – the data has some aspect of sameness, Herero – different.

Choosing Ranges of Values of binned Variables

Binning is a method of grouping populations together, such as age groups (20-30) in place of age (22, 26,27).

Creating New Variables

Depending on the data this may, or may not be helpful. Some examples are combining data of birth and date of loan application to give the applicant’s age at application or individual interest rates to average interest rate for loans made during the month.

Leave a Reply

%d bloggers like this: