# Data Science and Hadoop: Part 4, Outlier Analysis

## Context

This is the fourth part of a 5 part series on analyzing data with PySpark:

# Outlier Analysis

Generating summary statistics is very helpful for

• Understanding the overall shape of the data
• Looking at trends at the extreme (answering most, least, etc style questions)

One thing we’re doing when we’re looking for fishy things in data is looking to quickly zoom in on things which are outside of the norm. Furthermore, the aim is to automatically tag these as data is ingested so they can be acted on. This action can be raising an alert, logging a warning or generating a report, but ultimately we want a technique to find these outliers quickly and without human intervention.

## Median Absolute Deviation

We’re looking to create a mechanism to rank data points by their likelihood of being an outlier along with a threshold to differentiate them from inliers.

The area of outlier analysis is a vibrant one and there are quite a few techniques to choose from. Ranging from the exotic, like fitting an elliptic envelope to the straightforward, like setting a threshold based on standard deviations away from the mean. For our purposes, we’ll choose a middle path, but be aware that there are book-length treatments of the subject of outlier analysis.

Median Absolute Deviation is a robust statistic used, as standard deviations, as a measure of variability in a univariate dataset. It’s definition is straightforward:

Given univariate data $X$ with $\tilde{x}=$median($X$), MAD($X$)=median({$\forall x_i \in X \lvert |x_i - \tilde{x}|$}).

As compared to standard deviation, it’s a bit more resilient to outliers because it doesn’t have a square weighing large values very heavily. Quoting from the Engineering Statistics Handbook:

The standard deviation is an example of an estimator that is the best we can do if the underlying distribution is normal. However, it lacks robustness of validity. That is, confidence intervals based on the standard deviation tend to lack precision if the underlying distribution is in fact not normal.

The median absolute deviation and the interquartile range are estimates of scale that have robustness of validity. However, they are not particularly strong for robustness of efficiency.

If histograms and probability plots indicate that your data are in fact reasonably approximated by a normal distribution, then it makes sense to use the standard deviation as the estimate of scale. However, if your data are not normal, and in particular if there are long tails, then using an alternative measure such as the median absolute deviation, average absolute deviation, or interquartile range makes sense.

In the implementation we’ll be taking guidance the Engineering Statistics Handbook and Iglewicz and Hoaglin. As such, define an outlier like so:

For a set of univariate data $X$ with $\tilde{x} =$ median($X$), an outlier is an element $x_i \in X$ such that $M_i = \frac{0.6745(x_i − \tilde{x} )}{MAD(X)} > 3.5$, where $M_i$ is denoted the modified Z-score.

Before we jump into the actual algorithm, we create some helper functions to make the code a bit more readable and allow us to display the outliers.

Now, onto the implementation of the outlier analysis. But, before we start, I’d like to make a couple notes about the implementation and possible scalability challenges going forward.

We are partitioning the data by (payment reason, physician specialty). I do not want to analyze outliers based on a cohort of data across a whole reason, but rather I want to know if a point is an outlier for a given specialty and reason.

If a coarser partitioning strategy is taken or the amount of data per partition becomes very large, the median implementation may become a limiting factor scalability wise. There are a few things to do, including using a tighter implementation (numpy’s implementation could be tighter as of this writing) or a streaming estimate. Needless to say, this is something that bears some thought going forward.

Now that we have found the outliers per specialty/reason partition, and a sample of inliers, let’s display them so that we can get a sense of how sensitive the outlier detection is.

## Food and Beverage Purchase Outliers

Let’s look at the top 4 outliers for Food and Beverage payments. I could have shown all of the outliers, but I found the first few to be the biggest bang for our buck in terms of interesting findings.

As we can see, the misclassified data from Teleflex is rearing its head again with a huge single payment for food. However, looking down the list, Biolase is paying quite a bit to some dentist for food.

Physician Specialty Payer Amount
200720 Allopathic & Osteopathic Physicians/ Surgery Teleflex Medical Incorporated $68,750.00 28946 Dental Providers/ Dentist/ General Practice BIOLASE, INC.$13,297.15
28946 Dental Providers/ Dentist/ General Practice BIOLASE, INC. $8,111.82 28946 Dental Providers/ Dentist/ General Practice BIOLASE, INC.$8,111.82

Below is a density plot of outliers and a sample of inliers with a rug plot and the outliers marked in red. You can see how far along the tail of the densitiy plot the outliers here are. Most food and payment data hovers much closer to $0$.

## Travel and Lodging Outliers

All I can say is that Physician 106320 travels far more than I do to get paid $155k in 2013. Hope that triple platinum on Delta is worth it. :) Physician Specialty Payer Amount 106320 Allopathic & Osteopathic Physicians/ Psychiatry & Neurology/ Neurology Boehringer Ingelheim Pharma GmbH & Co.KG$155,772.00
472722 Allopathic & Osteopathic Physicians/ Internal Medicine/ Nephrology Merck Sharp & Dohme Corporation $75,000.00 371379 Allopathic & Osteopathic Physicians/ Orthopaedic Surgery Exactech, Inc.$65,798.00
198801 Allopathic & Osteopathic Physicians/ Internal Medicine/ Cardiovascular Disease Medtronic Vascular, Inc. $41,232.80 382697 Allopathic & Osteopathic Physicians/ Internal Medicine/ Nephrology Genentech, Inc.$39,978.80
169095 Allopathic & Osteopathic Physicians/ Surgery Medtronic Vascular, Inc. $37,683.00 80052 Allopathic & Osteopathic Physicians/ Family Medicine Boehringer Ingelheim Pharma GmbH & Co.KG$24,911.25
202461 Allopathic & Osteopathic Physicians/ Thoracic Surgery (Cardiothoracic Vascular Surgery) Covidien LP $21,594.51 378722 Allopathic & Osteopathic Physicians/ Internal Medicine GlaxoSmithKline, LLC.$20,112.40
243205 Allopathic & Osteopathic Physicians/ Internal Medicine/ Interventional Cardiology Medtronic Vascular, Inc. $19,273.90 You can see on the density plot that the next nearest outlier is pretty far away and we have clumps of outliers around 20k and 40k. Interesting things to look into. ## Consulting Fee Outliers Looking at consulting fee outliers, you can see some clumping, but most of the data is lower than 50k. That makes the 200k outlier from Teva all that much more interesting. Of course, none of this is any indication of wrong-doing, just interesting spikes in the data. Physician Specialty Payer Amount 104930 Allopathic & Osteopathic Physicians/ Psychiatry & Neurology/ Neurology Teva Pharmaceuticals USA, Inc.$207,500.00
151515 Other Service Providers/ Specialist Alcon Research Ltd $150,000.00 309376 Allopathic & Osteopathic Physicians/ Internal Medicine Teva Pharmaceuticals USA, Inc.$137,559.67
231913 Allopathic & Osteopathic Physicians/ Orthopaedic Surgery Exactech, Inc. $108,125.00 465481 Allopathic & Osteopathic Physicians/ Internal Medicine/ Rheumatology Vision Quest Industries Inc.$102,196.09
409799 Allopathic & Osteopathic Physicians/ Internal Medicine/ Endocrinology, Diabetes & Metabolism Pfizer Inc. $100,000.00 206227 Allopathic & Osteopathic Physicians/ Orthopaedic Surgery DePuy Synthes Sales Inc.$93,750.00
436192 Allopathic & Osteopathic Physicians/ Internal Medicine Pfizer Inc. $90,000.00 306965 Allopathic & Osteopathic Physicians/ Psychiatry & Neurology/ Neurology Teva Pharmaceuticals USA, Inc.$64,125.00
163888 Allopathic & Osteopathic Physicians/ Internal Medicine/ Cardiovascular Disease Boehringer Ingelheim Pharmaceuticals, Inc. $61,025.00 You can see most of the density is less than 20k, which makes that 200k outlier so interesting. ## Gift Outliers Gifts, I think, are the most interesting payment reasons in this whole dataset. I am intrigued when a physician might get a gift versus an outright fee. I imagined, going in, that gifts would be low-value items, but the table clearly shows that dentists are getting substantial gifts. What is most interesting to me is that all of the top 10 outliers are dentists. Physician Specialty Payer Amount 225073 Dental Providers/ Dentist/ General Practice Dentalez Alabama, Inc.$56,422.00
167931 Dental Providers/ Dentist DENTSPLY IH Inc. $8,672.50 380517 Dental Providers/ Dentist DENTSPLY IH Inc.$8,672.50
380073 Dental Providers/ Dentist/ General Practice Benco Dental Supply Co. $7,570.00 403926 Dental Providers/ Dentist A-dec, Inc.$5,430.00
429612 Dental Providers/ Dentist PureLife, LLC $5,058.72 404935 Dental Providers/ Dentist A-dec, Inc.$5,040.00
8601 Dental Providers/ Dentist/ General Practice DentalEZ, Inc. $3,876.35 385314 Dental Providers/ Dentist/ General Practice Henry Schein, Inc.$3,789.99
389592 Dental Providers/ Dentist/ General Practice Henry Schein, Inc. \$3,789.99

## Up Next

Next, we look for anomalies in our payment data by using Benford’s Law. This is part of a broader series of posts about Data Science and Hadoop.

24 October 2014 Cleveland, OH