This is the fourth part of a 5 part series on analyzing data with PySpark:
Generating summary statistics is very helpful for
One thing we’re doing when we’re looking for fishy things in data is looking to quickly zoom in on things which are outside of the norm. Furthermore, the aim is to automatically tag these as data is ingested so they can be acted on. This action can be raising an alert, logging a warning or generating a report, but ultimately we want a technique to find these outliers quickly and without human intervention.
We’re looking to create a mechanism to rank data points by their likelihood of being an outlier along with a threshold to differentiate them from inliers.
The area of outlier analysis is a vibrant one and there are quite a few techniques to choose from. Ranging from the exotic, like fitting an elliptic envelope to the straightforward, like setting a threshold based on standard deviations away from the mean. For our purposes, we’ll choose a middle path, but be aware that there are book-length treatments of the subject of outlier analysis.
Median Absolute Deviation is a robust statistic used, as standard deviations, as a measure of variability in a univariate dataset. It’s definition is straightforward:
Given univariate data $X$ with $\tilde{x}=$median($X$), MAD($X$)=median({$\forall x_i \in X \lvert |x_i - \tilde{x}|$}).
As compared to standard deviation, it’s a bit more resilient to outliers because it doesn’t have a square weighing large values very heavily. Quoting from the Engineering Statistics Handbook:
The standard deviation is an example of an estimator that is the best we can do if the underlying distribution is normal. However, it lacks robustness of validity. That is, confidence intervals based on the standard deviation tend to lack precision if the underlying distribution is in fact not normal.
The median absolute deviation and the interquartile range are estimates of scale that have robustness of validity. However, they are not particularly strong for robustness of efficiency.
If histograms and probability plots indicate that your data are in fact reasonably approximated by a normal distribution, then it makes sense to use the standard deviation as the estimate of scale. However, if your data are not normal, and in particular if there are long tails, then using an alternative measure such as the median absolute deviation, average absolute deviation, or interquartile range makes sense.
In the implementation we’ll be taking guidance the Engineering Statistics Handbook and Iglewicz and Hoaglin. As such, define an outlier like so:
For a set of univariate data $X$ with $\tilde{x} =$ median($X$), an outlier is an element $x_i \in X$ such that $M_i = \frac{0.6745(x_i − \tilde{x} )}{MAD(X)} > 3.5$, where $M_i$ is denoted the modified Z-score.
Before we jump into the actual algorithm, we create some helper functions to make the code a bit more readable and allow us to display the outliers.
Now, onto the implementation of the outlier analysis. But, before we start, I’d like to make a couple notes about the implementation and possible scalability challenges going forward.
We are partitioning the data by (payment reason, physician specialty). I do not want to analyze outliers based on a cohort of data across a whole reason, but rather I want to know if a point is an outlier for a given specialty and reason.
If a coarser partitioning strategy is taken or the amount of data per partition becomes very large, the median implementation may become a limiting factor scalability wise. There are a few things to do, including using a tighter implementation (numpy’s implementation could be tighter as of this writing) or a streaming estimate. Needless to say, this is something that bears some thought going forward.
Now that we have found the outliers per specialty/reason partition, and a sample of inliers, let’s display them so that we can get a sense of how sensitive the outlier detection is.
Let’s look at the top 4 outliers for Food and Beverage payments. I could have shown all of the outliers, but I found the first few to be the biggest bang for our buck in terms of interesting findings.
As we can see, the misclassified data from Teleflex is rearing its head again with a huge single payment for food. However, looking down the list, Biolase is paying quite a bit to some dentist for food.
Physician | Specialty | Payer | Amount |
---|---|---|---|
200720 | Allopathic & Osteopathic Physicians/ Surgery | Teleflex Medical Incorporated | $68,750.00 |
28946 | Dental Providers/ Dentist/ General Practice | BIOLASE, INC. | $13,297.15 |
28946 | Dental Providers/ Dentist/ General Practice | BIOLASE, INC. | $8,111.82 |
28946 | Dental Providers/ Dentist/ General Practice | BIOLASE, INC. | $8,111.82 |
Below is a density plot of outliers and a sample of inliers with a rug plot and the outliers marked in red. You can see how far along the tail of the densitiy plot the outliers here are. Most food and payment data hovers much closer to $0$.
All I can say is that Physician 106320 travels far more than I do to get paid $155k in 2013. Hope that triple platinum on Delta is worth it. :)
Physician | Specialty | Payer | Amount |
---|---|---|---|
106320 | Allopathic & Osteopathic Physicians/ Psychiatry & Neurology/ Neurology | Boehringer Ingelheim Pharma GmbH & Co.KG | $155,772.00 |
472722 | Allopathic & Osteopathic Physicians/ Internal Medicine/ Nephrology | Merck Sharp & Dohme Corporation | $75,000.00 |
371379 | Allopathic & Osteopathic Physicians/ Orthopaedic Surgery | Exactech, Inc. | $65,798.00 |
198801 | Allopathic & Osteopathic Physicians/ Internal Medicine/ Cardiovascular Disease | Medtronic Vascular, Inc. | $41,232.80 |
382697 | Allopathic & Osteopathic Physicians/ Internal Medicine/ Nephrology | Genentech, Inc. | $39,978.80 |
169095 | Allopathic & Osteopathic Physicians/ Surgery | Medtronic Vascular, Inc. | $37,683.00 |
80052 | Allopathic & Osteopathic Physicians/ Family Medicine | Boehringer Ingelheim Pharma GmbH & Co.KG | $24,911.25 |
202461 | Allopathic & Osteopathic Physicians/ Thoracic Surgery (Cardiothoracic Vascular Surgery) | Covidien LP | $21,594.51 |
378722 | Allopathic & Osteopathic Physicians/ Internal Medicine | GlaxoSmithKline, LLC. | $20,112.40 |
243205 | Allopathic & Osteopathic Physicians/ Internal Medicine/ Interventional Cardiology | Medtronic Vascular, Inc. | $19,273.90 |
You can see on the density plot that the next nearest outlier is pretty far away and we have clumps of outliers around 20k and 40k. Interesting things to look into.
Looking at consulting fee outliers, you can see some clumping, but most of the data is lower than 50k. That makes the 200k outlier from Teva all that much more interesting. Of course, none of this is any indication of wrong-doing, just interesting spikes in the data.
Physician | Specialty | Payer | Amount |
---|---|---|---|
104930 | Allopathic & Osteopathic Physicians/ Psychiatry & Neurology/ Neurology | Teva Pharmaceuticals USA, Inc. | $207,500.00 |
151515 | Other Service Providers/ Specialist | Alcon Research Ltd | $150,000.00 |
309376 | Allopathic & Osteopathic Physicians/ Internal Medicine | Teva Pharmaceuticals USA, Inc. | $137,559.67 |
231913 | Allopathic & Osteopathic Physicians/ Orthopaedic Surgery | Exactech, Inc. | $108,125.00 |
465481 | Allopathic & Osteopathic Physicians/ Internal Medicine/ Rheumatology | Vision Quest Industries Inc. | $102,196.09 |
409799 | Allopathic & Osteopathic Physicians/ Internal Medicine/ Endocrinology, Diabetes & Metabolism | Pfizer Inc. | $100,000.00 |
206227 | Allopathic & Osteopathic Physicians/ Orthopaedic Surgery | DePuy Synthes Sales Inc. | $93,750.00 |
436192 | Allopathic & Osteopathic Physicians/ Internal Medicine | Pfizer Inc. | $90,000.00 |
306965 | Allopathic & Osteopathic Physicians/ Psychiatry & Neurology/ Neurology | Teva Pharmaceuticals USA, Inc. | $64,125.00 |
163888 | Allopathic & Osteopathic Physicians/ Internal Medicine/ Cardiovascular Disease | Boehringer Ingelheim Pharmaceuticals, Inc. | $61,025.00 |
You can see most of the density is less than 20k, which makes that 200k outlier so interesting.
Gifts, I think, are the most interesting payment reasons in this whole dataset. I am intrigued when a physician might get a gift versus an outright fee. I imagined, going in, that gifts would be low-value items, but the table clearly shows that dentists are getting substantial gifts. What is most interesting to me is that all of the top 10 outliers are dentists.
Physician | Specialty | Payer | Amount |
---|---|---|---|
225073 | Dental Providers/ Dentist/ General Practice | Dentalez Alabama, Inc. | $56,422.00 |
167931 | Dental Providers/ Dentist | DENTSPLY IH Inc. | $8,672.50 |
380517 | Dental Providers/ Dentist | DENTSPLY IH Inc. | $8,672.50 |
380073 | Dental Providers/ Dentist/ General Practice | Benco Dental Supply Co. | $7,570.00 |
403926 | Dental Providers/ Dentist | A-dec, Inc. | $5,430.00 |
429612 | Dental Providers/ Dentist | PureLife, LLC | $5,058.72 |
404935 | Dental Providers/ Dentist | A-dec, Inc. | $5,040.00 |
8601 | Dental Providers/ Dentist/ General Practice | DentalEZ, Inc. | $3,876.35 |
385314 | Dental Providers/ Dentist/ General Practice | Henry Schein, Inc. | $3,789.99 |
389592 | Dental Providers/ Dentist/ General Practice | Henry Schein, Inc. | $3,789.99 |
Next, we look for anomalies in our payment data by using Benford’s Law. This is part of a broader series of posts about Data Science and Hadoop.
Casey Stella 24 October 2014 Cleveland, OH