I posted about a paper on a family of definitions of percentile functions, but what about weighted percentiles?
The use case is that, if we have a set of measurements with a lot of duplicates such that there are count of , and so on until count of . How can we express the percentile function in terms of pairs? Moreover, if we are given a sampling on these measurements, can we also have a percentile function defined?
Just like the percentile function, we can have a family of definitions of weighted percentile function. But if we are given a set of pairs, we might want to have the percentile and inverse percentile function defined such that
- percentile function is defined over the domain of and inverse percentile function has its range over
- minima defined from data:
- maxima defined from data:
We will discuss about the last two points later. But the first is what the percentile function should be. If we consider pairs, and try to plot them, it should be like this:
import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns sns.set() # generate data aw = pd.DataFrame(np.random.random((10,2)), columns=['a','w']).sort_values(by='a') aw['cw'] = np.cumsum(aw['w']/aw['w'].sum()) cumw = np.insert(aw['cw'].values, 0, 0) # staircase chart list_data = [pd.Series([a, a], index=[w0, w1]) for a, w0, w1 in zip(aw['a'].values, cumw, cumw[1:])] fig, ax = plt.subplots(figsize=(6,4)) for series in list_data: sns.lineplot(data=series, color='blue', ax=ax) sns.scatterplot(data=aw, x='cw', y='a', color='red') sns.scatterplot(x=cumw[:-1], y=aw['a'], color='green') sns.scatterplot(x=[(a+b)/2 for a, b in zip(cumw, cumw[1:])], y=aw['a'], color='violet') plt.show()
for the following data:
The above chart should be viewed as a weighted version of histogram. Each line
segment corresponds to the data value
a in the table above, which is
pre-sorted in ascending order. The width of the line segment corresponds to the
w. We normalized the weight so that they sum to 1 on the chart (for
percentile defined over ) and the cumulative of the normalized weight
cw column above.
I deliberately put the green, red, and violet dots on the line segments in the above chart to mark the min, max, and mid point. For any given , what can we infer from the above chart? An intuitive way is to simply treat the chart as against chart for function and read the answer directly from it. Then will have the range of exact the set of discrete values. The inverse percentile function however, is not defined for continuous range. If is given, what should be ? From the chart, it seems any where of the line segment may work. We can take the green dot, which is the minimum valid value for , or the red dot for the maximum, or the violet dot for the mean.
So here it is: We may want to define on so that it is defined over a continuous range. This is useful if the values and weights are just approximate we used to model a bigger system. In doing so, instead of drawing the staircase chart above, we join the (say) right dots:
fig, ax = plt.subplots(figsize=(6,4)) for series in list_data: sns.lineplot(data=series, color='blue', ax=ax) sns.lineplot(y=np.insert(aw["a"].values, 0, aw["a"]), x=np.insert(aw["cw"].values, 0, 0), estimator=None, sort=False, color="red", ax=ax) plt.show()
This way, we have a continuous spectrum of range for and we reinterpret the weight roughly as the slope for value . However, we still have a flat region at . Surely we can try to connect not on the right end of the “staircase” but on the left end:
fig, ax = plt.subplots(figsize=(6,4)) for series in list_data: sns.lineplot(data=series, color='blue', ax=ax) sns.lineplot(y=np.insert(aw["a"].values, len(aw), aw['a'].values[-1]), x=np.insert(aw["cw"].values, 0, 0), estimator=None, sort=False, color="green", ax=ax) plt.show()
and this continuous spectrum will have a flat region at . If connect on the mid-point, we can have the flat region at both:
fig, ax = plt.subplots(figsize=(6,4)) for series in list_data: sns.lineplot(data=series, color='blue', ax=ax) sns.lineplot( y = np.insert(np.insert(aw["a"].values, len(aw), aw['a'].values[-1]), 0, aw['a'].values), x = np.array( + [(w1+w2)/2 for w1, w2 in zip(np.insert(aw["cw"].values, 0, 0), aw["cw"].values)] + ), estimator=None, sort=False, color="violet", ax=ax) plt.show()
The flat region is surely due the condition of and we imposed. Otherwise we might consider extending the slope of the ends of the spectrum to complete the curve.
Below is the weighted percentile function modified from the percentile function in numpy, implementing the above algorithm: