Examples · Monitoring

band_drift

Compare each band's observed bad rate with the PD the artifact emitted on the same decisions, grouped by the band each decision logged.

Code#

band_drift.py

from _setup import X_test, artifact, y_test
 
from compileml.monitor import band_drift
from compileml.runtime import decide
 
# The holdout's outcomes are known, which is what band_drift needs: matured
# outcomes on decisions the artifact made. Calibration was fitted on the
# training split, so this is an out-of-sample check.
decisions = [decide(artifact, r.tolist(), explain=False) for r in X_test]
report = band_drift(artifact, decisions, y_test)
 
print("band      n   observed   95% interval       emitted PD  outside")
for band in report["bands"]:
    lo, hi = band["observed_ci"]
    print(
        f"{band['band']}  {band['n']:>5}   {band['observed_bad_rate']:7.2%}   "
        f"[{lo:6.2%}, {hi:6.2%}]   {band['mean_pd']:7.2%}     {band['outside_interval']}"
    )
 
flagged = sum(1 for band in report["bands"] if band["outside_interval"])
print()
print(f"{flagged} of {len(report['bands'])} bands outside their interval")

Output#

Captured from an actual run against compileml 0.9.0 and the UCI credit panel. If this script stops working, the build fails.

captured in CIband_drift.py

band      n   observed   95% interval       emitted PD  outside
G01    722     5.12%   [ 3.74%,  6.98%]     3.69%     True
G02    737     6.78%   [ 5.18%,  8.83%]     7.52%     False
G03    735     9.66%   [ 7.73%, 12.01%]     9.63%     False
G04    769    13.00%   [10.81%, 15.57%]    12.17%     False
G05    773    14.88%   [12.54%, 17.56%]    13.08%     False
G06    767    15.12%   [12.76%, 17.83%]    14.23%     False
G07    763    20.18%   [17.49%, 23.18%]    20.81%     False
G08    723    26.28%   [23.20%, 29.61%]    26.18%     False
G09    739    39.24%   [35.79%, 42.81%]    41.79%     False
G10    772    69.43%   [66.09%, 72.58%]    71.30%     False
 
1 of 10 bands outside their interval

Notes#

Outcomes must have matured. Decisions whose performance window is still open look better calibrated than they are.
At 95% intervals a perfectly calibrated artifact still flags about one band in twenty, so one flag in a period is expected; a band that flags period after period is the signal.
Nothing is recalibrated. A persistent flag is evidence for an explicit recalibrate_artifact call, which refits the PD table without moving a band edge.

API reference: band_drift →