Examples · Monitoring

baseline_staleness

Measure how far the frozen baseline — the imputation value and the attribution reference — has moved in population rank, per feature.

Code#

baseline_staleness.py

import numpy as np
from _setup import X_test, X_train, artifact, feature_names
 
from compileml.monitor import baseline_staleness
 
# The same constructed shift as the drift_decomposition example: extra
# accounts that were behind on repayment last month.
behind = X_test[:, feature_names.index("PAY_1")] >= 1
current_rows = np.vstack([X_test, X_test[behind]])
 
report = baseline_staleness(artifact, X_train, current_rows)
 
print("baseline percentile, development -> current")
print()
print("feature      baseline   development   current    shift")
for row in report["features"][:5]:
    print(
        f"{row['feature']:<10} {row['baseline']:>10,.0f}   {row['reference_percentile']:>11.3f}"
        f"   {row['current_percentile']:>7.3f}   {row['percentile_shift']:+.3f}"
    )

Output#

Captured from an actual run against compileml 0.9.0 and the UCI credit panel. If this script stops working, the build fails.

captured in CIbaseline_staleness.py

baseline percentile, development -> current
 
feature      baseline   development   current    shift
PAY_1               0         0.526     0.432   -0.095
PAY_2               0         0.591     0.542   -0.049
PAY_AMT1        2,121         0.500     0.546   +0.046
PAY_3               0         0.597     0.564   -0.032
LIMIT_BAL     140,000         0.496     0.528   +0.032

Notes#

Rank rather than raw deltas, because a movement of 3 means something different for every feature. Ties with the baseline count half.
Missing rates are reported alongside: a missing value is imputed at the baseline and contributes exactly zero to the decision.
No pass/fail threshold is applied; how much movement matters depends on the feature.

API reference: baseline_staleness →