Examples · Validation & governance

fit_reference

Fit the weight-of-evidence logistic floor and report the artifact against it — the other side of teacher retention.

Code#

fit_reference.py

from _setup import X_test, X_train, feature_names, y_test, y_train
 
from compileml.reference import fit_reference, reference_gini
 
reference = fit_reference(X_train, y_train, feature_names=feature_names)
 
print("kind          ", reference.kind)
print("coefficients  ", len(reference.coefficients))
print("test gini     ", round(reference_gini(reference, X_test, y_test), 4))
 
top = sorted(reference.information_value.items(), key=lambda kv: -kv[1])[:5]
print()
print("highest information value:")
for name, value in top:
    print(f"  {name:<14} {value:.4f}")
 
print()
print("Retention against a teacher cannot report that this number is higher")
print("than the compiled artifact's. That is why the floor exists.")

Output#

Captured from an actual run against compileml 0.9.0 and the UCI credit panel. If this script stops working, the build fails.

captured in CIfit_reference.py

kind           woe_logit
coefficients   19
test gini      0.5243
 
highest information value:
  PAY_1          0.8811
  PAY_2          0.5538
  PAY_3          0.4187
  PAY_4          0.3601
  PAY_5          0.3362
 
Retention against a teacher cannot report that this number is higher
than the compiled artifact's. That is why the floor exists.

Notes#

Retention against a teacher is one-sided: it can never reveal that a plain logistic regression outscores the artifact.
A champion scorecard's Gini is a better floor than a fitted one — reference= also accepts a bare float.
Without require_reference_floor the check records evidence instead of failing.

API reference: fit_reference →

Related notebook