Examples · Selecting

compile_selected

Choose the target, the tree count and the depth on data the reported figure never sees, then read retention and the floor ratio once, with intervals.

Code#

compile_selected.py

import hashlib
 
import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier
 
from compileml import compile_selected
from compileml.datasets import load_credit_default
 
X, y, feature_names = load_credit_default()
 
# This panel repeats rows: 2.7% are byte-identical to an earlier one. Grouping
# by content keeps every copy in one partition. Without group=, the protocol
# refuses to run rather than let a Report row also sit in Fit.
group = [
    hashlib.blake2b(row.tobytes(), digest_size=8).hexdigest()
    for row in np.ascontiguousarray(X)
]
 
result = compile_selected(
    X,
    y,
    # The ceiling: the strongest model trained under a declared budget. It
    # prices the compilation and never ships.
    ceiling=lambda X_fit, y_fit: HistGradientBoostingClassifier(random_state=0).fit(X_fit, y_fit),
    ceiling_budget={"search": "none; scikit-learn defaults"},
    reference="woe",
    feature_names=feature_names,
    group=group,
    seed=42,
)
 
parts, chosen, gate = result.partitions, result.selected, result.floor_gate
print(f"partitions : Fit {len(parts.fit):,} · Select {len(parts.select):,} · Report {len(parts.report):,}")
print(
    f"selected   : alpha={chosen['alpha']:g}, {chosen['n_estimators']} trees, "
    f"depth {chosen['max_depth']}"
)
print(
    f"             chosen on Select from {len(result.selection_curve)} configurations; "
    f"{chosen['n_tied']} tied within one standard error"
)
print(f"floor gate : passed={gate['passed']} (Select Gini {gate['select_gini']:.3f} vs floor {gate['floor_gini']:.3f})")
 
# Report is read once, after the winner is refit on Fit and Select together.
report = result.report()
ceiling, artifact, floor = report["ceiling"], report["artifact"], report["floor"]
 
print()
print(f"On Report ({report['rows']:,} rows, never used to choose):")
print(f"  ceiling Gini  {ceiling['gini']:.3f}")
print(f"  artifact Gini {artifact['gini']:.3f}   KS {artifact['ks']:.3f}   Brier {artifact['brier']:.3f}")
print(f"  floor Gini    {floor['gini']:.3f}   (WoE logistic regression)")
print(
    f"  retention     {report['retention_pct']:.1f}% "
    f"(95% CI {report['retention_ci'][0]:.1f}–{report['retention_ci'][1]:.1f})"
)
print(
    f"  vs the floor  {report['floor_ratio_pct']:.1f}% "
    f"(95% CI {report['floor_ratio_ci'][0]:.1f}–{report['floor_ratio_ci'][1]:.1f})"
)
print()
print(f"Shipped artifact: {len(result.artifact['model']['trees'])} trees, depth {chosen['max_depth']}.")
print("The selection curve ranks configurations; only the Report figures describe this artifact.")

Output#

Captured from an actual run against compileml 0.9.0 and the UCI credit panel. If this script stops working, the build fails.

captured in CIcompile_selected.py

partitions : Fit 18,087 · Select 5,927 · Report 5,986
selected   : alpha=1, 20 trees, depth 2
             chosen on Select from 40 configurations; 27 tied within one standard error
floor gate : passed=True (Select Gini 0.560 vs floor 0.542)
 
On Report (5,986 rows, never used to choose):
  ceiling Gini  0.536
  artifact Gini 0.520   KS 0.399   Brier 0.138
  floor Gini    0.514   (WoE logistic regression)
  retention     96.9% (95% CI 94.7–99.1)
  vs the floor  101.1% (95% CI 99.2–103.2)
 
Shipped artifact: 20 trees, depth 2.
The selection curve ranks configurations; only the Report figures describe this artifact.

Notes#

Three roles: the ceiling is the strongest model you can train under a declared budget and never ships; the floor is a WoE logistic regression or your champion scorecard's Gini; the candidate is the depth-2 whitebox.
Three partitions: Fit trains, Select chooses, Report is read once. Choosing and reporting on the same holdout flatters the figure, which is the leak this protocol removes.
Every configuration is quantized, calibrated and banded before it is scored, so banding and quantization are priced rather than assumed.
Among configurations tied within one standard error, the simplest wins — labels over soft targets, then fewer trees, then lower depth. On this panel 27 of 40 were tied, and the tie rule picked 20 trees trained on labels alone.
Retention and the floor ratio are reported together: the first prices the guarantees, the second says whether adopting them is worth it. Here the floor interval reaches below 100%.

API reference: compile_selected →