validate_artifact

def validate_artifact(artifact_or_path, X_val=None, y_val=None, *, model=None, latent_train=None, n_reconciliation: int = 30, n_stability: int = 50, monotonicity_tolerance: float = 0.01, churn_threshold: float = 0.15, jaccard_threshold: float = 0.5, noise_scale: float = 0.01, min_fidelity_spearman: float = 0.999, max_within_band_auc: float | None = None, require_full_reason_coverage: bool = False, reference=None, require_reference_floor: bool = False, require_selection_hygiene: bool = False, seed: int = 42) -> dict

from compileml.validate import validate_artifact

Run the ten-check framework. Checks lacking inputs skip, not fail.

Args: artifact_or_path: Artifact dict or path to its JSON (paths get the full load-and-verify treatment). X_val: Validation rows for the empirical checks (2–7). y_val: Binary outcomes aligned with X_val (checks 5). model: The source float model; enables the fidelity check (3). latent_train: Training latents; enables the churn baseline (6). require_full_reason_coverage: Make check 8 fail below 100% coverage (recommended for consumer-facing deployments). reference: A fitted compileml.reference.ReferenceModel or a bare Gini float, enabling check 10. Retention against a teacher is one-sided and cannot reveal that a plain logistic regression outscores the artifact; this is the other side of it. A champion scorecard's Gini is a better floor than a fitted one — pass the number. require_reference_floor: Make check 10 *fail* when the artifact does not clear the reference, rather than recording it as evidence (the max_within_band_auc precedent). require_selection_hygiene: Make check 11 *fail* on a provenance block that records a broken protocol — soft targets that were not cross-fitted, partitions that overlap, Report read more than once — rather than listing the issues as evidence. Check 11 skips when the artifact carries no provenance block; it can verify what the block records, not that the block is honest.

Returns: {"all_pass": bool, "checks": {name: {"pass", "skipped", ...evidence}}}

Parameters#

Name	Type	Default	Kind
artifact_or_path	—	required	positional
X_val	—	None	positional
y_val	—	None	positional
model	—	None	keyword-only
latent_train	—	None	keyword-only
n_reconciliation	int	30	keyword-only
n_stability	int	50	keyword-only
monotonicity_tolerance	float	0.01	keyword-only
churn_threshold	float	0.15	keyword-only
jaccard_threshold	float	0.5	keyword-only
noise_scale	float	0.01	keyword-only
min_fidelity_spearman	float	0.999	keyword-only
max_within_band_auc	float \| None	None	keyword-only
require_full_reason_coverage	bool	False	keyword-only
reference	—	None	keyword-only
require_reference_floor	bool	False	keyword-only
require_selection_hygiene	bool	False	keyword-only
seed	int	42	keyword-only

Returns#

dict

Example#

python

from compileml.validate import validate_artifact
 
report = validate_artifact(artifact, X_holdout, y_holdout, reference=ref)
assert report["all_pass"]

Worked example: validate_artifact →

View source