How do you challenge the assumptions of your Machine Learning models?

some ways to check if the training data is representative of the general population data the model would score on post production is to compare population stability index between the training data and test data. Feature stability analysis comparing summary statistics amongst the train and test split also helpful.

To figure out the "assumptions" of a model, let's take a step back and define the "model". In simpler terms, an ML model is a relationship that the machine learns (from data) between the dependent and independent variables. The underlying assumption is that such a relationship exists (casual or correlation). With the rise of complex ML models, a relationship may seem to exist but is not explainable if we look at it from a business standpoint. In my experience, usually, it is good to challenge (or verify with domain knowledge) the relationship ("the model", y = f(X) ) itself before going into statistical artifacts and fine-tuning (CV, grid search, etc). The use of "feature importance" could be useful to verify the relationship assumptions.

Smaller size data or easy model interpretability go for simpler or linear models whereas if the size of your data is large, Boosting,Stacking and deep learning methods can provide higher accuracy over simpler models.

Model validation is important because it ensures that a model performs as its intended to when it is scored on unseen data. When it comes to model validation, the test data used for the validation should first of all be similarly distributed like the training data. Adding confidence intervals or prediction intervals for prediction, sensitivity analysis, considering a variety of metrics for performance evaluation on the test set(AIC,BIC,Detection Rate,F-meaure,AUC,PR, average precision-recall, area under precision-recall curve etc.),different ways of deriving feature importance using SHAP values, permutation importance, boruta algorithm should all be considered to give a wide and overwhelming confidence in the validity of the model.

Automating Continuous Integration and Continuous Delivery(CI/CD) with tools like github actions can improve speed, reliability and efficiency during model deployment. It is also important to incorporate model monitoring and data drift monitoring tools post deployment to check model decay and data drift.

Direct engagement with stakeholders is crucial for assessing the real-world impact of your machine-learning models. While traditional metrics can quantify performance, they often fall short of capturing the nuanced ethical and social implications. Surveys and regular meetings with stakeholders provide a qualitative understanding that can help fill this gap. These conversations can reveal how the model affects individuals and communities in ways that aren't immediately obvious, allowing for more comprehensive and ethical model evaluation.

In industries where compliance is of principal importance such as finance, insurance, debt collection etc. model fairness, explainability are always crucial. No feature can be included in a model if it negatively targets a particular demographic based on age, race, gender etc. Every step must be taken to increase a model fairnes and transparency.