Tidymodels
Math 150 - Spring 2023
Comparing Models
Data Modeling: The analysis in this culture starts with assuming a stochastic data model for the inside of the black box…The values of the parameters are estimated from the data and the model then used for information and/or prediction
Algorithmic Modeling: The analysis in this culture considers the inside of the box complex and unknown. The approach is to find a function f(x) — an algorithm that operates on x to predict the responses y.
Reference: Leo Breiman (2001)
Other variables
What about some of the variables describing the habitat?
Visualizing accuracy
Evaluate testing data
How does cross validation work?
4-fold CV is depicted. Notice that the holdout group is never used as part of the coefficient estimation process.
Cross validation, step 1
Consider the example below where the training data are randomly split into 3 partitions:
Splitting the data into a partition of v=3 groups. Source: (Kuhn and Silge 2022)
Cross validation, steps 2 and 3
- Use 1 partition for assessment, and the remaining v-1 partitions for analysis
- Repeat v times, updating which partition is used for assessment each time
With the data split into three groups, we can see how 2/3 of the observations are used to fit the model and 1/3 of the observations are used to estimate the performance of the model. Source: (Kuhn and Silge 2022)
Fit the model separately to each fold
feeder_rec1 <- recipe(squirrels ~ ., data = feeder_train) %>%
# delete the habitat variables
step_rm(contains("hab")) %>%
# delete the tree/shrub info
step_rm(contains("atleast")) %>%
step_impute_mean(all_numeric_predictors()) %>%
step_nzv(all_numeric_predictors())
feeder_rec1Recipe
Inputs:
role #variables
outcome 1
predictor 58
Operations:
Variables removed contains("hab")
Variables removed contains("atleast")
Mean imputation for all_numeric_predictors()
Sparse, unbalanced variable filter on all_numeric_predictors()══ Workflow ════════════════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: logistic_reg()
── Preprocessor ────────────────────────────────────────────────────────────────
4 Recipe Steps
• step_rm()
• step_rm()
• step_impute_mean()
• step_nzv()
── Model ───────────────────────────────────────────────────────────────────────
Logistic Regression Model Specification (classification)
Computational engine: glm 
# A tibble: 4 × 6
.metric .estimator mean n std_err .config
<chr> <chr> <dbl> <int> <dbl> <chr>
1 accuracy binary 0.809 3 0.000247 Preprocessor1_Model1
2 roc_auc binary 0.663 3 0.00180 Preprocessor1_Model1
3 sensitivity binary 0.996 3 0.000190 Preprocessor1_Model1
4 specificity binary 0.0265 3 0.000545 Preprocessor1_Model1Repeat the CV analysis for the second model
feeder_rec2 <- recipe(squirrels ~ ., data = feeder_train) %>%
# delete the variables on when the birds were fed
step_rm(contains("fed")) %>%
# delete the variables about the bird feeders
step_rm(contains("feed")) %>%
step_impute_mean(all_numeric_predictors()) %>%
step_nzv(all_numeric_predictors())
feeder_rec2Recipe
Inputs:
role #variables
outcome 1
predictor 58
Operations:
Variables removed contains("fed")
Variables removed contains("feed")
Mean imputation for all_numeric_predictors()
Sparse, unbalanced variable filter on all_numeric_predictors()══ Workflow ════════════════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: logistic_reg()
── Preprocessor ────────────────────────────────────────────────────────────────
4 Recipe Steps
• step_rm()
• step_rm()
• step_impute_mean()
• step_nzv()
── Model ───────────────────────────────────────────────────────────────────────
Logistic Regression Model Specification (classification)
Computational engine: glm 
# A tibble: 4 × 6
.metric .estimator mean n std_err .config
<chr> <chr> <dbl> <int> <dbl> <chr>
1 accuracy binary 0.813 3 0.000325 Preprocessor1_Model1
2 roc_auc binary 0.698 3 0.00228 Preprocessor1_Model1
3 sensitivity binary 0.990 3 0.000424 Preprocessor1_Model1
4 specificity binary 0.0693 3 0.000291 Preprocessor1_Model1Cross validation then test data
Nested cross-validation: two cross-validation loops are run one inside the other. (Varoquaux et al. 2017)
Bias-variance trade-off
Test and training error as a function of model complexity. Note that the error goes down monotonically only for the training data. Be careful not to overfit!! (Hastie, Tibshirani, and Friedman 2001)
