# A tibble: 235,685 × 59
squirrels yard_type_pavement yard_type_garden yard_type_landsca
<fct> <dbl> <dbl> <dbl>
1 no squirrels 0 0 1
2 no squirrels 0 0 1
3 no squirrels 0 0 1
4 no squirrels 0 0 1
5 no squirrels 0 0 1
6 no squirrels 0 0 1
7 squirrels 0 0 0
8 squirrels 0 0 0
9 squirrels 0 0 0
10 squirrels 0 0 0
# ℹ 235,675 more rows
# ℹ 55 more variables: yard_type_woods <dbl>, yard_type_desert <dbl>,
# hab_dcid_woods <dbl>, hab_evgr_woods <dbl>,
# hab_mixed_woods <dbl>, hab_orchard <dbl>, hab_park <dbl>,
# hab_water_fresh <dbl>, hab_water_salt <dbl>,
# hab_residential <dbl>, hab_industrial <dbl>,
# hab_agricultural <dbl>, hab_desert_scrub <dbl>, …Tidymodels
Math 150 - Spring 2025
Jo Hardin
Agenda
- Workflow to help us think about model building
- Breaking up data for independent assessment
- Assessment metrics
Comparing Models
Data Modeling: The analysis in this culture starts with assuming a stochastic data model for the inside of the black box…The values of the parameters are estimated from the data and the model then used for information and/or prediction
Algorithmic Modeling: The analysis in this culture considers the inside of the box complex and unknown. The approach is to find a function f(x) — an algorithm that operates on x to predict the responses y.
Reference: Leo Breiman (2001)
Data & goal
Data: Project FeederWatch#TidyTuesday dataset on Project FeederWatch, a citizen science project for bird science, by way of TidyTuesday.
Can the characteristics of the bird feeder site like the surrounding yard and habitat predict whether a bird feeder site will be used by squirrels?
site_data <- read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-01-10/PFW_count_site_data_public_2021.csv') |>
mutate(squirrels = as.factor(ifelse(squirrels, "squirrels", "no squirrels"))) |>
filter(!is.na(squirrels))
site_data <- site_data |>
select(where(~!all(is.na(.x)))) |>
select(-loc_id, -proj_period_id, -fed_yr_round) |>
select(squirrels, everything())
site_dataHow many squirrels?
Feeders
How are other characteristics related to the presence of squirrels?
# A tibble: 2 × 2
squirrels nearby_feeders
<fct> <dbl>
1 no squirrels 0.344
2 squirrels 0.456For those observations that saw squirrels, 45% had feeders nearby; for those observations that did not see squirrels, 34% had feeders nearby.
Other variables
What about some of the variables describing the habitat?
site_data |>
group_by(squirrels) |>
summarise(across(contains("hab"), mean, na.rm = TRUE)) |>
pivot_longer(contains("hab")) |>
mutate(name = str_remove(name, "hab_")) |>
ggplot(aes(value, fct_reorder(name, value), fill = squirrels)) +
geom_col(alpha = 0.8, position = "dodge") +
scale_x_continuous(labels = scales::percent) +
labs(x = "% of locations", y = NULL, fill = NULL)Modeling
Train / test
Create an initial split:
Save training data
Save testing data
Training data
# A tibble: 176,763 × 59
squirrels yard_type_pavement yard_type_garden yard_type_landsca
<fct> <dbl> <dbl> <dbl>
1 no squirrels 0 0 1
2 no squirrels 0 0 1
3 no squirrels 0 0 1
4 no squirrels 0 0 1
5 no squirrels 0 0 1
6 no squirrels 0 0 1
7 no squirrels 0 0 0
8 no squirrels 0 0 1
9 no squirrels 0 0 1
10 no squirrels 0 0 1
# ℹ 176,753 more rows
# ℹ 55 more variables: yard_type_woods <dbl>, yard_type_desert <dbl>,
# hab_dcid_woods <dbl>, hab_evgr_woods <dbl>,
# hab_mixed_woods <dbl>, hab_orchard <dbl>, hab_park <dbl>,
# hab_water_fresh <dbl>, hab_water_salt <dbl>,
# hab_residential <dbl>, hab_industrial <dbl>,
# hab_agricultural <dbl>, hab_desert_scrub <dbl>, …Feature engineering
We prefer simple models when possible, but parsimony does not mean sacrificing accuracy (or predictive performance) in the interest of simplicity
Variables that go into the model and how they are represented are just as critical to success of the model
Feature engineering allows us to get creative with our predictors in an effort to make them more useful for our model (to increase its predictive performance)
Feature engineering is the act of creating new variables from the ones that already exist.
Feature engineering - examples
- date: change to weekend / weekday
- categorical: reduce # levels, one-hot encoding
- numerical: transform using square, ln, normalize, scale
- bin numerical data to make it categorical
- missing data: its own level? impute?
- adjust outliers
Modeling workflow
Create a recipe for feature engineering steps to be applied to the training data
Fit the model to the training data after these steps have been applied
Using the model estimates from the training data, predict outcomes for the test data
Evaluate the performance of the model on the test data
Building recipes
Initiate a recipe
Working with recipes
- When building recipes you in a pipeline, you don’t get to see the effect of the recipe on your data, which can be unsettling
- You can take a peek at what will happen when you ultimately apply the recipe to your data at the time of fitting the model
- This requires two functions:
prep()to train the recipe andbake()to apply it to your data
Note
Using prep() and bake() are shown here for demonstrative purposes. They do not need to be a part of your pipeline. I do find them assuring, however, so that I can see the effects of the recipe steps as the recipe is built.
Impute missing values & Remove zero variance predictors
Impute missing values (replace with mean).
Remove all predictors that contain only a single value
feeder_rec <- feeder_rec |> step_impute_mean(all_numeric_predictors()) |> step_nzv(all_numeric_predictors()) feeder_recfeeder_rec <- feeder_rec |> step_impute_mean(all_numeric_predictors()) |> step_nzv(all_numeric_predictors()) feeder_recfeeder_rec <- feeder_rec |> step_impute_mean(all_numeric_predictors()) |> step_nzv(all_numeric_predictors()) feeder_rec
Prep and bake
feeder_rec_trained <- prep(feeder_rec)
bake(feeder_rec_trained, feeder_train) |>
select(starts_with("hab"), everything()) # re-ordering the columns# A tibble: 176,763 × 59
hab_dcid_woods hab_evgr_woods hab_mixed_woods hab_orchard hab_park
<dbl> <dbl> <dbl> <dbl> <dbl>
1 1 0.223 1 0.0932 0.450
2 1 0.223 1 0.0932 0.450
3 1 0 1 0 0
4 1 0.223 0.655 0.0932 0.450
5 1 0 0 0 0
6 1 0.223 1 0.0932 0.450
7 0 0 1 0 0
8 0.495 0.223 0.655 0.0932 0.450
9 1 1 0.655 0.0932 1
10 0 0 1 0 0
# ℹ 176,753 more rows
# ℹ 54 more variables: hab_water_fresh <dbl>, hab_water_salt <dbl>,
# hab_residential <dbl>, hab_industrial <dbl>,
# hab_agricultural <dbl>, hab_desert_scrub <dbl>,
# hab_young_woods <dbl>, hab_swamp <dbl>, hab_marsh <dbl>,
# yard_type_pavement <dbl>, yard_type_garden <dbl>,
# yard_type_landsca <dbl>, yard_type_woods <dbl>, …Building workflows
Specify model
Build workflow
Workflows bring together models and recipes so that they can be easily applied to both the training and test data.
See next slide for workflow…
View workflow
══ Workflow ════════════════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: logistic_reg()
── Preprocessor ────────────────────────────────────────────────────────────────
2 Recipe Steps
• step_impute_mean()
• step_nzv()
── Model ───────────────────────────────────────────────────────────────────────
Logistic Regression Model Specification (classification)
Engine-Specific Arguments:
family = binomial
Computational engine: glm Fit model to training data
# A tibble: 59 × 5
term estimate std.error statistic p.value
<chr> <dbl> <dbl> <dbl> <dbl>
1 (Intercept) -1.42 0.0505 -28.2 2.26e-174
2 yard_type_pavement -0.854 0.149 -5.74 9.23e- 9
3 yard_type_garden -0.175 0.0392 -4.47 7.89e- 6
4 yard_type_landsca 0.168 0.0219 7.69 1.45e- 14
5 yard_type_woods 0.309 0.0170 18.1 1.59e- 73
6 yard_type_desert -0.297 0.0789 -3.76 1.68e- 4
7 hab_dcid_woods 0.336 0.0161 20.9 6.55e- 97
8 hab_evgr_woods -0.0797 0.0192 -4.15 3.36e- 5
9 hab_mixed_woods 0.420 0.0158 26.5 3.17e-155
10 hab_orchard -0.307 0.0252 -12.2 4.00e- 34
# ℹ 49 more rowsSo many predictors!
Model fit summary
# A tibble: 59 × 5
term estimate std.error statistic p.value
<chr> <dbl> <dbl> <dbl> <dbl>
1 (Intercept) -1.42 0.0505 -28.2 2.26e-174
2 yard_type_pavement -0.854 0.149 -5.74 9.23e- 9
3 yard_type_garden -0.175 0.0392 -4.47 7.89e- 6
4 yard_type_landsca 0.168 0.0219 7.69 1.45e- 14
5 yard_type_woods 0.309 0.0170 18.1 1.59e- 73
6 yard_type_desert -0.297 0.0789 -3.76 1.68e- 4
7 hab_dcid_woods 0.336 0.0161 20.9 6.55e- 97
8 hab_evgr_woods -0.0797 0.0192 -4.15 3.36e- 5
9 hab_mixed_woods 0.420 0.0158 26.5 3.17e-155
10 hab_orchard -0.307 0.0252 -12.2 4.00e- 34
# ℹ 49 more rowsAll together
══ Workflow ════════════════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: logistic_reg()
── Preprocessor ────────────────────────────────────────────────────────────────
2 Recipe Steps
• step_impute_mean()
• step_nzv()
── Model ───────────────────────────────────────────────────────────────────────
Logistic Regression Model Specification (classification)
Computational engine: glm # A tibble: 59 × 5
term estimate std.error statistic p.value
<chr> <dbl> <dbl> <dbl> <dbl>
1 (Intercept) -1.42 0.0505 -28.2 2.26e-174
2 yard_type_pavement -0.854 0.149 -5.74 9.23e- 9
3 yard_type_garden -0.175 0.0392 -4.47 7.89e- 6
4 yard_type_landsca 0.168 0.0219 7.69 1.45e- 14
5 yard_type_woods 0.309 0.0170 18.1 1.59e- 73
6 yard_type_desert -0.297 0.0789 -3.76 1.68e- 4
7 hab_dcid_woods 0.336 0.0161 20.9 6.55e- 97
8 hab_evgr_woods -0.0797 0.0192 -4.15 3.36e- 5
9 hab_mixed_woods 0.420 0.0158 26.5 3.17e-155
10 hab_orchard -0.307 0.0252 -12.2 4.00e- 34
# ℹ 49 more rowsfeeder_train_pred <- predict(object = feeder_fit, new_data = feeder_train, type = "prob") |>
mutate(.pred_class = as.factor(ifelse(.pred_squirrels >=0.5,
"squirrels", "no squirrels"))) |>
bind_cols(feeder_train |> select(squirrels))
feeder_train_pred# A tibble: 176,763 × 4
`.pred_no squirrels` .pred_squirrels .pred_class squirrels
<dbl> <dbl> <fct> <fct>
1 0.168 0.832 squirrels no squirrels
2 0.168 0.832 squirrels no squirrels
3 0.358 0.642 squirrels no squirrels
4 0.166 0.834 squirrels no squirrels
5 0.274 0.726 squirrels no squirrels
6 0.191 0.809 squirrels no squirrels
7 0.288 0.712 squirrels no squirrels
8 0.184 0.816 squirrels no squirrels
9 0.227 0.773 squirrels no squirrels
10 0.350 0.650 squirrels no squirrels
# ℹ 176,753 more rowsEvaluate model
sensitivty = number of squirrels that are accurately predicted to be squirrels (true positive rate)
specificity = number of non-squirrels that are accurately predicted to be non-squirrels (true negative rate)
Make predictions for training data
feeder_train_pred <- predict(object = feeder_fit, new_data = feeder_train, type = "prob") |>
mutate(.pred_class = as.factor(ifelse(.pred_squirrels >=0.5,
"squirrels", "no squirrels"))) |>
bind_cols(feeder_train |> select(squirrels))
feeder_train_pred# A tibble: 176,763 × 4
`.pred_no squirrels` .pred_squirrels .pred_class squirrels
<dbl> <dbl> <fct> <fct>
1 0.168 0.832 squirrels no squirrels
2 0.168 0.832 squirrels no squirrels
3 0.358 0.642 squirrels no squirrels
4 0.166 0.834 squirrels no squirrels
5 0.274 0.726 squirrels no squirrels
6 0.191 0.809 squirrels no squirrels
7 0.288 0.712 squirrels no squirrels
8 0.184 0.816 squirrels no squirrels
9 0.227 0.773 squirrels no squirrels
10 0.350 0.650 squirrels no squirrels
# ℹ 176,753 more rowsAccuracy
rbind(accuracy(data = feeder_train_pred, truth = squirrels,
estimate = .pred_class),
sensitivity(data = feeder_train_pred, truth = squirrels,
estimate = .pred_class, event_level = "second"),
specificity(data = feeder_train_pred, truth = squirrels,
estimate = .pred_class, event_level = "second"))# A tibble: 3 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 accuracy binary 0.815
2 sensitivity binary 0.985
3 specificity binary 0.101Visualizing accuracy
But, really…
who cares about predictions on training data?
Make predictions for testing data
feeder_test_pred <- predict(object = feeder_fit, new_data = feeder_test, type = "prob") |>
mutate(.pred_class = as.factor(ifelse(.pred_squirrels >=0.5,
"squirrels", "no squirrels"))) |>
bind_cols(feeder_test |> select(squirrels))
feeder_test_pred# A tibble: 58,922 × 4
`.pred_no squirrels` .pred_squirrels .pred_class squirrels
<dbl> <dbl> <fct> <fct>
1 0.201 0.799 squirrels no squirrels
2 0.156 0.844 squirrels no squirrels
3 0.352 0.648 squirrels no squirrels
4 0.197 0.803 squirrels squirrels
5 0.121 0.879 squirrels squirrels
6 0.272 0.728 squirrels squirrels
7 0.0459 0.954 squirrels squirrels
8 0.0341 0.966 squirrels squirrels
9 0.0631 0.937 squirrels squirrels
10 0.0312 0.969 squirrels squirrels
# ℹ 58,912 more rowsEvaluate testing data
rbind(accuracy(data = feeder_test_pred, truth = squirrels,
estimate = .pred_class),
sensitivity(data = feeder_test_pred, truth = squirrels,
estimate = .pred_class, event_level = "second"),
specificity(data = feeder_test_pred, truth = squirrels,
estimate = .pred_class, event_level = "second"))# A tibble: 3 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 accuracy binary 0.814
2 sensitivity binary 0.985
3 specificity binary 0.0988Evaluate testing data
Different predictions for test data
Note that you now have to be predicted 0.75 to be a squirrel!
feeder_test_pred_0.75 <- predict(object = feeder_fit, new_data = feeder_test, type = "prob") |>
mutate(.pred_class = as.factor(ifelse(.pred_squirrels >=0.75,
"squirrels", "no squirrels"))) |>
bind_cols(feeder_test |> select(squirrels))
feeder_test_pred_0.75# A tibble: 58,922 × 4
`.pred_no squirrels` .pred_squirrels .pred_class squirrels
<dbl> <dbl> <fct> <fct>
1 0.201 0.799 squirrels no squirrels
2 0.156 0.844 squirrels no squirrels
3 0.352 0.648 no squirrels no squirrels
4 0.197 0.803 squirrels squirrels
5 0.121 0.879 squirrels squirrels
6 0.272 0.728 no squirrels squirrels
7 0.0459 0.954 squirrels squirrels
8 0.0341 0.966 squirrels squirrels
9 0.0631 0.937 squirrels squirrels
10 0.0312 0.969 squirrels squirrels
# ℹ 58,912 more rowsDifferent prediction cutoff
rbind(accuracy(data = feeder_test_pred_0.75, truth = squirrels,
estimate = .pred_class),
sensitivity(data = feeder_test_pred_0.75, truth = squirrels,
estimate = .pred_class, event_level = "second"),
specificity(data = feeder_test_pred_0.75, truth = squirrels,
estimate = .pred_class, event_level = "second"))# A tibble: 3 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 accuracy binary 0.744
2 sensitivity binary 0.803
3 specificity binary 0.497rbind(accuracy(data = feeder_test_pred, truth = squirrels,
estimate = .pred_class),
sensitivity(data = feeder_test_pred, truth = squirrels,
estimate = .pred_class, event_level = "second"),
specificity(data = feeder_test_pred, truth = squirrels,
estimate = .pred_class, event_level = "second"))# A tibble: 3 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 accuracy binary 0.814
2 sensitivity binary 0.985
3 specificity binary 0.0988Training vs. testing
rbind(accuracy(data = feeder_train_pred, truth = squirrels,
estimate = .pred_class),
sensitivity(data = feeder_train_pred, truth = squirrels,
estimate = .pred_class, event_level = "second"),
specificity(data = feeder_train_pred, truth = squirrels,
estimate = .pred_class, event_level = "second"))# A tibble: 3 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 accuracy binary 0.815
2 sensitivity binary 0.985
3 specificity binary 0.101rbind(accuracy(data = feeder_test_pred, truth = squirrels,
estimate = .pred_class),
sensitivity(data = feeder_test_pred, truth = squirrels,
estimate = .pred_class, event_level = "second"),
specificity(data = feeder_test_pred, truth = squirrels,
estimate = .pred_class, event_level = "second"))# A tibble: 3 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 accuracy binary 0.814
2 sensitivity binary 0.985
3 specificity binary 0.0988What’s up with training data?
The training set does not have the capacity to be a good arbiter of performance.
It is not an independent piece of information; predicting the training set can only reflect what the model already knows.
Suppose you give a class a test, then give them the answers, then provide the same test. The student scores on the second test do not accurately reflect what they know about the subject; these scores would probably be higher than their results on the first test.
All together
══ Workflow ════════════════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: logistic_reg()
── Preprocessor ────────────────────────────────────────────────────────────────
2 Recipe Steps
• step_impute_mean()
• step_nzv()
── Model ───────────────────────────────────────────────────────────────────────
Logistic Regression Model Specification (classification)
Computational engine: glm # A tibble: 59 × 5
term estimate std.error statistic p.value
<chr> <dbl> <dbl> <dbl> <dbl>
1 (Intercept) -1.42 0.0505 -28.2 2.26e-174
2 yard_type_pavement -0.854 0.149 -5.74 9.23e- 9
3 yard_type_garden -0.175 0.0392 -4.47 7.89e- 6
4 yard_type_landsca 0.168 0.0219 7.69 1.45e- 14
5 yard_type_woods 0.309 0.0170 18.1 1.59e- 73
6 yard_type_desert -0.297 0.0789 -3.76 1.68e- 4
7 hab_dcid_woods 0.336 0.0161 20.9 6.55e- 97
8 hab_evgr_woods -0.0797 0.0192 -4.15 3.36e- 5
9 hab_mixed_woods 0.420 0.0158 26.5 3.17e-155
10 hab_orchard -0.307 0.0252 -12.2 4.00e- 34
# ℹ 49 more rowsfeeder_test_pred <- predict(object = feeder_fit, new_data = feeder_test, type = "prob") |>
mutate(.pred_class = as.factor(ifelse(.pred_squirrels >=0.5,
"squirrels", "no squirrels"))) |>
bind_cols(feeder_test |> select(squirrels))
feeder_test_pred# A tibble: 58,922 × 4
`.pred_no squirrels` .pred_squirrels .pred_class squirrels
<dbl> <dbl> <fct> <fct>
1 0.201 0.799 squirrels no squirrels
2 0.156 0.844 squirrels no squirrels
3 0.352 0.648 squirrels no squirrels
4 0.197 0.803 squirrels squirrels
5 0.121 0.879 squirrels squirrels
6 0.272 0.728 squirrels squirrels
7 0.0459 0.954 squirrels squirrels
8 0.0341 0.966 squirrels squirrels
9 0.0631 0.937 squirrels squirrels
10 0.0312 0.969 squirrels squirrels
# ℹ 58,912 more rowsrbind(accuracy(data = feeder_test_pred, truth = squirrels, estimate = .pred_class),
sensitivity(data = feeder_test_pred, truth = squirrels, estimate = .pred_class, event_level = "second"),
specificity(data = feeder_test_pred, truth = squirrels, estimate = .pred_class, event_level = "second"))# A tibble: 3 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 accuracy binary 0.814
2 sensitivity binary 0.985
3 specificity binary 0.0988Cross Validation
- Cross validation for model comparison (which model is best?)
- Cross validation for model evaluation (what is the accuracy rate of the model?)
Competing models
we use the test data to assess how the model does. But we haven’t yet thought about how to use the data to build a particular model.
compare two different models to predict whether or not there are squirrels
- Model 1: removes the information about the habitat and about the trees and shrubs
- Model 2: removes the information about feeding the birds
Compare recipes
How can we decide?
measure which model does better on the test datameasure which model does better on the training data- measure which model does better on the cross validated data
How does cross validation work?
4-fold CV is depicted. Notice that the holdout group is never used as part of the coefficient estimation process.
Cross validation
More specifically, v-fold cross validation:
- Randomly partition the training data into v group
- Use v-1 groups to build the model (calculate MLEs); use 1 group for prediction / assessment
- Repeat v times, updating which group is used for assessment each time
Cross validation, step 1
Consider the example below where the training data are randomly split into 3 partitions:
Splitting the data into a partition of v=3 groups. Source: (Kuhn and Silge 2022)
Cross validation, steps 2 and 3
- Use 1 partition for assessment, and the remaining v-1 partitions for analysis
- Repeat v times, updating which partition is used for assessment each time
With the data split into three groups, we can see how 2/3 of the observations are used to fit the model and 1/3 of the observations are used to estimate the performance of the model. Source: (Kuhn and Silge 2022)
Cross validation using tidymodels
Fit the model separately to each fold
══ Workflow ════════════════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: logistic_reg()
── Preprocessor ────────────────────────────────────────────────────────────────
4 Recipe Steps
• step_rm()
• step_rm()
• step_impute_mean()
• step_nzv()
── Model ───────────────────────────────────────────────────────────────────────
Logistic Regression Model Specification (classification)
Engine-Specific Arguments:
family = binomial
Computational engine: glm 
# A tibble: 4 × 6
.metric .estimator mean n std_err .config
<chr> <chr> <dbl> <int> <dbl> <chr>
1 accuracy binary 0.809 3 0.000247 Preprocessor1_Model1
2 roc_auc binary 0.663 3 0.00180 Preprocessor1_Model1
3 sensitivity binary 0.996 3 0.000190 Preprocessor1_Model1
4 specificity binary 0.0265 3 0.000545 Preprocessor1_Model1Repeat the CV analysis for the second model
feeder_rec2 <- recipe(squirrels ~ ., data = feeder_train) |>
# delete the variables on when the birds were fed
step_rm(contains("fed")) |>
# delete the variables about the bird feeders
step_rm(contains("feed")) |>
step_impute_mean(all_numeric_predictors()) |>
step_nzv(all_numeric_predictors())
feeder_rec2══ Workflow ════════════════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: logistic_reg()
── Preprocessor ────────────────────────────────────────────────────────────────
4 Recipe Steps
• step_rm()
• step_rm()
• step_impute_mean()
• step_nzv()
── Model ───────────────────────────────────────────────────────────────────────
Logistic Regression Model Specification (classification)
Engine-Specific Arguments:
family = binomial
Computational engine: glm 
# A tibble: 4 × 6
.metric .estimator mean n std_err .config
<chr> <chr> <dbl> <int> <dbl> <chr>
1 accuracy binary 0.813 3 0.000325 Preprocessor1_Model1
2 roc_auc binary 0.698 3 0.00228 Preprocessor1_Model1
3 sensitivity binary 0.990 3 0.000424 Preprocessor1_Model1
4 specificity binary 0.0693 3 0.000291 Preprocessor1_Model1Compare two models
Model 1
# A tibble: 4 × 6
.metric .estimator mean n std_err .config
<chr> <chr> <dbl> <int> <dbl> <chr>
1 accuracy binary 0.809 3 0.000247 Preprocessor1_Model1
2 roc_auc binary 0.663 3 0.00180 Preprocessor1_Model1
3 sensitivity binary 0.996 3 0.000190 Preprocessor1_Model1
4 specificity binary 0.0265 3 0.000545 Preprocessor1_Model1Model 2
# A tibble: 4 × 6
.metric .estimator mean n std_err .config
<chr> <chr> <dbl> <int> <dbl> <chr>
1 accuracy binary 0.813 3 0.000325 Preprocessor1_Model1
2 roc_auc binary 0.698 3 0.00228 Preprocessor1_Model1
3 sensitivity binary 0.990 3 0.000424 Preprocessor1_Model1
4 specificity binary 0.0693 3 0.000291 Preprocessor1_Model1Cross validation then test data
Nested cross-validation: two cross-validation loops are run one inside the other. (Varoquaux et al. 2017)
Bias-variance trade-off
Test and training error as a function of model complexity. Note that the error goes down monotonically only for the training data. Be careful not to overfit!! (Hastie, Tibshirani, and Friedman 2001)
Hastie, T., R. Tibshirani, and J. Friedman. 2001. The Elements of Statistical Learning. Springer.
Kuhn, Max, and Julia Silge. 2022. Tidy Modeling with r. https://www.tmwr.org/.
Varoquaux, G., P. Reddy Raamana, D. Engemann, A. Hoyos-Idrobo, Y. Schwartz, and B. Thirion. 2017. “Assessing and Tuning Brain Decoders: Cross-Validation, Caveats, and Guidelines.” NeuroImage 145: 166–79.
Tidymodels Math 150 - Spring 2025 Jo Hardin