## Context and Data

Before proceeding, install the `e1071` package (utilities for evaluating classification models) by entering `install.packages("e1071")` in the Console. 

We'll be working with a [spam dataset](https://archive.ics.uci.edu/ml/datasets/Spambase) that contains information on different features of emails and whether or not the email was spam. The variables are as follows:

- `spam`: Either `spam` or `not spam`
- `word_freq_WORD`: percentage of words in the e-mail that match `WORD` (0-100)
- `char_freq_CHAR`: percentage of characters in the e-mail that match `CHAR` (e.g., exclamation points, dollar signs)
- `capital_run_length_average`: average length of uninterrupted sequences of capital letters
- `capital_run_length_longest`: length of longest uninterrupted sequence of capital letters
- `capital_run_length_total`: sum of length of uninterrupted sequences of capital letters

Our goal will be to use email features to predict whether or not an email is spam - essentially, to build a spam filter!

```{r}
library(dplyr)
library(readr)
library(ggplot2)
library(e1071) # install.packages("e1071")
library(tidymodels)
library(probably) #install.packages('probably')
tidymodels_prefer()

spam <- read_csv("https://www.dropbox.com/s/leurr6a30f4l32a/spambase.csv?dl=1")
```



## Exercise 1: Visualization warmup

Let's take a look at the frequency of the word "George" (the email recipient's name is George) (`word_freq_george`) and the frequency of exclamation points (`char_freq_exclam`). 

Create appropriate visualizations to assess the predictive ability of these two predictors. 

```{r}
# If you want to adjust the axis limits, you can add the following to your plot:
# + coord_cartesian(ylim = c(0,1))
# + coord_cartesian(xlim = c(0,1))

ggplot(spam, aes(x = ___, y = spam)) +
    geom_point()

ggplot(spam, aes(x = ___, y = spam)) +
    geom_point()
```



## Exercise 2: Implementing logistic regression in `tidymodels`

Our goal is to fit a logistic regression model with `word_freq_george` and `char_freq_exclam` as predictors.

a. Write down the corresponding logistic regression model formula using general notation.

b. Use `tidymodels` to fit this logistic regression model.

```{r}
# Make sure you set reference level (to the outcome you are NOT interested in)
spam <- spam %>%
  mutate(spam = relevel(factor(spam), ref='not spam')) #set reference level

# Logistic Regression Model Spec
logistic_spec <- logistic_reg() %>%
    set_engine('glm') %>%
    set_mode('classification')

# Recipe
logistic_rec <- recipe(spam ~ _____, data = spam)

# Workflow (Recipe + Model)
log_wf <- workflow() %>% 
    add_recipe(logistic_rec) %>%
    add_model(logistic_spec) 

# Fit Model
log_fit <- fit(log_wf, data = ___)
```


## Exercise 3: Interpreting the model

a. Take a look at the log-scale coefficients with `tidy(log_fit)`. Do the signs of the coefficients for the 2 predictors agree with your visual inspection from Exercise 1?

b. Display the exponentiated coefficients, and provide contextual interpretations for them (not the intercept).


## Exercise 4: Making predictions

Consider a new email where the frequency of "George" is 0.25% and the frequency of exclamation points is 1%.

a. Use the model summary to make both a soft (probability) and hard (class) prediction for this test case **by hand**. Use a default probability threshold of 0.5. (You can use math expressions to use R as a calculator. The `exp()` function exponentiates a number.)

b. Check your work from part a by using `predict()`.

```{r}
predict(___, newdata = data.frame(word_freq_george = 0.25, char_freq_exclam = 1), type = "prob")
predict(___, newdata = data.frame(word_freq_george = 0.25, char_freq_exclam = 1), type = "class")

```

## Exercise 5: Evaluate the model

a. Visualize the soft predictions and comment on a potential threshold for doing hard predictions.

```{r}
# Soft predictions
logistic_output <-  spam %>%
  bind_cols(predict(log_fit, new_data = spam, type = 'prob'))  %>%
  select(spam, .pred_spam)

head(logistic_output)

logistic_output %>%
  ggplot(aes(x = spam, y = .pred_spam)) +
  geom_boxplot()
```

b. Choose a threshold and calculate training metrics based on hard predictions (sensitivity, specificity, accuracy). Write a sentence interpreting these values in context.

```{r}
# Hard predictions (you pick threshold)
logistic_output <- logistic_output %>%
  mutate(.pred_class = make_two_class_pred(`.pred_not spam`, levels(spam), threshold = 1 - ___)) 

# Confusion Matrix
logistic_output %>%
  conf_mat(truth = spam, estimate = .pred_class)

# Calculate them by hand first and then confirm below
#sens: sensitivity = chance of correctly predicting second level, given second level (Spam)
#spec: specificity = chance of correctly predicting first level, given first level (Not Spam)
#accuracy: accuracy = chance of correctly predicting outcome


sensi =  
speci =  /
acc = 




log_metrics <- metric_set(sens, yardstick::spec, accuracy)
logistic_output %>% 
  log_metrics(estimate = .pred_class, truth = spam, event_level = "second")
```

c. Now visualize the soft predictions using all possible thresholds by visualizing the ROC curve and calculating the area under the roc curve. Discuss what this tells you about how well the model classifies spam emails.

```{r}
logistic_roc <- logistic_output %>% 
    roc_curve(spam, .pred_spam, event_level = "second") 

autoplot(logistic_roc) + theme_classic()

logistic_output %>% 
  roc_auc(spam, .pred_spam, event_level = "second")
```

d. Now, let’s use CV to evaluate the model on test data. Comment on the difference in CV metrics to the training metrics.

```{r}
set.seed(123)
data_cv10 <- vfold_cv(spam, v = 10)

# CV Fit Model
log_cv_fit <- fit_resamples(
    log_wf, 
    resamples = data_cv10,
    metrics = metric_set(sens, yardstick::spec, accuracy, roc_auc),
    control = control_resamples(save_pred = TRUE, event_level = 'second'))  # you need predictions for ROC calculations

collect_metrics(log_cv_fit) #default threshold is 0.5
```





