Homework 3

Due Wednesday, March 29th at 11:59pm CST

Deliverables: Please use this template to knit an HTML or PDF document, and submit to Moodle.




Project Work

Note: this is a repeat of HW2. Many of you struggled with this section, so this gives you an opportunity to incorporate feedback from the preceptors, or try additional models.

Goal: Finish an analysis of your dataset to answer your regression research question.


Required Analyses:

  1. Initial investigation: ignoring nonlinearity (for now)
    • Use ordinary least squares (OLS) regression and LASSO to build initial models for your quantitative outcome as a function of the predictors of interest. (As part of data cleaning, exclude any variables that you don’t want to consider as predictors.)
      • These models should not include any transformations to deal with nonlinearity. You’ll explore this in the next investigation.
    • Estimate test performance of the models from these different methods. Report and interpret (with units) these estimates along with a measure of uncertainty in the estimate.
    • Compare estimated test performance across methods. Which method(s) might you prefer?
    • Note that if some (but not all) of the indicator terms for a categorical predictor are selected in the final models, the whole predictor should be treated as selected.


  1. Accounting for nonlinearity
    • Update your stepwise selection model(s) and LASSO model to use natural splines for the quantitative predictors.
      • You’ll need to update the model formula from y ~ . to something like y ~ cat_var1 + ns(quant_var1, df) + ....
      • It’s recommended to use few knots (e.g., 2 knots = 3 degrees of freedom).
    • Compare insights from variable importance analyses here and the corresponding results from Investigation 1. Now after having accounted for nonlinearity, have the most relevant predictors changed?
      • Note that if some (but not all) of the spline terms are selected in the final models, the whole predictor should be treated as selected.
    • Fit a GAM using LOESS terms using the set of variables deemed to be most relevant based on your investigations so far.
      • How does test performance of the GAM compare to other models you explored?
      • Do you gain any insights from the GAM output plots for each predictor?


  1. Summarize investigations
    • Decide on an overall best model based on your investigations so far. To do this, make clear your analysis goals. Predictive accuracy? Interpretability? A combination of both?


  1. Societal impact
    • Are there any harms that may come from your analyses and/or how the data were collected?
    • What cautions do you want to keep in mind when communicating your work?




Portfolio Work

Revisions:

  • Make any revisions desired to previous concepts.
    • Important formatting note: Please use a comment to mark the text that you want to be reread. (Highlight each span of text you want to be reread, and mark it with the comment “REVISION”.)
  • Rubrics to past homeworks will be available on Moodle (under the Solutions section). Look at these rubrics to guide your revisions. You can always ask for guidance in office hours as well.


New concepts to address:

Evaluating classification models: Consider this xkcd comic. Write a paragraph (around 250 words) that addresses the following questions. Craft this pargraph so it flows nicely and does not read like a disconnected list of answers. (Include transitions between sentences.)

  • What is the classification model here?
  • How do the ideas in this comic emphasize comparisons between overall accuracy and class-specific accuracy measures?
  • What are the names of the relevant class-specific accuracy measures here, and what are there values?
  • How does this comic connect to the no-information rate?

Logistic regression:

  • Algorithmic understanding: Write your own example of a logistic regression model formula. (Don’t use the example from the video.) Using this example, show how to use the model to make both a soft and a hard prediction.

  • Bias-variance tradeoff: What tuning parameters control the performance of the method? How do low/high values of the tuning parameters relate to bias and variance of the learned model? (3 sentences max.)

  • Parametric / nonparametric: Where (roughly) does this method fall on the parametric-nonparametric spectrum, and why? (3 sentences max.)

  • Scaling of variables: Does the scale on which variables are measured matter for the performance of this algorithm? Why or why not? If scale does matter, how should this be addressed when using this method? (3 sentences max.)

  • Computational time: SKIP

  • Interpretation of output: In general, how can the coefficient for a quantitative predictor be interpreted? How can the coefficient for a categorical predictor (an indicator variable) be interpreted?




Ethics in ML

Read the article Getting Past Identity to What You Really Want. Write a short (roughly 250 words), thoughtful response about the ideas that the article brings forth. What skills do you think are essential for the leaders and data analysts of organizations to have to handle these issues with care?