Machine learning for public health—a primer

Chad Heilig, PhD • [redacted: email address]
Scott Lee, PhD • [redacted: email address]
CSELS Machine Learning • [redacted: email address]
CSELS Data Hub Webinar, 2018-11-19

• 2015 Jeff Dean (Langmuir), Ventures/iFund
• 2018 Statistics Day
• CSELS/DHIS Data Hub

[redacted: intranet URL]
machine-learning-advances-surveillance/
“Statistics and machine learning are both important for the practice of data science,” said Chad Heilig, senior data scientist in CSELS. … “It’s worth exploring machine learning using open-source tools to get more value, more efficiently, from CDC’s surveillance data.”

• Build intuition for machine learning through public health examples
• Public health context: What’s the question to which ML is applied?
• Machine learning context: How does ML answer the question? Why ML?
• Data-analytic concepts: Explain and demystify through these examples
• Scope is limited by time and purpose
• Focus on data-analytic applications
• Won’t go deep; won’t go mathy (no formulae)
• Mention other-than-analytic examples (“artificial intelligence”) in passing
• Many, many other applications and methods, even in public health

umm, no
Machine learning can be just as disciplined (or not) as other data-analytic schemas.

https://xkcd.com/1838/

• “How can computers learn to solve problems without being explicitly programmed?"
• “Computer: Look at these examples and figure it out”
• How does that relate to the practice of statistics?
• Breiman’s “Statistical Modeling: The Two Cultures” (2001)
• Given input and response variables, the data-analytic goals are
• Predict responses to future (or not-yet-seen) input variables
• Infer how nature associates the response variables to input variables
• Respectively, “algorithmic modeling” and “data modeling”

• Breiman: “the focus in the statistical community on data [generative] models has
• Led to irrelevant theory and questionable scientific conclusions;
• Kept statisticians from using more suitable algorithmic [predictive] models;
• Prevented statisticians from working on exciting new problems.”
• Generative modeling culture: true model generates the data, ‘best’ way to analyze the data. Use true/best models to make inferences about data-generating mechanism, usually through notions of probability.
• Predictive modeling culture: consider any model that predicts [performs] well, without particular regard for underlying mechanism generating the data.

• 3 most important lessons (as of 2001)
• Multiplicity of good models
• Conflict between simplicity/interpretability, and accuracy (Occam’s razor)
• Dimensionality—curse or blessing
• To solve a wider range of data problems, a larger set of tools is needed.
• “Looking for a data model restricts the ability of statisticians to deal with a wide range of statistical problems.”
• Best solution might be a data model, might be an algorithmic model.
• “The trick to being a scientist is to be open to using a wide variety of tools.”

• For data analysis, machine learning is a set of methods that
• Focus on finding patterns and making predictions
• Emphasize performance > explanation > interpretation (in general practice)
• Tend to emphasize algorithmic characteristics over model structure*
• Establish rigor through generalizing and reproducing (and math)
• Machine learning and statistics overlap and extend each other
• Differ mostly in emphasis without much daylight between them
• (consider “statistical learning”)

• Task: the (kind of) problem that you want to solve: segment groups, classify new example, predict timing or size of an event, generate synthetic data
• Model: the method for using data to achieve a task: spline regression, random forest, support vector machine, neural network; evaluated by performance
• Feature: representation (possibly derived) of data that models operate on: continuous variable, indicator, count of word frequencies
• Train/learn, validate, test: fit a model, refine some parameters, check on unseen data
• Unsupervised, supervised learning: find structure vs relate an output to inputs
• (examples …)

• Group counties by population similarity (cluster analysis) to compare like with like
• Combine disparate flu models to improve prediction (ensemble learning)
• Extract and organize elements of a cancer pathology report (entity recognition)
• Identify sidewalks from Google Street View (image recognition)
• Detect surges in social media patterns based on “meaning” (semantic culling)
• Summarize topics across hundreds of documents (topic modeling)
• Predict (e.g., from text, recordings) who meets a case definition (classification)
• Autocode death records and injury claims (classification)

• Structure of this talk evolved over the 2 months of planning
• Was going to be a set of concepts, defined, with examples where possible
• Reviewed ~100 manuscripts that mention machine learning and
• either mention public health
• or include a CDC/ATSDR-affiliated author
• Whirlwind tour of 7 kinds of applications (15 manuscripts),
• how to interpret them (in < 60 minutes), punctuated by commentary
• (Annotated bibliography will be made available on SharePoint)

(because they are)
also known as curve-fitting

https://xkcd.com/2048/

https://xkcd.com/2048/

https://xkcd.com/2048/

• In-sample performance ≠ out-of-sample performance (Good fit ≠ good prediction)
• Some tradeoffs (all of which can be mathematically formalized):
• Goodness-of-fit (or prediction) and smoothness: smooth might predict better
• Bias and variance: biased but precise might predict better
• Complexity and generalizability: less complex might generalize better
• Classes of models vary complexity, and they vary in their ability to generalize
• How to assure good out-of-sample performance?
• Typically split data, train, validate, test, as with n-fold cross-validation
• Bonus: How are these characteristics affected by sample size?

• Henry KE et al. A targeted real-time early warning score for septic shock. 2015 https://doi.org/10.1126/scitranslmed.aab3719
• ICU EHRs, 54 features → proportional hazards model with lasso regularization
• Chekroud AM et al. Cross-trial prediction of treatment outcome in depression: A machine learning approach. 2016 https://doi.org/10.1016/S2215-0366(15)00471-X
• Elastic net regularization to select features, followed by gradient boosting;
• trained on data from one study, tested on data from another
• Goldstick JE et al. A clinical screening tool for predicting future firearm violence risk. 2017. https://doi.org/10.7326/M16-1927 [CDC/ATSDR][Open Access]
• Elastic-net penalized logistic regression: 188 features to 20 features to 4 features

• Feature selection is variable selection and then some
• Examples of variable selection in regression models: forward selection, backward elimination, stepwise regression, and best subset regression; all are bad
• Also refers to constructing and using derived variables
  • Represent categorical variables as a set of 0-1 indicators
  • Convert all the words in a document to word counts
• What are the risks and remedies for having too many predictors?
  • Overfitting, model instability, variance inflation
  • Algorithms and heuristics for eliminating features
• (e.g., recursive feature elimination through information gain)

• Regularization
• A way of dealing with too many predictors, to keep them from getting too big (think salary cap)
• Ridge, ℓ2: sum of squared coefs ≤ K2. Shrink K2 → many small coefficients
• Lasso, ℓ1: sum of absolute coefs ≤ K1. Shrink K1 → many coefficients become 0, inducing feature selection!
• Elastic net: combine ridge and lasso, with 2 hyperparameters K1 and K2
• Regularization hyperparameters need to be tuned, typically on a validation set

• Henry KE et al. A targeted real-time early warning score for septic shock. 2015 https://doi.org/10.1126/scitranslmed.aab3719
• ICU EHRs, 54 features → proportional hazards model with lasso regularization
• Chekroud AM et al. Cross-trial prediction of treatment outcome in depression: A machine learning approach. 2016 https://doi.org/10.1016/S2215-0366(15)00471-X
• Elastic net regularization to select features, followed by gradient boosting;
• trained on data from one study, tested on data from another
• Goldstick JE et al. A clinical screening tool for predicting future firearm violence risk. 2017. https://doi.org/10.7326/M16-1927 [CDC/ATSDR][Open Access]
• Elastic-net penalized logistic regression: 188 features to 20 features to 4 features

• Regularization methods have been common in statistics, with and without ML
• Regression-based methods also extend to rich (i.e., curvy) classes of functions
• Cubic splines, B-splines
• Smoothing splines, penalized splines (both involve regularization)
• Loess (locally weighted regression), kernel smoothing
• Multivariate adaptive regression splines (MARS)
• Think of “statistical learning” as (in part) principled ways of figuring out how rich the class of fitting functions should be: how curvy, where curves are allowed

how to capitalize on
a separating hyperplane

graphics from https://en.wikibooks.org/wiki/Support_Vector_Machines

• Ladd-Acosta C et al. Presence of an epigenetic signature of prenatal cigarette smoke exposure in childhood. 2016. https://doi.org/10.1016/j.envres.2015.11.014 [CDC/ATSDR]
• SVM with radial basis function kernel; 81% classification accuracy for childhood DNA methylation at 26 CpG loci as biomarker of prenatal smoking exposure
• Yanamala N et al. Sparse supervised classification methods predict and characterize nanomaterial exposures. 2018. https://doi.org/10.1177/0192623317730575 [CDC/ATSDR]
• Proteins in bronchoalveolar lavage to predict exposure to multiwalled carbon nanotubes; sparse SVM (ℓ1- and ℓ2-norm, recursive feature elimination): macrophage-derived chemokine associated with exposures

Green (H3) fails; red (H2) beats blue (H1)

Hard margin and 3 support vectors

graphics from https://en.wikibooks.org/wiki/Support_Vector_Machines

• The simplest support vector classifier separates 2 sets of points by cutting between them, with maximum separation (maximal margin classifier)
• Separating hyperplane is the (hard) margin; points at the margin are support vectors
• If points can only be approximately linearly separated, a soft margin is found
• Penalty hyperparameter for misclassified points on the wrong side of margin, hence some tuning is needed
• Kernel trick extends linear separation through a rich set of separating curves

Support vector machines with nonlinear kernels

Left: An SVM with a polynomial kernel of degree 3.
Right: An SVM with a radial kernel..
James G et al. 2017. An Introduction to statistical learning with applications in R. New York: Springer. figure 9.9

• Ladd-Acosta C et al. Presence of an epigenetic signature of prenatal cigarette smoke exposure in childhood. 2016. https://doi.org/10.1016/j.envres.2015.11.014 [CDC/ATSDR]
• SVM with radial basis function kernel; 81% classification accuracy for childhood DNA methylation at 26 CpG loci as biomarker of prenatal smoking exposure
• Yanamala N et al. Sparse supervised classification methods predict and characterize nanomaterial exposures. 2018. https://doi.org/10.1177/0192623317730575 [CDC/ATSDR]
• Proteins in bronchoalveolar lavage to predict exposure to multiwalled carbon nanotubes; sparse SVM (ℓ1- and ℓ2-norm, recursive feature elimination): macrophage-derived chemokine associated with exposures

• SVMs with nonlinear kernels were state-of-the-art classifiers until deep learning
• SVM classifiers tend to perform well and apply quickly once trained; training depends on subset of points (“support vectors”) near boundary; can make dimensionality a blessing rather than a curse, sometimes rewarding higher-dimensional representation
• Can be expensive to train and can perform poorly if data aren’t separable
• Platt scaling can extend the {0, 1} support vector classifier to a continuous (0, 1) score
• Support vector technology extends to multiclass problems and to regression

and other tree-based methods

graphic adapted from M Maenner

• Maenner MJ et al. ... [S]urveillance of autism spectrum disorder. 2016. https://doi.org/10.1371/journal.pone.0168224 [CDC/ATSDR][Open Access]
• Predict ASD case status using 1-, 2-, 3-word phrases in children’s evaluations. 86.5% concordant with clinician-determined case status (Se 84.0%, PPV 89.4%).
• Gu W et al. Use of random forest to estimate population attributable fractions from a case-control study of Salmonella enterica serotype Enteritidis infections. 2015. https://doi.org/10.1017/S095026881500014X [CDC/ATSDR]
• PAFs for Salmonella illness, 7 routes, by reduction in illness associated with removing grouped exposures. Using 72 features, egg-related exposures have highest PAF (22%). Superior to logistic forward/backward selection.

graphic by M Maenner

graphic by M Maenner

graphic by M Maenner

graphic by M Maenner

• Supervised task: classification, i.e., separate cases from noncases
• A single tree partitions recursively
• Start: consider all inputs, determine best input to split cases from noncases
• Continue: for each subgroup, consider inputs and split accordingly
• Stop when pure enough (no meaningful split), or certain depth
• Random forest does this many times (say, 1,000 or 10,000)
• Randomly divide data into training set (63.2%) and test set (other 36.8%)
• Each tree: random subset of training set, variables, or both
• Construct all the trees; given new input, trees vote with equal weight to classify

• Maenner MJ et al. ... [S]urveillance of autism spectrum disorder. 2016. https://doi.org/10.1371/journal.pone.0168224 [CDC/ATSDR][Open Access]
• Predict ASD case status using 1-, 2-, 3-word phrases in children’s evaluations. 86.5% concordant with clinician-determined case status (Se 84.0%, PPV 89.4%).
• Gu W et al. Use of random forest to estimate population attributable fractions from a case-control study of Salmonella enterica serotype Enteritidis infections. 2015. https://doi.org/10.1017/S095026881500014X [CDC/ATSDR]
• PAFs for Salmonella illness, 7 routes, by reduction in illness associated with removing grouped exposures. Using 72 features, egg-related exposures have highest PAF (22%). Superior to logistic forward/backward selection.

• Single-tree methods, like CART’s classification and regression trees, are inherently unstable: small change in input → large change in result
• RFs are ensembles:
• Combine a large number of weak classifiers (single trees) → stronger classifier
• Each tree predicts a class, and all trees in the forest vote with equal weight
• Bootstrap aggregation = bagging
• AdaBoost is another ensemble tree method, giving more weight to weaker trees to improve fit; extra attention to avoid overfitting

• More characteristics of random forests:
• Fewer settings to tune compared to SVM or boosting
• Built-in variable importance measures
• Fairly robust to noninformative features; inherently resistant to overfitting
• Out-of-bag error rate approximates error in validation sample

ensembles turn
individual weakness into
group strength

Ray EL, Reich NG. https://doi.org/10.1371/journal.pcbi.1005910 figure 2

• Petersen ML et al. Super learner analysis of electronic adherence data improves viral prediction and may provide strategies for selective HIV RNA monitoring. 2015. https://doi.org/10.1097/QAI.0000000000000548 [CDC/ATSDR][Open Access]
• Super Learner (5 component models) to classify viral failure among 1478 adults on ART and monitored for adherence; cut ≥ 25% tests and still maintain Se ≥ 95%
• Herrera R et al. Estimating the causal impact of proximity to gold and copper mines on respiratory diseases in Chilean children: An application of targeted maximum likelihood estimation. 2018. https://doi.org/10.3390/ijerph15010039 [Open Access]
• Super learner: causal attributable risk for asthma or allergic rhinoconjunctivitis; living away from mine would decrease conjunctivitis risk by 4.7 percentage points

Stacking to create ensemble predictive distribution

A: predictive distributions from 3 component models. B: distributions from A, scaled (multiplied) by model weights. C: scaled distributions are literally stacked to create ensemble predictive distribution.
Ray EL, Reich NG. 2018 https://doi.org/10.1371/journal.pcbi.1005910 figure 2, page 8

• We saw 2 kinds of tree ensembles:
• Random forests use bagging—equal weight to several models
• AdaBoost uses boosting—extra weight to poorly performing models to improve learning
• These methods, and a third, combine any set of models—even different architectures
• Stacking constructs a weighted average of component models; weights are learned
• Super Learner is a popular, theory-supported stacking routine with good properties

• Petersen ML et al. Super learner analysis of electronic adherence data improves viral prediction and may provide strategies for selective HIV RNA monitoring. 2015. https://doi.org/10.1097/QAI.0000000000000548 [CDC/ATSDR][Open Access]
• Super Learner (5 component models) to classify viral failure among 1478 adults on ART and monitored for adherence; cut ≥25% tests and still maintain Se ≥ 95%
• Herrera R et al. Estimating the causal impact of proximity to gold and copper mines on respiratory diseases in Chilean children: An application of targeted maximum likelihood estimation. 2018. https://doi.org/10.3390/ijerph15010039 [Open Access]
• Super learner: causal attributable risk for asthma or allergic rhinoconjunctivitis; living away from mine would decrease conjunctivitis risk by 4.7 percentage points

• Influenza season forecasts from distinct models can be combined as an ensemble
• Influenza Division’s FluSight uses unweighted average
• Some researchers propose stacking with a dynamic weighted average:
• Ray EL, Reich NG. Prediction of infectious disease epidemics via weighted density ensembles. 2018. https://doi.org/10.1371/journal.pcbi.1005910

Deep learning starts here

Credit Hassen Bouzgou

Credit Hassen Bouzgou

• Quinn JA et al. Deep convolutional neural networks for microscopy-based point of care diagnostics. 2016. http://proceedings.mlr.press/v56/Quinn16.html [Open Access]
• CNNs on 3 microscopy tasks in Uganda: malaria in thick blood smears, TB in sputum, hookworm eggs in stool. AUROC ≥ 0.99, way better than traditional medical imaging techniques.
• Jean N et al. Combining satellite imagery and machine learning to predict poverty. 2016. https://doi.org/10.1126/science.aaf7894 [Open Access]
• Fine-tuned pretrained CNN to predict cluster-level nighttime light intensities from daytime satellite imagery; ridge regression to estimate cluster-level expenditures or assets; sidestepped need for labeled data

• Neural networks begin with the perceptron and the multilayer perceptron
• Connected, mutually dependent logistic regression models—“hidden layers”
  • “Activation functions”—sigmoid of logistic model is a smoothed step function
  • Can extend to other nonlinear forms (e.g., hyperbolic arctangent, rectified linear)
  • Feature extraction!
• MLPs are universal function approximators, meaning they can model any suitably smooth function, given enough hidden units
• Early work on neural networks focused on how to train (i.e., fit) MLPs
• How to optimize efficiently to estimate parameters internal to model (backpropagation)
• How many nodes (and layers) to use

• Convolutional neural networks’ hidden units build in more dependency (“shared weights”) across nodes, to reuse/share information (filter) across the network, which can reduce the number of parameters compared to MLPs
• Typically trained using stochastic gradient descent (cf. GPUs)
• Data might be augmented by distortions of inputs (rotate, reflect, etc.)
• Each pass over the data is an epoch
• Regularize using “dropout”
• With conditional random fields, how USPS parses handwritten ZIP Codes
• In contrast to previous computer vision tasks, CNNs can learn without user-constructed features

Smartphone microscopy

3D-printed adapter mounted on microscope; smartphone inserted into adapter
Quinn JA et al. 2016. http://proceedings.mlr.press/v56/Quinn16.html figure 1, page 4

System performance—tuberculosis

ROC and precision-recall for TB detection; Mtb bacilli in test images, where white boxes show expert annotations, and red boxes show detections by the system
Quinn JA et al. 2016. http://proceedings.mlr.press/v56/Quinn16.html figures 3 and 7

• Quinn JA et al. Deep convolutional neural networks for microscopy-based point of care diagnostics. 2016. http://proceedings.mlr.press/v56/Quinn16.html [Open Access]
• CNNs on 3 microscopy tasks in Uganda: malaria in thick blood smears, TB in sputum, hookworm eggs in stool. AUROC ≥ 0.99, way better than traditional medical imaging techniques.
• Jean N et al. Combining satellite imagery and machine learning to predict poverty. 2016. https://doi.org/10.1126/science.aaf7894 [Open Access]
• Fine-tuned pretrained CNN to predict cluster-level nighttime light intensities from daytime satellite imagery; ridge regression to estimate cluster-level expenditures or assets; sidestepped need for labeled data

• Multilayer perceptrons were a popular machine learning solution in the 1980s.
• SVMs took over for a while.
• Then backpropagation (and, later, GPUs) led to deep learning.
• While MLP is a universal function approximator, CNN approximately resolves into component objects (manifolds) that are complicated representations of features
• Other applications of CNN, beyond static images:
• NLP: parsing, search query, sentence modeling
• Drug discovery by predicting interaction between molecules and proteins (treatments for Ebola virus and multiple sclerosis)

Deep learning takes on
deep dependence structures

• Lee SH et al. Chief complaint classification with recurrent neural networks. 2018. https://arxiv.org/abs/1805.07574 [CDC/ATSDR][Open Access]
• LSTM and GRU RNNs (and 2 bag-of-word classifiers) predict discharge diagnosis group from chief complaint text. 3.6 M ED records. GRU best (F1=.47); RNN models outperformed the bag-of-word classifiers for all diagnosis groups.
• Rajkomar A et al. Scalable and accurate deep learning with electronic health records. 2018. https://doi.org/10.1038/s41746-018-0029-1 [Open Access]
• 114 K pts patients’ entire raw EHR records; ~47 B data points, no site-specific harmonization. ensemble of LSTM + 2 other deep NNs. in-hospital mortality AUROC ~.93, 30-day unplanned readmission AUROC ~.75, prolonged length of stay AUROC ~.85, all final discharge diagnoses (freq-weighted AUROC .90).

• MLPs and CNNs are feedforward networks: dependencies go in 1 direction; they cannot represent temporal dynamic behavior
• Enter recurrent neural networks (RNNs)
• Back-connections maintain (“remember”) previous states during updates, enabling sequence prediction
• Long short-term memory (LSTM) stabilizes by preventing certain kinds of errors (using a “forget gate”), thus improving ability to handle sequential data
• Gated recurrent unit (GRU) simplifies LSTM (but doesn’t do all the same tasks)
• Bi-directional RNNs process sequential input in forward and backward

• Lee SH et al. Chief complaint classification with recurrent neural networks. 2018. https://arxiv.org/abs/1805.07574 [CDC/ATSDR][Open Access]
• LSTM and GRU RNNs (and 2 bag-of-word classifiers) predict discharge diagnosis group from chief complaint text. 3.6 M ED records. GRU best (F1=.47); RNN models outperformed the bag-of-word classifiers for all diagnosis groups.
• Rajkomar A et al. Scalable and accurate deep learning with electronic health records. 2018. https://doi.org/10.1038/s41746-018-0029-1 [Open Access]
• 114 K pts patients’ entire raw EHR records; ~47 B data points, no site-specific harmonization. ensemble of LSTM + 2 other deep NNs. in-hospital mortality AUROC ~.93, 30-day unplanned readmission AUROC ~.75, prolonged length of stay AUROC ~.85, all final discharge diagnoses (freq-weighted AUROC .90).

• RNNs seem well-suited to working with relatively short bits of text, whether character-by-character or word-by-word
• Long texts (e.g., dozens or more words) are more difficult to train on, as there is more memory to manage
• RNNs preserve dependencies that bag-of-words models don’t, or that would make bag-of-words models huge to mimic
• RNN value for public health remains to be demonstrated in time series and other sequentially dependent structures

most-useful models
in head-to-head experiments

• "Essentially, all models are wrong, but some are useful.”
• — George E P Box

• Lee SH et al. A comparison of machine learning algorithms for the surveillance of autism spectrum disorder. 2018. https://arxiv.org/abs/1804.06223 [CDC][Open Access]
• Expands Maenner 2016. 8 models: RF, 5 other bag-of-words models, 2 NNs adapted from fastText. RF, NB-SVM, and SVM competitive; RF had edge.
• Reps JM et al. Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data. 2018. https://doi.org/10.1093/jamia/ocy032; Shiny http://data.ohdsi.org/PredictionViewer/
• Prediction models for 21 outcomes in pop’n of persons with depression across 4 observational databases. L1-regularized logistic regression, gradient boosting machine, RF, naive bayes models. Open (GitHub), extensible, amazing!

• We have conventional tools for comparing statistical models with similar architecture, such as information criteria (e.g., Akaike, Bayes/Schwarz)
• Compare machine-learned on empirical performance, typically by cross-validation
• Common performance measures
  • F1, harmonic mean of sensitivity and PPV, insensitive to true negatives
  • AUROC, function of sensitivity and specificity
  • Calibration
• Ensure that pseudorandomness is managed properly (k folds, train/val/test)
• Over multiple datasets: Wilcoxon signed-rank, Friedman, multiplicity adjustments and post hoc corrections

• Lee SH et al. A comparison of machine learning algorithms for the surveillance of autism spectrum disorder. 2018. https://arxiv.org/abs/1804.06223 [CDC][Open Access]
• Expands Maenner 2016. 8 models: RF, 5 other bag-of-words models, 2 NNs adapted from fastText. RF, NB-SVM, and SVM competitive; RF had edge.
• Reps JM et al. Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data. 2018. https://doi.org/10.1093/jamia/ocy032; Shiny http://data.ohdsi.org/PredictionViewer/
• Prediction models for 21 outcomes in pop’n of persons with depression across 4 observational databases. L1-regularized logistic regression, gradient boosting machine, RF, naive bayes models. Open (GitHub), extensible, amazing!

• Machine learning experiments should be reproducible
• Lee 2018 Python code is publicly available (though the data are privileged)
• Reps 2018 is an open-source example (R and Python components) of a reproducible, extensible workflow for machine learning experiments (though the data in the manuscript are privileged)

and other thoughts

for now

https://xkcd.com/1838/

• Unstructured, minimally structured, or complex data
  • Free text
  • Electronic health records
  • Images
• Exploring latent structure
• Helping to reduce or manage large numbers of features or dimensions
• Learning classification problems with large, labeled datasets
  • Including multiclass data and multilabel data

• Hearkening back to Breiman’s 2 cultures, “statistical”/generative models might be preferred to machine learning/predictive models if
• The data-generating model is of interest in itself
• The data-generating mechanism is known or controlled
• Features of interest have been identified
• Too little data available to assess out-of-sample properties (e.g., train/test)
• Successful “statistical” methods outperform known machine learning methods (e.g., forecasting as in Makridakis S et al. Statistical and machine learning forecasting methods. 2018. https://doi.org/ 10.1371/journal.pone.0194889)

• Concept of “importance” is used in many ways in ML; interpret with care
• Unchecked class imbalance can skew interpretation of results
• In epidemiology, we call this “prevalence” and note its effects on PPV and NPV
• Manage through adjusting prior class probability
• Models learn from prior assumptions and data seen; can learn bias
• Training bias is not unique to machine learning
• Women's Health Initiative revealed uncharacterized effects of hormone replacement therapy, countering understanding from observational studies
• Trained bots pick up user behaviors
• Dermatology models need to account for varieties of skin pigmentation

• No machine learning algorithm can be uniformly best for all problems, but
• Real-world learning problems aren’t uniformly distributed
• No learning algorithm can outperform another when evaluated over all possible classification problems.
• The performance of any learning algorithm, over the set of all possible learning problems, is no better than random guessing.
• Meta-learning over all possible learning problems is futile, as no single hybrid model can tell us which base model would achieve better than random performance on a particular data set.
• Escape “no-free-lunch” by understanding which problems are actually of interest to inform choice of learning algorithm.

not a machine learning task,
per se

• Tasks that look like regression
• Support vector machines
• Random forests (and other tree-based methods)
• Combining different models
• Convolutional neural networks
• Recurrent neural networks
• Comparing models

Machine learning has been applied to an increasing variety of public health problems. The set of interesting questions expands with more awareness of methods and more kinds of data.

Material will be available, including this talk, annotated bibliography, discussion boards, sample code
SharePoint: [redacted: intranet URL]
GitLab: [redacted: intranet URL]
Data Hub:
Chad Heilig, PhD • [redacted: email address]
Scott Lee, PhD • [redacted: email address]
CSELS Machine Learning • [redacted: email address]
CSELS Data Hub Webinar, 2018-11-19

https://xkcd.com/2048/

• Menon R et al. Multivariate adaptive regression splines analysis to predict biomarkers of spontaneous preterm birth. 2014. https://doi.org/10.1111/aogs.12344 [CDC/ATSDR]
• Predict preterm birth; 36 biomarkers, 191 women; feature selection stratified by maternal race
• Wheeler MW. Bayesian additive adaptive basis tensor product models for modeling high dimensional surfaces: an application to high-throughput toxicity testing. 2018. https://doi.org/10.1111/biom.12942 [CDC/ATSDR]
• Response surface from high-throughput toxicity testing platform. Simulation study shows method outperforms MARS (and some other methods).

• Holt AC et al. Spatial analysis of plague in California: Niche modeling predictions of the current distribution and potential response to climate change. 2009. https://doi.org/10.1186/1476-072X-8-38 [CDC/ATSDR]
• Maximum entropy and regularization to map wildlife plague foci from bioclimatic data. Identified case locations (AUC 0.913 to 0.948). Forecast 6 future scenarios.
• Goertzel BN et al. Combinations of single nucleotide polymorphisms in neuroendocrine effector and receptor genes predict chronic fatigue syndrome. 2006. https://doi.org/10.2217/14622416.7.3.475 [CDC/ATSDR]
• Predict CFS from SNP profiles. 43 CFS patients, 58 controls, 28 SNPs. Search ~122K rules with 1—5 SNPs; randomization test. Max accuracy 76.3%.

• It can be instructive to evaluate a model’s performance in relation to a simpler and a more complicated model
• Find model(s) that perform well—good fit (in-sample error), predict well (out-of-sample error), or both
• Find a slightly simpler model with relatable performance (could be slightly worse)
• Find slightly more complex model with relatable performance
• Consider what high-performing model offers relative to simpler or more complex
• In this way, a simpler model can be justified by reference to a more complicated model based on performance rather than just a parsimony heuristic

• Software
• R: general packages include caret, h2o; many specialized packages
• Python: general package is scikit-learn; many specialized packages
• Tutorials
• Other resources
• Machine learning methods have been developed to deal with a variety of complex and unstructured data sources