Data science and data wisdom: cultivating a data generation

[title] Thank you for having me today. “Associate Director for Data Science” is a fairly new title at CDC. I’m going to talk today about some ways that I think CDC can better support a culture for doing good things with data, especially in view of fast-moving methods, technology, and tools.

[foreword] CDC consumes a lot of data to support our public health mission. Surveillance, surveys, program evaluation, studies of health services, clinical trials. From small, structured data to big, unstructured data, the scope and scale of those data change over time, as do the methods, technology, and tools for extracting meaning from them. At CDC, we have the luxury and advantage of a shared mission, a commitment to public service, and an intense, pragmatic need to draw on expertise across disciplines in multifaceted teams.
One popular definition of data science is the science of extracting information from data. Here’s my working definition of data science: a set of core activities for asking good questions and lining up the tools to answer them rigorously using data. The tasks of stating and solving problems are often separated. This notion of data science links them, mediated through data.

[who get to] Every scientist who wants to do good things with data should have the intellectual support to do so. Their end of the bargain is to proceed with rigor in all implicated domains – the things they do with data, and the reasons they do them. (More on that soon.) I’m thinking here especially about how to empower folks who are in learning programs, like EIS or ORISE, and scientists at the beginning of their careers.
Why should we focus on data science? Because we need to keep up with rapidly changing methods, technology, and tools. Since data affords a certain amount of autonomy, unlike some other scientific domains, the enterprising inquirer can capitalize on that autonomy in order to keep up. She does so by developing know-how (through technical skills), focusing her drive (through intellectual character), and receiving support (through mentoring and community).

[sample issues]
Here are some examples of issues that data science can help with:
[click] Data science makes available tools for classic problems, such as working with data through the life cycle from collection to analytic data management, to analysis and presentation. And some problems presented as arising with “big data” differ in scale rather than kind. For example, we confront the same issues using administrative or incidental data, whether they are massive stores of insurance claims data or paper-based registers in African or Asian countries.
[click] Contemporary issues in data science arise from drives to be open and to expand the scale of data and intensity of analytic methods. Making data available for wide audiences, while ensuring adequate protections of individual privacy. Describing practices to make analyses reproducible. Working with high-volume data, such as genomics, and high-velocity data, as found in syndromic surveillance and claims data feeds. And making sense out of unstructured text.

[transition: data science story]
I want to start with a story about my identity and motivation as a data scientist. In January 2011, I had the pleasure of talking to first-year liberal-arts students at Bard College. It was a 40-minute in which I shared a story about a part that I played in the progression of data-driven evidence to improve how we can better find TB in order to stop TB. Here’s a 5-minute version.

[finding TB] TB is an important global health problem, and the effort to stop TB requires (among other efforts) that we get better at finding cases of TB, especially among persons with HIV. The Stop TB Strategy, set out in 2006, envisioned a world free of TB. As recently at 2007, the World Health Organization’s case-finding guidelines were based on expert opinion.
I collaborated as data scientist on an influential cross-sectional study in Southeast Asia. Several other rigorous studies were published around the same time.
Then I led a meta-analysis of about a dozen of these studies to synthesize this accumulating data.
In 2011, WHO published new, data-driven guidelines for finding TB to stop TB by using information on 4 symptoms among persons with HIV.
<say more about methods: searching 80 million combinations in a novel way; bivariate random-effects meta-analysis; have graphics as backup slides?>
<citations: [corrected: sources as on Find TB to stop TB (2011), slide 40, https://goodthingswithdata.com/doing/tb-find/pres/pr_20110113_find-tb-stop-tb/#slide-040]>

[stop TB/HIV syndemic] For my audience of first-year liberal arts students, I posed the broader question of what it will take to stop the TB/HIV syndemic. [click] As seen in narratives like La bohème, the arts are powerless. [click] As evidenced by the fact that TB is still with us [click], science has advanced in fits and starts and is insufficient to control TB. [click] The social dimension of TB demands compassion and political will. What is a scientist to do? Here’s <a gently updated version of> what I told them.
I am a scientist, trained in math, statistics, some philosophy, a smattering in other disciplines. I derive comfort in the realm of reason, where one can measure and count and—paradoxically—quantify uncertainty. I entered public health because I wanted to use rigorous scientific methods to help better the human condition.
<Image credit Becton-Dickinson>
<sources: [arts powerless: [corrected: La bohème, as on Find TB to stop TB (2011), slide 5, https://goodthingswithdata.com/doing/tb-find/pres/pr_20110113_find-tb-stop-tb/#slide-005]]; [hubristic: [corrected: Iseman 2000, cited below, as on Find TB to stop TB (2011), slide 11, https://goodthingswithdata.com/doing/tb-find/pres/pr_20110113_find-tb-stop-tb/#slide-011]]; “Science is crucial” quotation from pg 426 of GB Migliori et al. 125 years after Robert Koch’s discovery of the tubercle bacillus: the new XDR-TB threat. Is ‘‘science’’ enough to tackle the epidemic? European Respiratory Journal. 2007; 29: 423–427. The authors add: Basic science plus sound public health approach to use and preserve utility of scientific advances.
“Concept of scientific conquest” quotation from pg ix of MD Iseman. A Clinician's Guide to Tuberculosis. Lippincott Williams & Wilkins, 2000.>

[science and values] My personal values motivated me to pursue technical excellence and to offer those skills in public service. I wanted to honor the rigors of several domains: clinical medicine; microbiology; computation and analysis; inference and presentation.
In addition to technical skills, values motivate and shape scientific inquiry and learning, not least because TB has shown us that the arts alone, or the sciences alone, will fail to stop TB. Here are some of the values that animate my own actions: [click]
Courage, both to ask hard questions and to act on difficult answers. [click]
Humility, because it is often possible and even necessary to learn from interesting mistakes. [click]
Curiosity, because there’s no formula for telling us which interesting mistake to try to make next. [click]
Wisdom, because sometimes courage, humility, and curiosity are in tension with each other.
For finding TB to stop TB, it took willingness – courage, humility, curiosity, wisdom – to test the received expertise against competing alternatives. We obtained evidence patiently, rigorously, and incrementally. There are encouraging signs that WHO’s evidence-based guidance will be followed by the political will to act. I’m grateful to have played a small but substantial role in this one effort to stop TB and better the human condition.
That story tells you a little bit about my identity as a data scientist. In 5 years since this Bard talk, I’ve come to believe that values like courage, humility, curiosity, and wisdom are also fundamental to cultivating a data generation.
<Image credit http://imgs.xkcd.com/store/imgs/stand_back_square_0.png>
<wonder and delight?.

[transition: data-savvy skills] Let’s start with the technical skills that allow a scientist to extract information from data, that is, to ask and answer good scientific questions rigorously using data.

[core activities and epicycles] Roger Peng and Elizabeth Matsui, faculty at Johns Hopkins University, have written about 5 core activities for “the art of data science”: [click] state a good question; [click] obtain and explore data; [click] build a formal model; [click] interpret the results and their implications; and, [click] communicate the methods, results, and their significance. They go further: each core activity comes with what they call an epicycle—a practice for framing, checking, and possibly revising or revisiting each activity. [click] First, set your expectations. [click] Then collect information and compare your expectations to your information. [click] If they don't match, then take another look and either revise your expectations or your information. Maybe repeat the current core activity or return to a previous one. [click] If they do match, then you're in a good place. Sometimes check again for good measure.
<Source: Peng and Matsui 2015, pages 10, 13>

[what data scientists do] Data science affords a measure of autonomy for practicing and honing data skills, because of ready access to open methods, technology, and tools. Some technical skills require special equipment (like Petri dishes or microscopes) or access to humans (like clinical medicine or behavioral counseling). To learn about data and from data, it is often enough to have data in had, widely available software, and the persistence to jump into a problem and break it open. Software is now often freely available; consider the very active R and Python user communities. So data science can be practiced with a great deal of self-determination. With that autonomy comes the latitude to own and direct one’s learning.
Writing in 2012 for the Harvard Business Review, Tom Davenport and DJ Patil (now the US Chief Data Scientist) wrote, “[Data scientists] are able to bring structure to large quantities of formless data and make analysis possible. They ... help decision makers shift from ad hoc analysis to an ongoing conversation with data. The dominant trait among data scientists is an intense curiosity—a desire to go beneath the surface of a problem, find the questions at its heart, and distill them into a very clear set of hypotheses that can be tested.”
<examples: ID-TB/HIV in SE Asia (different solution spaces for screening rules, including combinatorial); BREMA; random forests in autism surveillance>
<sources: Schutt 2012; Davenport and Patil 2012>

[skills/modern data scientist] What skills allow someone to ask good questions and line up the tools to answer them rigorously using data? As depicted in this graphic from MarketingDistillery, data science crosses several disciplines:
Math and statistics, including statistical modeling and machine learning. Statistical modeling typically refers to using probability to think about how data might have been generated and then using data to figure out if we might be right. Contingency tables, regression models, time-to-event models are all examples. Machine learning typically refers to asking whether we can find patterns that relate inputs and outputs so that we can take new inputs and predict the associated outputs with high accuracy. Classification trees, random forests, support vector machines, natural language processing are examples. What Google search and Facebook and Microsoft Cortana and Apple Siri do.
Programming and database skills, such as computer science, software engineering, statistical computing, parallel processing for breaking up large or intensive big-data problems and reassembling output.
Communication and visualization for translating data insights into stories, decisions, actions; to convey understanding clearly to persons of varied backgrounds. <bring in examples?>
Domain knowledge, such as public health. Rigor goes both ways: using technical skills well in addition to respecting the rigors of specific domains, such as medicine and microbiology. “Soft skills” are personality traits, goals, motivations, and preferences that are valued in an applied domain. Many sources emphasize that those who practice data science should be passionate, curious problem-solvers.
<sources: Zawadzki 2014>

[map skills to core activities] Maybe you can start to appreciate how these data science skills support the core activities and their epicycles. Domain knowledge is needed to state and refine a good question. Programming and database skills help the scientist wrangle data in service to answering the question. Math and statistics skills undergird exploration and formal modeling as well as interpretation. Communication and visualization skills help develop and tell the story that answers the question.
I’ll come back to “soft skills”, but first a brief diversion about statistics.
<sources: Zawadzki 2014; Peng and Matsui 2015>

[whither statistics?] My degree is in statistics. There’s a lot of discussion about whether data science is just a marketing term for statistics.
Leo Breiman, the late UC Berkeley faculty member and an early bridge-builder between statistical and machine-learning communities, said that he might advise a young person (in 2001), “Don’t go into statistics.” In the end, he would say, “Take statistics, but remember that the great adventure of statistics is in gathering and using data to solve interesting and important real world problems.”
And Andrew Gelman, Columbia faculty member and prolific blogger, wrote in 2013, “Statistics is the least important part of data science … Statistics is important—don’t get me wrong … But it’s not the most important part of data science, or even close.”
Data science isn’t the exclusive domain of statisticians or computer scientists. Where else do data scientists come from?
<sources: Breiman 2001; Gelman 2013>

[hire or cultivate] Amidst the mixture of excitement and marketing hype about “data scientists”, there’s a recurring question about whether data scientists are recruited and hired or cultivated.
Harvard Business Review (2012) headlined data scientist as “the sexiest job of the 21st Century”. / This year, Fast Company has called it one of the best 25 jobs in America.
Data scientists are hard to find and attract. … What data scientists do … is simply not scaling at the pace of demand. <Millis 2015>
You learn data science on the job. … self-learners can catch up quickly .... But focusing only on people who call themselves data scientists is a mistake. <Van Cauwenberge 2015>
Technical skills cover the know-how for answering good scientific questions rigorously using data. Technical skills have limits and will become obsolete as methods, technology, and tools advance. To keep up with constant change, it is not enough to focus on technical skills and knowledge.
“… don’t hire on the basis of statistical or analytical capabilities alone. … seek both a skill set … and certain habits of mind, a feel for business issues and empathy for customers. Then build on all that on the job.” <Davenport and Patil 2012>
<sources: Davenport and Patil, 2012; Dishman 2016; Millis 2015; Van Cauwenberge 2015; Davenport and Patil 2012; Schutt 2012>
“Data scientists are hard to find and attract. … Data scientists are rare commodities. … What data scientists do – curate data, ask the right questions, build explanatory analytical models, implement the models into various applications – is simply not scaling at the pace of demand.” <Millis 2015>
“I believe you indeed learn data science on the job. It is true that data scientists should know [some specific technical skills] … And self-learners can catch up quickly .... But focusing only on people who call themselves data scientists is a mistake.” <Van Cauwenberge 2015>
“A prominent data scientist in Silicon Valley ... doesn’t hire on the basis of statistical or analytical capabilities. … [He] seeks both a skill set—a solid foundation in math, statistics, probability, and computer science—and certain habits of mind. He wants people with a feel for business issues and empathy for customers. Then, he says, he builds on all that with on-the-job training and an occasional course in a particular technology. <Davenport and Patil 2012>

[transition: intellectual character] For the second component of my model for a culture of data science, we can cultivate self-learning problem-solvers—responsible learners and inquirers who are better able to keep up with fast-moving methods, technology, and tools—by fostering intellectual character.
<source: Costa 2008>

[self-learning problem-solvers]
I opened with the question, “Who gets to do data science?” I can rephrase the question as, “Who wants to do good things with data, proceed with rigor, and stand behind their work?” And again as, “Who wants to do good things with data, proceed with rigor, and stand behind their work?”
My first answer is self-learning problem-solvers: people who ask thoughtful questions, pay close attention to details, honestly acknowledge what they don’t know, probe for deeper meaning, and persist in the face of obstacles. <cf Baehr 2013b>
Sure, there is a place for “training”, but in my mind the word “training” doesn’t adequately respect the agency and efficacy of the learner, who has to take an active role not only in receiving instruction but in practicing and honing what is learned.
These actions flow from a love of knowledge and learning, especially when mediated through working with data.
<I’m drawing heavily on the work of Jason Baehr, a faculty member and virtue epistemologist at Loyola Marymount University. He adds, “While soft skills are not the same thing as intellectual virtues, they include such virtues as curiosity, attentiveness, perseverance, open-mindedness, and creativity.” <Baehr 2013b>>
<sources: Baehr 2013b>

[dimensions of virtues] Intellectual virtues flow from a love of knowledge and learning, aiming at “cognitive goods”, like truth and understanding. <King 2014> Baehr <2015> describes 3 dimensions of an intellectual virtue: First, an ability or skill specific to a virtue and leading to action. For the trait of curiosity, this skill is asking good questions. Second, the motivation or commitment to apply the virtue. With curiosity, this means asking good questions because of a love of knowledge or learning. Third, the judgment or sensitivity to know when and how to exercise virtuous abilities or skills. With curiosity, this entails a sense for when to start, continue, or stop inquiry. In addition, each virtue can be seen as the mean between 2 vices—too little of a good thing and too much of a good thing. With curiosity, these vices might be indifference and obsession, respectively.
<sources: King 2014; Baehr 2015; comic by Leo Collum, © Condé Nast>

[virtues by stage of inquiry] [click] A few virtues relate to how to start learning or start an inquiry and ensure that it heads in the right direction: In addition to curiosity, intellectual autonomy is the ability to think for oneself, and intellectual humility is the ability to admit one’s limitations—to know what you don’t know. [click] Notice how these virtues relate to my operational definition of data science: “ask good questions and line up the tools to answer them rigorously using data”.
After starting an inquiry, a few virtues assist the learner in keeping on track: Attentiveness is the ability to engage, to look and listen, and to notice details. Carefulness is the ability to spot and avoid errors. Thoroughness is the ability to go deep in order to gain understanding and to explain. Observe how these virtues relate to the epicycles associated with the 5 core activities of the art of data science: setting expectations, gathering information, and checking expectations against information.
Even on track to learning, one is likely to encounter obstacles. Within an epicycle, expectations and gathered information might not match. A learner benefits from intellectual virtues that help her to work through or around obstacles: Open-mindedness helps her to think outside the box when confronted with a challenge to solve. Courage helps her to be bold and to take intellectual risks. Tenacity or perseverance helps her to embrace struggle while working through a challenge.
<sources: King 2014; Baehr 2015, 2013b>

[balance demands] But virtues can conflict with each other: courage can conflict with humility when the drive to take an intellectual risk runs counter to the limitations of one’s abilities. To navigate these conflicts, the good learner or thinker is aided by practical wisdom. This trait allows the inquirer <phronimos, per Baehr 2013a> to grasp which intellectual activity is most valuable for attaining her goals. Recall that Baehr <2015> identifies one dimension of an intellectual virtue as judgment or sensitivity about when and when not to exercise that virtue. Practical wisdom undergirds this dimension and also allows the good learner or thinker to take suitable action when intellectual virtues conflict. <see Greco and Turri 2011, Baehr 2013a>
<sources: Baehr 2013a, 2013b, 2015; Greco and Turri 2011; statue at La Rochelle city hall by Libersac, photo by Selbymay>
<weigh the demands of all the relevant intellectual virtues in a given situation and take suitable action. In a conflict between courage and humility, for example, the virtuous thinker might temper her intellectual risks to exceed only slightly her current abilities. In this way, she still does a new thing, but her reach is less likely to grossly exceed her grasp.>

[virtues for data science] Even if a good learner or thinker is motivated to apply intellectual virtues, she still needs to develop the abilities and judgment that connect her motivation to right action. How does she do that? Intellectual virtues are developed by practicing them. You get better at courage by practicing courage—by taking intellectual risks and learning from the consequences. You get better at humility by practicing humility—by owning your limitations and not shying away from them.
A culture of data science is stronger when self-learning problem-solvers can see these intellectual virtues modeled and receive guidance – even wisdom – from peers and experienced scientists.

[transition: data culture] The intellectually virtuous data scientist needs the support of a data culture. She needs a community of practice that empowers self-learning and fosters practical wisdom.

[mentors]
In my model for a data culture, data science mentors create supportive conditions to guide other scientists to learn about data and to learn from data. A mentor is responsible for guiding technical skills to encourage a personalized direction for self-learning. <such as managing relational databases, smoothing binary outcomes, modeling seasonality, or exploring categorical data using mosaic plots.> Mentors also model intellectual virtues and exercise practical wisdom. They show by example ways to practice curiosity, courage, humility. I often do this by thinking out loud, usually when I have no idea whether I’m making sense. This is honest, but also an exercise in vulnerability. Mentors also create regular opportunities for the student to practice intellectual virtues by stimulating curiosity, rewarding courage, and fostering humility.
<sources: Schutt 2012; Baehr 2015; >
“In education in traditional settings, we focus on answers. But what we probably should focus on is how a student behaves when they don’t know the answer. We need to have qualities that help us find the answer.” <Schutt 2012>

<personalize>
[mentor for growth] The mentor also guides the less experienced data scientist to develop wisdom for herself. The mentor must earn the trust and respect of the mentoree, so that the mentoree is able to make interesting mistakes.
In the context of education, Baehr <2013b> explains that educating for intellectual character growth is personal, because it involves thinking of students as persons whose basic beliefs, attitudes, and feelings about knowledge and learning also matter critically to the quality of their education; it is necessarily social or relational, because personal change and growth occur most readily in the context of trusting and caring relationships; and, it is reflective, because it involves reflecting on and discussing with students the value of thinking and learning, regularly pausing to identify or reflect on the significance of what is being learned.
The mentoring relationship responds to and nurtures a love of and interest in thinking, learning, answering good questions. “Intellectual virtues flow, not from a desire for praise or approval, but out of a genuine interest in thinking and learning.” <Baehr 2015>
“An intellectual virtues educational model is poised to strike a sensible and attractive balance between promoting academic rigor while also being sufficiently caring and personal.” <Baehr 2013b>
<sources: Baehr 2013b, 2015>

[mentoring directions]
In a robust culture supported by mentoring, the mentoring relationships go in several directions. The most apparent is the relationship of the mentor to the learner – fellows, students, and early-career scientists. In communities of peers, learners can mentor each other in the same personal, relational, and reflective ways as mentors with learners. Finally, the practice of mentoring itself warrants guidance and wisdom from experienced mentors, or what I call here “meta-mentoring”.

[community of mentors and peers] A robust culture of data science also needs a strong community. Some learning programs, such as EIS and the Prevention Effectiveness Fellowship, build community into the experience. Others, such as working with masters students on a practicum or ORISE research participants, require extra effort to create community. Where these communities exist, they can reinforce learning, but they don’t necessarily focus on data science.
CDC has a number of interest groups and workgroups that also create community, such as the Statistical Advisory Group, the R Users Group, the Health Economics Research Group, the Modeling Infectious Diseases group, the scientific computing facility, and more that I can’t enumerate here. These communities require energy and attention to nurture.
Finally, we’re launching a complementary effort to host conversations on thinking, learning, and talking about data in science: Expression of data science is a blog that is internal to CDC, launching with the first post today.

[transition: data generation] I have talked through 3 major components of my vision for a data science culture: know-how through technical skills, drive built on intellectual character, and support through mentoring and community.

[afterword]
At CDC, we have the luxury and advantage of a shared mission, a commitment to public service, and an intense, pragmatic need to draw on expertise across disciplines in multifaceted teams. By all means, we should hire great talent. But we also need to sow, till, and tend staff already on board. For our data generation, we must foster technical skills and a mastery of technique that allows scientists to extract information from data; we must foster intellectual virtues, including practical wisdom, that guide both inquiry and self-learning, and that enables scientists to ask good questions and to line up tools to answer those questions rigorously with data; and, we must foster a culture of mentoring, peer support, and advocacy in a community of practice that empowers data scientists to keep up with fast-moving methods, technology, and tools.
<democratize/liberate/promote practice among everyone who wants to do useful things with data; guided self-learning>

[CSELS ADDS] With this new “data science” title, I will promote the following:
Data resources and communities, such as DHIS Data Hub. Data science skills for learners in programs like those in DSEPD (EIS, PE, PHI) and others (ORISE), fostering technical skills and intellectual virtues; mentoring and peer support (direct mentoring of fellows, students, early-career scientists, meta-mentoring of other mentors, and communities of practice among peers), and a small part of the data science community and culture through a new data science blog.

PHI can be defined as “the systematic application of information and computer science and technology to public health practice, research and learning.” [p. 68 WA Yasnoff et al. Public health informatics: Improving and transforming public health in the information age. Journal of Public Health Management and Practice 2000; 6(6): 67–75.]
“In contrast to a health IT professional working specifically on troubleshooting technology or infrastructure issues, a PHI specialist leverages information and computer science to support public health goals and decision-making and defines the ‘how and why’ science behind the technological tool or approach” [TG Savel, S Foldy, S. The role of public health informatics in enhancing public health surveillance: CDC’s Vision for Public Health Surveillance in the 21st Century. Morbidity and Mortality Weekly Report 2012; 61(3): 20–24.].

<sources: King 2014; Baehr 2015, 2013b>

[favorite experiences]
As a collaborating analyst, my favorite experience is breaking open a data problem after struggling mightily, to delight in unexpected discovery. As a mentor, my favorite experience is guiding an early-career scientist through the same struggle, wonder, and delight – especially scientists who had limited experience with data.
Over the past 15 years or so, I’ve mentored about 3 dozen students, postdoctoral fellows, and early-career scientists from medicine, epidemiology, mathematics, philosophy, and other fields.
Sometimes, folks from other fields apologize for claiming any authority in using data science methods to address important questions. In my view, as long as they practice the methods with appropriate rigor—often guided by someone who identifies professionally with the field—then they should take pride in using those methods to address questions that they develop from their subject-matter expertise.

[mentors] The word mentor is borrowed from Homer's Odyssey, where the goddess Athena took on the appearance of the character Mentor to guide Telemachus in his search for his father Odysseus. Now it means someone who imparts wisdom to and shares knowledge with a less experienced colleague.
“In education in traditional settings, we focus on answers. But what we probably should focus on is how a student behaves when they don’t know the answer. We need to have qualities that help us find the answer.” <Schutt 2012>
Over the past 15 years or so, I’ve mentored about 3 dozen students, postdoctoral fellows, and early-career scientists from medicine, epidemiology, mathematics, philosophy, and other fields. In my model for cultivating a data generation, data science mentors create supportive conditions to guide other scientists to learn about data and to learn from data. A mentor is responsible for guiding technical skills and for modeling intellectual virtues, for exercising practical wisdom, for creating regular opportunities to practice intellectual virtues, and for guiding the less experienced data scientist to develop wisdom for herself. The mentor must earn the trust and respect of the mentoree, so that the mentoree is able to make interesting mistakes. It’s difficult for the student to be vulnerable on purpose, so the mentor must remain mindful of the difference in power, and to share power that enables the mentoree to thrive and grow intellectually.
<sources: Schutt 2012; Baehr 2015; image by Pablo E Fabisch 1699>

[mentorees, peers]
In turn, those being mentored in the practice of data science bear a few responsibilities.
They must take responsibility for self-learning.
They must seek opportunities to be mentored and to practice intellectual virtues, mimicking those that are modeled, such as open-mindedness, courage, and tenacity.
They must take risks and make interesting mistakes, acknowledging your vulnerability and accepting power when empowered.
They must know themselves (or get to know themselves).
They should strive to be problem-solvers, to be question-askers, to think about their process, to use data responsibly and make the world better, not worse. <cf Schutt 2012>
They should support and mentor their peers. In mutual encouragement, they are better able to keep up with fast-moving methods, technology, and tools; and to know when good questions call for them.
CDC has several communities of practice that promote peer support, of which the R Users Group is a leading example; also the SAG listserv, R Shiny subgroup, MiND, HERG, BSSWG, SMWG, and SciComp, to list a few.
<sources: Schutt 2012>

[advocates] A data culture also needs advocates. More than resources such as staffing or money or even “training”, I’m thinking here about advocates who influence how we think and talk and act about the practice and profession of data science. Experts in data science should celebrate, promote, and share intellectual engagements in data-oriented methods. Advocates should promote both the practices and the professions of data science.

[afterword] One popular definition of data science is the science of extracting information from data. Here’s my working definition of data science: a set of core activities for asking good questions and lining up the tools to answer them rigorously using data. Today I’m going to talk about [click] who gets to do data science and [click] how I think we should go about empowering the practice of data science. [click] To keep up with fast-moving methods, technology, and tools for learning from data, scientists—especially scientists of the current generation—need data-savvy skills, data-wise values, also called intellectual virtues, and data-empowering cultural resources. These 3 components are how we can cultivate a data generation.
<democratize/liberate/promote practice among everyone who wants to do useful things with data; guided self-learning>

[wall-of-data graphic] CDC consumes a lot of data to support our public health mission. Surveillance, surveys, program evaluation, studies of health services, clinical trials. From small, structured data to big, unstructured data, the scope and scale of those data change over time, as do the methods, technology, and tools for extracting meaning from them. At CDC, we have the luxury and advantage of a shared mission, a commitment to public service, and an intense, pragmatic need to draw on expertise across disciplines in multifaceted teams. How can we empower public health scientists to ask good questions and line up the tools to answer them rigorously using data?

[state] For the first core activity, State a good question, the epicycles might look something like this: [click] Set expectations by asking, “Is the question of interest to the audience? Will it advance public health?” Collect information by consulting experts and reviewing the literature and other sources.
If the initial question is too broad or narrow, is implausible or defies an answer, then the question needs to be refined. A certain amount of pluck might be needed at the outset. The need to revise might not become apparent until much later in the process.
<sources: Peng and Matsui 2015>

[obtain, explore] The next core activity is to obtain and explore data that might answer the question. Sometimes you get the chance to design how data are collected, as with a prospective research study. Often the task is to repurpose data that were collected for some other purpose—maybe even data that were never intended to support scientific inquiry. [click] Set expectations by asking whether the data are appropriate for the question. Collect information about the data by examining them early and often, both to learn about the data and to learn from the data. If the data don’t seem to address the point, then either the data or the question might need to be revisited.
<sources: Peng and Matsui 2015>

[build] When the data seem suitable to answer the question, the next activity is to develop a structured answer—a formal mathematical or statistical model. [click] The expectation is that the model answers the question. The analyst might start with an assumption about how the data were generated, like a set of events that follow some probability distribution to encode relationships between characteristics. This data-generation model might infer an association or explain causes and effects based on some ideal of how the data were produced. Or it might be a predictive model that accurately figures out a likely outcome given a set of inputs. You check your expectations through sensitivity analysis—asking how your conclusion might change if the assumptions or the data differed in some way. Challenge your results at every step of the analysis. You might need to revise your inputs or your model structure.
<sources: Peng and Matsui 2015>

[interpret] You next need to interpret your results and their implications. “Next” doesn’t mean follow a strict sequence, as these activities are closely related and iterative. This is called out as a separate core activity to ensure that the analyst can show others that she has rigorously answered the question. [click] Set expectations by asking whether the analysis provides a specific, meaningful answer to the question. Assess expectations by considering the totality of analyses: How large is the effect that answers the question? How accurate the prediction? How much uncertainty or error is left unresolved? If needed, revise analysis or models to provide a specific and meaningful answer.
<sources: Peng and Matsui 2015>

[communicate] Analytic work has its impact [click] when it can be conveyed to an audience in a way that is understandable, complete, and meaningful to an audience. You can check your expectation by seeking feedback from your audiences and, if needed, revise analyses or the approach to presentation.
<sources: Peng and Matsui 2015>