The following package(s) will be installed:
- tidyverse [2.0.0]
These packages will be installed into "~/Documents/Teaching/computational_statistics/websites/website_2024/renv/library/R-4.3/aarch64-apple-darwin20".
# Installing packages --------------------------------------------------------
- Installing tidyverse ... OK [linked from cache]
Successfully installed 1 package in 3 milliseconds.
Data manipulation with tidyverse
Introduction to R: Class 3
dplyr
Objectives
In the last section, we imported a clinical cancer dataset and learned to manipulate it using base R (tools included in every R installation).
In this session, we’ll continue to work with the same dataset, but will introduce a set of tools specifically designed for data science in R. By the end of this session, you should be able to:
- install and load packages
- use tidyverse tools to import data and access rows/columns
- combine commands using pipes
- transform and summarize data
Installing and loading packages
Please ensure RStudio is open with your project directory path (e.g., ~/Desktop/intro_r
) listed at the top of your Console. If you do not see the path to your project directory, go to File -> Open Project
and navigate to the location of your project directory. Alternatively, using your operating system’s file browser, double click on the r_intro.Rrpoj
file.
Create a new R script called class3.R
, and add # Introduction to R: Class 3
as a title.
For this lesson, we’ll be working with a group of R packages called tidyverse
. A package is a group of related functions that help you accomplish particular tasks. tidyverse
packages have been designed specifically to support tasks related to data science, such as data manipulation, filtering, and visualization.
The first thing we need to do is install the software:
A few notes about installing packages:
- You only need to perform this installation once per computer, or when updating R or the package.
- If you see red text output in the Console during this installation, don’t be alarmed: this doens’t necessarily indicate a problem. You are seeing a report of the various pieces of software being downloaded and installed.
- If prompted, you should install all packages (say yes or all), as well as yes to compiling any packages
- When the installation is complete (this may take several minutes), you’ll see the command prompt (
>
) in your Console.
Once you have the software installed, you’ll need to load it:
# load library/package
library(tidyverse)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.1.2 ✔ readr 2.1.4
✔ forcats 1.0.0 ✔ stringr 1.5.0
✔ ggplot2 3.4.3 ✔ tibble 3.2.1
✔ lubridate 1.9.2 ✔ tidyr 1.3.0
✔ purrr 1.0.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
Loading packages is similar to opening a software application on your computer; it makes a previously installed set of software available for use. A few notes about loading packages:
- You’ll need to load packages every time you open RStudio (or R restarts)
- Loading
tidyverse
loads a collection of packages; these are listed under “Attaching packages” - There are many other packages included as dependencies. If some of them did not install successfully, you will receive an error at this step. For this lesson, you can try
library(dplyr)
, and ask your instructor for help later. - The section in the output above referencing “Conflicts” shows you which functions you just loaded have names identical to packages you already have loaded (in base R). This shouldn’t affect the code we write in this lesson, though it’s useful to know the double colon syntax (::) allows you to reference functions in a different package with same name.
You can check to make sure the new package we’ll be using is available by executing ?select
in the Console, or by searching for that function in the help panel. You can also look in the “Packages” tab in the same panel. If the package (in this case, either tidyverse
or dplyr
) is present in the list, it’s installed. If the box next to the package name is checked, it’s loaded. In this lesson, if you receive an error saying a function isn’t available or recognized, check to make sure the package is loaded.
Selecting columns and rows
The first task we’ll undertake with our newly installed tidyverse
tools is importing our data:
# import data
<- read_csv("data/clinical.csv") clinical
Rows: 6832 Columns: 20
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (12): primary_diagnosis, tumor_stage, vital_status, morphology, state, t...
dbl (8): age_at_diagnosis, days_to_death, days_to_birth, days_to_last_follo...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Note that this function looks similar to what we used in the last lesson (read.csv
), but the underscore replacing the dot means it’s actually a different function. In fact, this is the data import function from tidyverse
. The output provided by this function indicates a few key differences with our import yesterday.
The data import execution includes a description of how each variable (column) is interpreted. In our data’s case, the numeric data are col_double
and the character data are col_character
(not factors!).
We can explore these differences further:
# inspect object
str(clinical)
spc_tbl_ [6,832 × 20] (S3: spec_tbl_df/tbl_df/tbl/data.frame)
$ primary_diagnosis : chr [1:6832] "C34.1" "C34.1" "C34.3" "C34.1" ...
$ tumor_stage : chr [1:6832] "stage ia" "stage ib" "stage ib" "stage ia" ...
$ age_at_diagnosis : num [1:6832] 24477 26615 28171 27154 29827 ...
$ vital_status : chr [1:6832] "dead" "dead" "dead" "alive" ...
$ morphology : chr [1:6832] "8070/3" "8070/3" "8070/3" "8083/3" ...
$ days_to_death : num [1:6832] 371 136 2304 NA 146 ...
$ state : chr [1:6832] "live" "live" "live" "live" ...
$ tissue_or_organ_of_origin : chr [1:6832] "C34.1" "C34.1" "C34.3" "C34.1" ...
$ days_to_birth : num [1:6832] -24477 -26615 -28171 -27154 -29827 ...
$ site_of_resection_or_biopsy: chr [1:6832] "C34.1" "C34.1" "C34.3" "C34.1" ...
$ days_to_last_follow_up : num [1:6832] NA NA 2099 3747 NA ...
$ cigarettes_per_day : num [1:6832] 10.96 2.19 1.64 1.1 NA ...
$ years_smoked : num [1:6832] NA NA NA NA NA NA NA NA NA NA ...
$ gender : chr [1:6832] "male" "male" "female" "male" ...
$ year_of_birth : num [1:6832] 1936 1931 1927 1930 1923 ...
$ race : chr [1:6832] "white" "asian" "white" "white" ...
$ ethnicity : chr [1:6832] "not hispanic or latino" "not hispanic or latino" "not hispanic or latino" "not hispanic or latino" ...
$ year_of_death : num [1:6832] 2004 2003 NA NA 2004 ...
$ bcr_patient_barcode : chr [1:6832] "TCGA-18-3406" "TCGA-18-3407" "TCGA-18-3408" "TCGA-18-3409" ...
$ disease : chr [1:6832] "LUSC" "LUSC" "LUSC" "LUSC" ...
- attr(*, "spec")=
.. cols(
.. primary_diagnosis = col_character(),
.. tumor_stage = col_character(),
.. age_at_diagnosis = col_double(),
.. vital_status = col_character(),
.. morphology = col_character(),
.. days_to_death = col_double(),
.. state = col_character(),
.. tissue_or_organ_of_origin = col_character(),
.. days_to_birth = col_double(),
.. site_of_resection_or_biopsy = col_character(),
.. days_to_last_follow_up = col_double(),
.. cigarettes_per_day = col_double(),
.. years_smoked = col_double(),
.. gender = col_character(),
.. year_of_birth = col_double(),
.. race = col_character(),
.. ethnicity = col_character(),
.. year_of_death = col_double(),
.. bcr_patient_barcode = col_character(),
.. disease = col_character()
.. )
- attr(*, "problems")=<externalptr>
You may notice the presence of tbl
and related labels in the classes for this object. tbl
stands for tibble, which is a type of data frame with specific constraints to ensure better data handling.
If you preview the dataset, it will look the same, and we can interact with the data in the same way. These assumptions about the data mesh nicely with the other tools in the tidyverse
.
Now that our data are imported, we can explore the tidyverse
functions for extracting parts of the dataset.
First, we can explore selecting certain columns by name:
# selecting columns with tidyverse (dplyr)
<- select(clinical, tumor_stage, ethnicity, disease) sel_columns
The syntax for the select
function is to specify the dataset first, then the names of each of the columns you would like to retain in the output object. If we look at the object, we’ll see it has only three columns but all rows.
You’ll note that the column headers don’t require quotation marks; this is a shortcut programmed into tidyverse
functions.
As with base R functions, we can also select a range of columns:
# select range of columns
<- select(clinical, tumor_stage:vital_status) sel_columns2
In addition to these approaches, we can also use other helper functions for selecting columns: starts_with()
, ends_with()
, and contains()
are examples that assist in extracting columns with headers that meet certain conditions. For example, using starts_with(tumor)
in place of the column names will give you all columns that start with the word tumor.
We can use a separate function to extract rows that meet particular conditions:
# select rows conditionally: keep only lung cancer cases
<- filter(clinical, disease == "LUSC") filtered_rows
The syntax here is similar to select
, and the conditional filters can be applied in similarly to base R functions.
Challenge-columns
Create a new object from clinical called race_disease that includes only the race, > ethnicity, and disease columns.
Challenge-rows
Create a new object from race_disease called race_BRCA that includes only BRCA (from
disease
).
Combining commands
The last challenges used an intermediate object to obtain an object with two subsetting methods applied. It’s common in data science to apply more than one requirement for extracting data. If you want to avoid creating an intermediate object, you could nest one command inside the other:
# same task as challenges, but nested commands
<- select(filter(clinical, disease == "BRCA"), race, ethnicity, disease) race_BRCA2
In this case, filter(clinical, disease == "BRCA")
becomes the input for select
.
While this is a common approach, especially in base R, it can be difficult for us as coders to read and interpret the code.
One of the most useful features of tidyverse
is its inclusion of a programming method called pipes. This approach can be found in many programming languages, in part because of its utility: a pipe sends the output from the lefthand side of the symbol as the input for the righthand side. In R, pipes are represented as %>%
.
We can use pipes to connect the same two data extraction tasks:
# same task as above, but with pipes
<- clinical %>%
piped select(race, ethnicity, disease) %>%
filter(disease == "BRCA")
The command above starts by naming the object that will result from this assignment. The dataset is named as the first input. Because executing the name of an object sends the object contents as output, this means the second line receives the object as input. The output from the select
line is sent as input to the filter
line. This effectively demonstrates how pipes can be used to connect multiple commands together.
Now that we are running code in chunks that span multiple lines, you can see one of the other nice features of RStudio: your cursor can be placed on any line of the multi-line chunk when you execute, and the entire set of code will run together.
These examples also help highlight the importance of style and convention in code formatting. After the first line, the code is indented. While this isn’t necessary for the code to work, it does make it a lot easier to read and understand the code.
Let’s take a look at another example of piped commands:
# extract race, ethinicity, and disease from cases born prior to 1930
<- clinical %>%
piped2 filter(year_of_birth < 1930) %>%
select(race, ethnicity, disease)
In the code above, we’re applying a mathematical condition to find specific rows, and the selecting certain columns. Does the order of commands differ? We can switch the order of the filter
and select
lines to see:
<- clinical %>%
piped3 select(race, ethnicity, disease) %>%
filter(year_of_birth < 1930)
The code above should give you an error, because in this case, the order does matter! The output from the second line does not include the year_of_birth
column, so R is unable to apply the filter in the third line.
Challenge-pipes
Use pipes to extract the columns
gender
,years_smoked
, andyear_of_birth
from the objectclinical
for only living patients (vital_status) who have smoked fewer than 1cigarettes_per_day
.
Mutate
This lesson so far has mostly shown new ways of accomplishing the same tasks we learned in the last lesson. tidyverse
includes much more functionality, however, including the ability to mutate
columns. Common tasks for which mutate
is useful include unit conversions, transformation, and creating ratios from among existing columns.
We can use this function to convert the days_to_death
column to years:
# convert days to years
<- clinical %>%
clinical_years mutate(years_to_death = days_to_death / 365)
The actual conversion works by providing a formula (days_to_death / 365
) and the name of the new column (years_to_death
). If you inspect the resulting object, you’ll see years_to_death
added as a new column at the end of the table.
mutate
works by retaining all previous columns and creating new columns as per the formula specified.tidyverse
also includestransmute
, which drops the existing columns used to calculcate the new columns.
We can use mutate
to perform multiple conversions at once:
# convert days to year and months at same time, and we don't always need to assign to object
%>%
clinical mutate(years_to_death = days_to_death / 365,
months_to_death = days_to_death / 30) %>%
glimpse() # preview data output
Rows: 6,832
Columns: 22
$ primary_diagnosis <chr> "C34.1", "C34.1", "C34.3", "C34.1", "C34.3…
$ tumor_stage <chr> "stage ia", "stage ib", "stage ib", "stage…
$ age_at_diagnosis <dbl> 24477, 26615, 28171, 27154, 29827, 23370, …
$ vital_status <chr> "dead", "dead", "dead", "alive", "dead", "…
$ morphology <chr> "8070/3", "8070/3", "8070/3", "8083/3", "8…
$ days_to_death <dbl> 371, 136, 2304, NA, 146, NA, 345, 716, 280…
$ state <chr> "live", "live", "live", "live", "live", "l…
$ tissue_or_organ_of_origin <chr> "C34.1", "C34.1", "C34.3", "C34.1", "C34.3…
$ days_to_birth <dbl> -24477, -26615, -28171, -27154, -29827, -2…
$ site_of_resection_or_biopsy <chr> "C34.1", "C34.1", "C34.3", "C34.1", "C34.3…
$ days_to_last_follow_up <dbl> NA, NA, 2099, 3747, NA, 3576, NA, NA, 1810…
$ cigarettes_per_day <dbl> 10.9589041, 2.1917808, 1.6438356, 1.095890…
$ years_smoked <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 26…
$ gender <chr> "male", "male", "female", "male", "male", …
$ year_of_birth <dbl> 1936, 1931, 1927, 1930, 1923, 1942, 1953, …
$ race <chr> "white", "asian", "white", "white", "not r…
$ ethnicity <chr> "not hispanic or latino", "not hispanic or…
$ year_of_death <dbl> 2004, 2003, NA, NA, 2004, NA, 2005, 2006, …
$ bcr_patient_barcode <chr> "TCGA-18-3406", "TCGA-18-3407", "TCGA-18-3…
$ disease <chr> "LUSC", "LUSC", "LUSC", "LUSC", "LUSC", "L…
$ years_to_death <dbl> 1.0164384, 0.3726027, 6.3123288, NA, 0.400…
$ months_to_death <dbl> 12.366667, 4.533333, 76.800000, NA, 4.8666…
The code above also features a new function, glimpse
, that can be useful when developing new piped code. Note that we did not assign the output above to a new object; we allowed it to be printed to the Console. Because this is a large dataset, that type of output can be unweildy. glimpse
allows us to see a preview of the data, including the two new columns created.
Challenge-lung
Extract only lung cancer patients (LUSC, from disease) and create a new column called
total_cig
representing an estimate of the total number of cigarettes smoked (use columnsyears_smoked
andcigarettes_per_day
).
Split-apply-combine
Our clinical
dataset includes categorical (character) data. One example is the gender
column. We can assess the different categories available using a base R function:
# show categories in gender
unique(clinical$gender)
[1] "male" "female" NA
tidyverse
includes an approach called split-apply-combine that allows us to:
- split data into groups,
- apply a task for each group,
- combine the results back together into a single table.
We can try out this approach by counting the number of each gender in our dataset:
# count number of individuals of each gender
%>%
clinical group_by(gender) %>%
tally()
# A tibble: 3 × 2
gender n
<chr> <int>
1 female 3535
2 male 3258
3 <NA> 39
group_by
is not particularly useful by itself, but powerful together with a second function like tally
. The two columns in the resulting tibble represent the categories from group_by
and the number of cases for each gender (n).
An additional function for use with group_by
is summarize
:
# summarize average days to death by gender
%>%
clinical group_by(gender) %>%
summarize(mean_days_to_death = mean(days_to_death, na.rm = TRUE))
# A tibble: 3 × 2
gender mean_days_to_death
<chr> <dbl>
1 female 947.
2 male 826.
3 <NA> NaN
Similar to mutate
, we provide summarize
with a formula indicating how we would like the groups to be handled.
In the command above, we use na.rm = TRUE
to exclude missing data from the calculation of mean from days_to_death
. We still have NA
reported in the output table, though, because of the NA
category in gender
.
We can apply an additional filter to remove this missing data, prior to grouping:
# remove NA
%>%
clinical filter(!is.na(gender)) %>%
group_by(gender) %>%
summarize(mean_days_to_death = mean(days_to_death))
# A tibble: 2 × 2
gender mean_days_to_death
<chr> <dbl>
1 female NA
2 male NA
Challenge-smoke-complete
Create object called smoke_complete from clinical that contains no missing data for cigarettes per day or age at diagnosis.
Challenge-save
How do you save resulting table to file? How would you find this answer?
The solution to the challenges above represent the first of two datasets we’ll be using for data visualization in our next class. Make sure you’ve executed this code to save the filtered data file for use next time:
<- clinical %>%
smoke_complete filter(!is.na(age_at_diagnosis)) %>%
filter(!is.na(cigarettes_per_day))
write_csv(smoke_complete, "data/smoke_complete.csv")
The command above uses write_csv
, which is the tidyverse
method of saving a csv file. Base R possesses a function, write.csv
, that performs a similar task, but by default includes quotation marks around cells with character data as well as row names (sequential numbers, unless otherwise specified).
Challenge-birth-complete
Create a new object called birth_complete that contains no missing data for year of birth or vital status.
This challenge begins filtering the second of our two datasets for next time. Make sure you include the filter to remove missing data that’s been encoded as “not reported”!
# make sure ALL missing data is removed!
<- clinical %>%
birth_complete filter(!is.na(year_of_birth)) %>%
filter(!is.na(vital_status)) %>%
filter(vital_status != "not reported")
Filtering data based on number of cases of each type
We’re going to perform one last manipulation on this second dataset for next time, which will allow us to reduce the total number of cancer types present in this dataset.
First, we’ll need to count how many cases for each cancer type exist in the dataset:
# counting number of records in each cancer
<- clinical %>%
cancer_counts count(disease) %>%
arrange(n)
The count
function is similar to tally
, but doesn’t need to have group_by
applied first. The arrange
function added at the end sorts the table using the column specified. Although this isn’t necessary for the analysis to proceed, it makes it easier for us to interpret the results.
Next, we’ll identify which cancer types are represented by at least 500 cases in this dataset:
# get names of frequently occurring cancers
<- cancer_counts %>%
frequent_cancers filter(n >= 500)
We can then use this object to filter based on the number of cases:
# extract data from cancers to keep
<- birth_complete %>%
birth_reduced filter(disease %in% frequent_cancers$disease)
The new syntax here is %in%
, which allows you to compare each entry in disease
from birth_complete
to the disease column in frequent_cancers
(remember that frequent_cancers$disease
means the disease
column from frequent_cancers
). This keeps only cases from the birth_complete
dataset that are from cancers that are frequently occurring.
Finally, we’ll write the final output to a file:
# save results to file in data/ named birth_reduced
write_csv(birth_reduced, "data/birth_reduced.csv")
Challenge-tumor
Extract all tumor stages with more than 200 cases (Hint: also check to see if there are any other missing/ambiguous data!)
Wrapping up
In this session, we acquainted ourselves with tidyverse
, and learned some tools for data filtering and manipulation. We covered examples from many of the main categories of data manipulation tasks; if you’d like more information on these functions and others available (including methods of joining multiple tables together), please check out the dplyr
cheatsheet.
In the next session, we’ll wrap up the course by creating publication-quality images using ggplot2
, a data visualization package in tidyverse
, and the two datasets we filtered in the sections above.
Extra exercises
The following exercises all use the same clinical
data from this class.
Challenge-ethnicity
How many hispanic or latino individuals in clinical are not also white? What are their races?
Challenge-years
Create a new column for clinical called age_at_death
that calculates this statistic (in years) from year_of_birth
and year_of_death
.
Challenge-helpers
dplyr
includes several “helpers” that allows selection of columns meeting particular criteria (described on the first page of the dplyr cheatsheet near the top of the right hand column: https://github.com/rstudio/cheatsheets/raw/master/data-transformation.pdf). Using one of these tools, extract all columns that include “diagnosis”.
Challenge-combine
How many patients are hispanic or latino patients (column ethnicity), died after the year 2000 (year_of_death), and possess no missing data for cigarettes per day?