importing.qmd

# Importing data and saving analysis outputs {#sec-import-save}

## Questions

-   Can users access and use internal datasets for analysis in R? How do I load the datasets?

-   How does one import external datasets into R? What functions and packages are commonly used for this purpose?

-   How do I handle different data formats such as CSV, Excel, or text files?

-   Once the user completes their data analysis, how can their output(s) be saved?

-   How can I create new data objects or modify existing ones within R?

## Learning Objectives

-   Learn how to load and access internal R datasets.

-   Import external datasets into your R project folder.

-   Demonstrate how to save analysis outputs from R such as tables and datasets.

-   Understand methods for importing data stored internally in R and from external sources.

-   Utilize built-in functions and packages to read various data formats like CSV, Excel, and text files.

## Lesson Content

### Introduction

A dataset is a collection of data, and R offers the user the opportunity to either use internal datasets or import external datasets for analysis. In this lesson, we will cover the basics of accessing internal datasets, the importation of external datasets, as well as the saving of analysis outputs, such as plots and tables.

### Working with internal datasets

R has several built-in datasets. To access them, we use the `data()` function after loading in the datasets library.

```{r}

#| eval: false
#| include: false
library(datasets)

print(data())
```

To find out more about a specific dataset, we use the `?` operator.

`?dataset`

Example:

```{r}
#| eval: false
#| include: false

print(?airquality)
```

**Helpful functions when working with datasets**

#### Loading datasets

`datasets::dataset`

OR

`datasets("dataset")`

Example:

```{r}
datasets::airquality
```

```{r}
data("airquality")
```

#### Print datasets

`print(dataset)`

Example:

```{r}
print(airquality)
```

#### Inspect head and tail of datasets

`head(dataset)`

OR

`tail(dataset)`

Example:

```{r}
head(airquality)
```

```{r}
tail(airquality)
```

#### Inspect the column and row names in the dataset

Names

`names(dataset)`

Row names

`rownames(dataset)`

Column names

`colnames(dataset)`

Example:

```{r}
names(airquality)
```

```{r}
rownames(airquality)
```

```{r}
colnames(airquality)
```

#### Dataset dimensions

Dimensions

`dim()`

Number of rows

`nrow()`

Number of columns

`ncol()`

Example:

```{r}
dim(airquality)
```

```{r}
nrow(airquality)
```

```{r}
ncol(airquality)
```

#### Summary statistics for the dataset

Summary of the dataset

`summary()`

Example:

```{r}
summary(airquality)
```

Access specific columns in a dataset

`dataset$column`

Example:

```{r}
airquality$Ozone
```

### Importing external datasets

As an R user, one of the tasks that you will often engage in is the importation of data from various sources. The commonly used data types include, but are not limited to CSV, TXT, JSON, SQL, SAS, STATA, SPSS, and XLSX. Additionally, R has two native data formats: Rdata (Rda) and Rds. Rdata is for multiple objects, while Rds is for a single R object.

When importing external data, first set the correct working directory. The function `getwd()` will show the current working directory, and `setwd(file_path_name)` will set the working directory to the file path name specified.

**NOTE**: Download the datasets from [Appendix section](https://wokech.github.io/r4novice/datasets.html) of my book and save them to your working directory.

Data importation can use base R functions or the `readr` package, which is part of the tidyverse. We will review both sets of methods in this section.

#### Base R

1.  Commonly used data importation formats

There is a wide variety of base R functions used for importing external data into R. The main format for importing data is shown below:

`read.___(path to file, header = __, sep = “_”, dec = “_”)`

-   The argument “path to file” provides the location of the file the user intends to import.

-   The argument header is set as either TRUE or FALSE, if TRUE, R assumes that the file has a header.

-   The sep argument is a group of field separation characters which include , `,`, `;`, or `\t`.

-   The dec argument allows one to set the different characters for decimal points, which include `.` or `,`.

-   The stringsAsFactors argument, by default, allows strings to be read as factors. This can be set as TRUE or FALSE

The main functions for importing data using base R are listed below:

-   `read.table()` is a general function to read in a file in table format as a data frame.

Example:

```{r}
read.table("datasets/r4novice_datafile.csv", header = TRUE, sep = ",")
```

-   `read.csv()` imports data with comma separated values. Alternatively, we use `read.csv2()` is used where a comma (“,”) indicates a decimal point and a semicolon (“;”) is a field separator.

Example:

```{r}
read.csv("datasets/r4novice_datafile.csv", header = TRUE, sep = ",")
```

-   `read.delim()` is used for files with tab-separated values, such as TXT, and where point “.” is used as a decimal point. Alternatively, `read.delim2()` is used for files with tab-separated values such as TXT and where comma “,” is used as a decimal point.

Example:

```{r}
read.delim("datasets/r4novice_datafile.txt", header = TRUE)
```

-   General data import

For beginners, one can use `read.___(file.choose())` for the interactive selection of files.

Example:

`read.table(file.choose())`

**NOTE**:Run this command in your R console and select the appropriate file from the working directory.

-   Excel (XLS and XLSX) files are read in with `read.xlsx()`

Load the required library

```{r}
library(xlsx)
```

Example:

```{r}
read.xlsx("datasets/r4novice_datafile.xls", sheetIndex = 1)
```

```{r}
read.xlsx("datasets/r4novice_datafile.xlsx", sheetIndex = 1)
```

2.  Less frequently used data importation formats

<!-- -->

a.  Fixed-width formats: `read.fwf("file path", header = TRUE)`

b.  SPSS/Stata/SAS Data Files: The foreign package reads SPSS SAV files, Stata DTA files, and SAS XPORT libraries.

Steps for use:

Install the library

`install.packages("foreign")`

Load the library

`library(foreign)`

SPSS: `read.spss(“file path”, to.data.frame = __, use.value.labels = __)`

-   to.data.frame: indicates whether R should treat the loaded data as a dataframe (options are TRUE/FALSE).

-   use.value.labels: indicates whether R should convert variables with value labels into R factors (options are TRUE/FALSE).

Stata: `read.dta("file", convert.dates = __, convert.factors = __)`

convert.dates: converts Stata dates to R's Date class

convert.factors: creates factors with Stata value labels

SAS: `read.xport("file path")`

Alternatively, one can utilize specific packages like `rio` or `haven` for more complex data formats like SPSS or SAS.

Install the library

`install.packages("haven")`

Load the library

`library(haven)`

SPSS: `read_sav("file path")`

SAS: `read_sas("file path")`

c.  Native data formats in R

R Data Files: `load("file_name.rda")`

RDS Files: `readRDS("file_name.rds")`

Online Files

To download and import an online file, we use `read.html()` and `read.xml()`

Example:

`read.html("file path")`

`read.xml("file path")`

e.  Miscellaneous file formats

JSON

Install the library

`install.packages("rjson")`

Load the library

`library(rjson)`

`fromJSON(file = "file name")`

SQL

Install the library

`install.packages("RSQLite")`

Load the library

`library(RSQLite)`

`sql_connect <- dbConnect(RSQLite::SQLite(), "file name")`

`dbListTables(sql_connect)`

Matlab

Install the library

`install.packages("Rmatlab")`

Load the library

`library(Rmatlab)`

`readMat("file name")`

#### Tidyverse and the `readr`/`readxl` packages

To enhance the speed and efficiency of data imports, the user can work with the `readr` and `readxl` packages that are part of the Tidyverse. This is because the functions in this package allows for faster data imports and work similarly despite the data type that is imported.

1.  Commonly used data importation formats in the tidyverse

```{r}
# install.packages("readr")
```

```{r}
library(readr)
```

a.  Import flat files

-   `read_table()` is used to import whitespace-separated files.

Example:

```{r}
read_table("datasets/r4novice_datafile.txt", col_names = TRUE)
```

-   `read_csv()` is for comma-separated values (CSV), while `read_csv2()` is used for semicolon-separated values with `,` as the decimal mark.

Example:

```{r}
read_csv("datasets/r4novice_datafile.csv")
```

-   `read_tsv()` is for tab-separated values (TSV)

Example:

```{r}
read_tsv("datasets/r4novice_datafile.txt")
```

-   `read_delim()` is for all delimited files, such as CSV and TSV.

Example:

```{r}
read_delim("datasets/r4novice_datafile.csv", delim = ",")
```

b.  Import spreadsheets

```{r}
# install.packages("readxl")
```

```{r}
library(readxl)
```

Use excel_sheets() to read the names of the different worksheets in the Excel workbook.

```{r}
excel_sheets("datasets/r4novice_datafile.xlsx")
```

```{r}
read_excel("datasets/r4novice_datafile.xlsx", sheet = "Sheet1")
```

2.  Less frequently used data importation formats

-   Import Google Sheets

```{r}
# install.packages("googlesheets4")
```

```{r}
library(googlesheets4)
```

Use `read_sheet("link to file")` to read in the file via the link.

-   `read_fwf()` is used to read in fixed-width files.

-   `read_log()` is the standard format for reading in web log files.

-   `readRDS()` is used to read data stored as a single R object.

-   `read_lines()` is used to read data up to a specified number of lines in a file.

**NOTE**: Missing data is a major challenge in data analysis. Strategies for dealing with this are discussed in Chapter 12 "Handling missing data" (@sec-missing).

#### Inspecting the imported data

Once the data is imported, the user can inspect the data with `str()`, `head()`, and `summary()` functions. Additionally, one can check various aspects of the data such as `names()`, `dim`, and `class` to validate that the import was successful.

Example:

Import a specific dataset

```{r}
import_inspect <- read_csv("datasets/r4novice_datafile.csv")
```

```{r}
head(import_inspect)
```

```{r}
tail(import_inspect)
```

```{r}
summary(import_inspect)
```

```{r}
str(import_inspect)
```

```{r}
names(import_inspect)
```

```{r}
dim(import_inspect)
```

```{r}
class(import_inspect)
typeof(import_inspect)
```

### Saving data and analysis outputs

Both the base R functions and the `readr` package can be used to save data. This data can be in the form of cleaned/rearranged tables, or it can be other analysis outputs such as plots.

#### Base R

Before we explore the functions used to save data, I will create a dataframe. A dataframe is one of the most common data objects used to store tabular data in R. A more in-depth look at dataframes will be provided in the chapter "Data Structures (Part I)" (@sec-data-structure-1).

```{r}
df_to_save <- data.frame(one=c(1, 3, 2, 9, 5),
                         two=c(7, 7, 3, 8, 2),
                         three=c(3, 3, 9, 7, 1),
                         four=c(5, 2, 2, 8, 9))
```

1.  Commonly used R functions

-   `write.csv()` saves data to a CSV file.

Example:

```{r}
write.csv(df_to_save, file = "saved_datasets/r4novice_saved_data.csv")
```

-   `write.table()` saves data to a specified file type.

Example:

```{r}
write.table(df_to_save, file = "saved_datasets/r4novice_saved_data.txt", sep = ",")
```

-   `fwrite()` saves data to a specified file type. It is obtained from the `data.table` package.

```{r}
library(data.table)
```

Example:

```{r}
fwrite(df_to_save, file = "saved_datasets/r4novice_saved_data_2.csv", sep = ",")
```

#### Tidyverse and the `readr`/`readxl` packages

The `readr` and `readxl` also offer complementary functions that allow the user to save data.

a.  Flat files

-   `write_csv()` and `write_csv2()` can be used to write comma-delimited and semicolon-delimited files, respectively.

Example:

```{r}
write_csv(df_to_save, file = "saved_datasets/r4novice_saved_data_3.csv")
```

-   `write_tsv()` is used to write a tab delimited file.

Example:

```{r}
write_tsv(df_to_save, file = "saved_datasets/r4novice_saved_data_2.txt")
```

-   `write_delim()` is used to write files with any delimiter.

Example:

```{r}
write_delim(df_to_save, file = "saved_datasets/r4novice_saved_data_4.csv", delim = ";")
```

b.  Excel files

```{r}
# install.packages("writexl")
```

```{r}
library(writexl)
```

Create an XLS and XLSX file

```{r}
#write_xlsx(df_to_save, "saved_datasets/r4novice_saved_data_4.xls")
write_xlsx(df_to_save, "saved_datasets/r4novice_saved_data_4.xlsx")
```

## Exercises

i.  Access the `mtcars`, `iris`, and `airquality` datasets and read the documentation using `?`. Additionally, explore the different characteristics of the data, such as names, dimensions, and summary statistics.

ii. Import datasets from different formats like [CSV](), [Excel](), and [text]() files into your R environment using the base R functions such as read.\*().

iii. Save your file using a new name with write.\*(), save(), or .RData?

iv. How can you check the structure and summary statistics of an imported dataset in R?

v.  What options do you have when importing data from non-local sources like URLs or databases? Explore functions like url() and dbConnect() for remote data access.

vi. How can you specify import options for different file formats? Learn about arguments like sep, header, and na.strings for customized data reading.

vii. What is the difference between overwriting a dataset and appending new data to it in R?

## Summary

In this lesson, the learner has been guided through the process of accessing internal datasets. Additionally, methods for importing external datasets have been demonstrated for various data formats. Lastly, the learner has been shown how to save the outputs of the data analysis. With this knowledge, the learner is now ready to perform basic arithmetic operations, which will be covered in the next chapter.