Phony data analysis

1. \(\underline{\text{Create and Open Project then Create Folders}}\)

In the terminal:

• mkdir example_project_2 \(\quad\quad\quad\text{: make directory }\)

• subl example_project_2 \(\quad\quad\quad\quad\text{: open in sublime }\)

Then in a terminal in sublime create some folders:

• mkdir code \(\quad\quad\quad\quad\quad\quad\quad\text{: make directory }\)

• mkdir raw data \(\quad\quad\quad\quad\quad\quad\text{: make directory }\)

• mkdir clean data \(\quad\quad\quad\quad\quad\quad\text{: make directory }\)

• mkdir output \(\quad\quad\quad\quad\quad\quad\quad\text{: make directory }\)

2. \(\underline{\text{Save Raw Data}}\)

In the terminal:

• R \(\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\text{: open R }\)

• data(mtcars) \(\quad\quad\quad\quad\quad\quad\quad\text{: load mtcars data set }\)

• head(mtcars) \(\quad\quad\quad\quad\quad\quad\quad\text{: show head of mtcars }\)

• getwd() \(\quad\quad\quad\quad\quad\quad\quad\quad\quad\text{: get working directory }\)

• saveRDS(mtcars, “raw_data/mtcars.rds”) \(\quad\text{: save as .rds file}\)

• saveRDS(mtcars, “raw_data/mtcars.RData”) \(\quad\text{: save as .RData file}\)

• q() \(\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\text{: quit R }\)

• y \(\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\text{: agree to quit }\)

• rm raw_data/mtcars.RData \(\quad\quad\text{: remove data from folder}\)

Note that .Rds is preferred.

3. \(\underline{\text{Create Clean Data File}}\)

[command] [N] : to create a new file.

Then save that file in the code directory as “clean_data.R” with the following code:

Note the 3 different places the here package is used.

4. \(\underline{\text{Create Makefile}}\)

[command] [N] : to create a new file.

Then save that file in the project directory as “Makefile” with the following code:

In the terminal:

• make clean_data/mtcars_cleaned.rds

• Rscript code/clean_data.R

Verify data is in clean_data folder

5. \(\underline{\text{Create make table file}}\)

[command] [N] : to create a new file.

Then save that file in the code folder as “make_table.R” with the following code:

6. \(\underline{\text{Update Makefile}}\)

Now update the Makefile with the following:

In the terminal:

• make output/data_for_table.rds

Verify contents in output folder.

7. \(\underline{\text{Create report.Rmd}}\)

[command] [N] : to create a new file.

Then save that file in the project directory as “report.Rmd” with the following code:

8. \(\underline{\text{Update Makefile}}\)

Update the Makefile with the following:

In the terminal:

• make report.html

• open report.html

Make Github aware of empty folder

In terminal

• rm output/* \(\quad\quad\quad\quad\text{: remove everything in output folder}\)

• gitinit \(\quad\quad\quad\quad\quad\quad\text{: initilize git}\)

• git status \(\quad\quad\quad\quad\quad\text{: status}\)

Notice the output folder isn’t included.

• cd output \(\quad\quad\quad\quad\quad\text{: change to output directory}\)

• touch .gitkeep \(\quad\quad\quad\text{: add empty file}\)

• ls -a \(\quad\quad\quad\quad\quad\quad\text{: list all files}\)

• cd .. \(\quad\quad\quad\quad\quad\quad\text{: change directory up one level}\)

• gitstatus \(\quad\quad\quad\quad\quad\text{: status}\)

Now the output folder is visible.

Adding a text editor to a docker file

In the terminal:

• mkdir docker_ex \(\quad\quad \text{: created docker_ex directory}\)

• subl docker_ex \(\quad\quad\text{: open in sublime}\)

[command] [s] : save file as “Dockerfile”

• FROM : gives a starting point at the ubuntu operating system.

• apt-get: is a package management system in ubuntu

• update: make sure apt-get is up to date

• install vim: installs the text editor vim

• Run: Docker syntax that tells it to run those two commands from the ubuntu operating system.

Now back in the terminal:

• cd docker_example/

• ls

• docker build -t ubuntu_plus_vim . \(\quad\quad\text{build docker file from current directory (thats what "." means)}\)

Docker is angery because we aren’t inside the container to hit yes.

Fix: add “-y” to code (line 7) that says yes:

Now that it works lets run it.

• docker run -it ubuntu_plus_vim /bin/bash \(\quad\quad\text{: open up ubunto with vim that we just created}\)

• vim \(\quad\quad\text{opens text editor for editing}\)

• :q \(\quad\quad\quad\text{to quit vim}\)

If you need to put a special piece of software inside a container you would google “how to install ____ on ubuntu”, because that is being installed.

COPY Example

create a new hello world file.

Now add the following line to your dockerfile:

COPY hello_world.txt home/hello_wolrd.txt

In the sublime terminal:

• docker build -t ubuntu_plus_vim_plus_helloworld .

• docker run -it ubuntu_plus_vim_plus_helloworld

• ls

• cd home

• ls

• cat hello_world.txt

• exit

This will be useful so that we can copy our code into the container.

WORKDIR

Sets the working directory for all of your commands.

Reason: We don’t always know what the directories are within docker (like home in the previous example), so we want docker to the directory where our project lives.

Update Dockerfile with the following:

In the terminal:

• docker build -t try_workdir .

• docker run -it try_workdir /bin/bash

Notice that we start out in the analysis_directory, and inside the directory is hello_world.txt.

• exit

CMD

Continers entry point: what is the container going to do by defualt?

Use cases:

• interactively run bash: CMD /bin/bash

• open R: CMD [“R”]

• run whole analysis and produce report: CMD Rscript -e “rmarkdown::render(…)”

Breakout Exercise 1:

Inside docker_ex directory:

In the terminal:

• echo “This file is silly” > silly_file.txt

• subl

Create and save the following Dockerfile within the same directory as newimgage.Dockerfile:

Then in the terminal:

• docker build -t br1 –file newimage.Dockerfile .

• docker run br1

(should print message: Hello from your computer)

• docker run -it br1 /bin/bash

• ls

• cat silly_file.txt

• echo $my_variable

• exit

Viewing / removing containers / images

docker container ps : list running containers, add -a to see all recently run containters.

docker images : list all downloaded images

docker image rm [id] : replace [id] with code from Imgae ID column of docker images

docker system prune : remove all unused images and containers. Add -a to remove everything

You can always download your containers through DockerHub, and dont need them on your computer all the time.

Typical Workflow:

• updating analysis code, running, …

• tweak figures, compile report

Docker Workflows (slightly more complicated)

• different R version/packages between container and local?

• update code locally, rebuild image, run code in container, …

• tweak figures locally, rebuild image, compile report in container,…

Output folder

Create one output folder that has anything you want the user to retrieve. Then instruct the user that if they want to retrieve the output from this container that they need to mount a local directory to this output folder.

Enviroment variables

A useful way to control the run-time behavior of docker containers is to set environment variables at runtime.

Recall we set our environment variable to “gears_vs_cylindars”. However if we check the code for make_barchart.R we notice an ifelse state with if(which_fig = “gears_vs_cylinders”) … else if(which_fig = “gears_vs_carb”).

In the terminal, in ex_proj docker

• R \(\quad\text{: open R}\)

• Sys.getenv(“which_figs”)

• q() \(\quad\text{: quit R}\)

• export some_other_variable=“hello!”

• R \(\quad\text{: open R}\)

• Sys.getenv(“some_other_variable”)

• q() \(\quad\text{: quit R}\)

• exit \(\quad\text{: close docker}\)

We can overide this variable at runtime using -e

In the terminal:

• docker run -e which_fig=“something_else” -e some_other_variable=“some_other_value” -it ex_proj

• echo $which_fig

• echo $some_other_variable

• R

• Sys.getenv(“some other variable”)

• q()

• exit