Forem: Peter Solymos

Manage Dependencies with the deps R Package for Docker Containers

Peter Solymos — Fri, 11 Nov 2022 13:00:00 +0000

The deps package gives you a lightweight option to manage package dependencies and you can install these inside containers.

When building Docker images for your R-based applications, the biggest hurdle is knowing exactly which packages and system libraries your package depends on. Luckily, the tools have evolved quite a bit over the past few years. In this post, I show you where the deps package fits in and how this can be a great choice for dependency management for Docker-based workflows.

Reproducibility

Tools like packrat, renv, and capsule let you go to great lengths to make your R projects perfectly reproducible. This requires knowing the exact package versions and the source where it was installed from (CRAN, remotes, local files). This information is registered in a lock file, which serves as the manifest for recreating the exact replica of the environment.

Full reproducibility is often required for reports, markdown-based documents, and scripts. A loosely defined project that is combined with strict versioning requirements, often erring on the side of “more dependencies are safer”.

In our previous post we covered how to manage dependencies with the renv package:

Package-based development

On the other end of the spectrum, we have package-based development. This is the main use case for dependency management-oriented packages, such as remotes and pak.

In this case, exact versions are managed only to the extent of avoiding breaking changes (given that testing can surface these). So what we have is a package-based workflow combined with a “no breaking changes” philosophy to version requirements. This approach often leads to leaner installation.

The middle ground

What if we are not writing an R package and wanted to combine the best of both approaches? – A loosely defined project with just strict-enough versioning requirements. All this without having to write a DESCRIPTION file by hand. Because why would you need a DESCRIPTION file when you have no package? Also, a DESCRIPTION file won’t let you pin an exact package version or specify alternative CRAN-like repositories.

What if you could manage dependencies by decorating your existing R code with special, roxygen-style comments? Just like this:

#' @remote analythium/rconfig@CRAN-v0.1.3
rconfig::config()

#' @repo sf https://r-spatial.r-universe.dev
library(sf)

#' @ver rgl 0.108.3
library(rgl)

This is exactly what deps does:

helps to find all dependencies from our files,
writes these into a dependencies.json file,
performs package installs according to the decorators.

The decorators make our intent explicit, just like if we were writing an R package. But we do not need to manually write these into a file and keep it up-to-date. We can just rerun create to update the JSON manifest file.

Tag	Description	Usage
`@sys`	System requirement(s)	`@sys req1,req2,...`
`@remote`	Remote source(s)	`@remote remote1,remote2,...`
`@local`	Local source(s)	`@local path1,path2,...`
`@ver`	Versioned package	`@ver pkg version`
`@dev`	Development package(s)	`@dev pkg1,pkg2,...`
`@repo`	CRAN-like source	`@repo pkg repo`
`@repos`	Global CRAN-like repo(s)	`@repos repo1,repo2,...`
`@rver`	R version	`@rver 4.1.3`

Using the deps package

The deps package has 2 main functions:

create() crawls the project directory for package dependencies. It will amend the dependency list and package sources based on the comments and query system requirements for the packages where those requirements are known for a particular platform; the summary is written into the dependencies.json file.
install() looks for the dependencies.json file in the root of the project directory (or runs create() when the JSON file is not found) and performs dependency installation according to the instructions in the JSON file.

In the simplest case, one might have a project folder with some R code inside. Running deps::install() will perform the package installation in one go. Additional arguments can be passed to install() so that local libraries etc. can be specified.

These arguments are passed to install.packages(). This is a really important consideration when it comes to utilizing RSPM or BSPM repositories on Linux systems. RSPM (RStudio Package Manager) provides rebuild binaries, BSPM (Bridge to System Package Manager) provides full system dependency resolution and integration with apt on top of binary packages.

Docker workflow

The following example is part of the deps package examples. We will use a Shiny app that we have used before to draw a 3D surface for a bivariate Normal distribution.

3D surface of a bivariate Normal distribution.

Let's say that we have a single file app/app.R with the following content:

library(shiny)
library(MASS)
options(rgl.useNULL = TRUE)
library(rgl)

ui <- fluidPage(
  titlePanel("Correlated variables"),
  sidebarLayout(
    sidebarPanel(
      sliderInput("n", "Sample size",
        min=2, max = 10^3, value = 200
      ),
      sliderInput("r", "Correlation",
        min = -1, max = 1, value = 0, step = 0.05
      )
    ),
    mainPanel(
      rglwidgetOutput("plot",
        width = "500px", height = "500px")
    )
  )
)

server <- function(input, output) {
  Sigma <- reactive({
    matrix(c(1, input$r, input$r, 1), 2, 2)
  })
  m <- reactive({
    mvrnorm(input$n, c(0, 0), Sigma())
  })
  output$plot <- renderRglwidget({
    d <- m()
    k <- kde2d(d[,1], d[,2])
    try(close3d())
    persp3d(k$x, k$y, k$z,
      ann = FALSE, axes = FALSE,
      xlab = "", ylab = "", zlab = "",
      aspect = c(1, 1, 0.5), col = "lightblue")
    rglwidget()
  })
}

shinyApp(ui, server)

We can start with the following Dockerfile:

FROM eddelbuettel/r2u:22.04

RUN installGithub.r analythium/deps
RUN apt-get update && apt-get install -y --no-install-recommends jq

RUN addgroup --system app && adduser --system --ingroup app app
WORKDIR /home/app
COPY app .

RUN R -q -e "deps::create()"
RUN apt-get install -y --no-install-recommends \
    $( jq -r '.sysreqs | join(" ")' dependencies.json )
RUN R -q -e "deps::install()"

RUN chown app:app -R /home/app
USER app

EXPOSE 8080
CMD ["R", "-e", "shiny::runApp(port = 8080, host = '0.0.0.0')"]

In this case, we just copy over the contents of the app folder, create the dependencies.json file inside the Docker image. We use the jq command line utility to parse the JSON file and pull out the system requirements that we want to install. Then install the R packages.

We are using the eddelbuettel/r2u:22.04 parent image, so we do not have to worry too much about the installation times. But copying the files and then performing the installation is not the best approach. This way we cannot use the Docker image layer catching very effectively. When the files change, the cache gets invalidated and we'll have to wait for the install step to complete.

We can do much better if we keep an up-to-date dependencies.json file as part of the project that we want to containerize. The JSON file has the following structure:

{
  "version": "1.0",
  "rver": "4.2.1",
  "repos": [],
  "sysreqs": ["libgl1-mesa-dev", "libglu1-mesa-dev", "libpng-dev", "pandoc pandoc-citeproc", "zlib1g-dev"],
  "packages": [
    {
      "package": "MASS",
      "installed": true,
      "dev": false
    },
    {
      "package": "rgl",
      "installed": false,
      "dev": false,
      "source": "cran"
    },
    {
      "package": "shiny",
      "installed": false,
      "dev": false,
      "source": "cran"
    }
  ]
}

dependencies.json

This JSON file can be copied over on its own, ensuring that installation happens only when the dependencies are updated:

FROM eddelbuettel/r2u:22.04

RUN installGithub.r analythium/deps
COPY dependencies.json .
RUN apt-get update && apt-get install -y --no-install-recommends jq
RUN apt-get install -y --no-install-recommends \
    $( jq -r '.sysreqs | join(" ")' dependencies.json )
RUN R -q -e "deps::install()"

RUN addgroup --system app && adduser --system --ingroup app app
WORKDIR /home/app
COPY app .
RUN chown app:app -R /home/app
USER app

EXPOSE 8080
CMD ["R", "-e", "shiny::runApp(port = 8080, host = '0.0.0.0')"]

Here are the steps from this Dockerfile:

pull the r2u parent image
install the deps package using the littler command line utility
copy the dependencies.json file
install the jq tool for manipulating the JSON file
install system requirements from the "sysreqs" property of dependencies.json
install R packages listed in dependencies.json
create a user and group called app to increase the security of the Docker image
set the home folder of the app user as the working directory
copy the contents of the app folder into the home folder
set permission and change the user from root to app
expose port 8080
set the command to start the Shiny app on port 8080

Let's build and test the Docker image to see deps in action:

# change directory to the example
cd inst/examples/99-docker

# change this as needed if you want to `docker push`
export TAG=analythium/deps-shiny-example:v1

# build the image
docker build -t $TAG .

# start the app
docker run -p 8080:8080 $TAG

Visit http://localhost:8080 and make sure the app is functioning as expected.

Conclusions

The deps package helps users be more intentional about the R package source and version requirements using text decorators in comments. This is similar to a package-based workflow without actually writing a package. But deps also lends itself to Dockerized development. It identifies system requirements for the R packages, which is a welcome addition to making the Docker experience for R as user-friendly and hands-off as possible.

Containerizing Shiny for Python and Shinylive Applications

Peter Solymos — Thu, 10 Nov 2022 13:00:00 +0000

By: Peter Solymos

Shiny for Python brings easy interactivity to web applications that build on the data and scientific stack of Python. Containerizing Py-Shiny apps is the next step towards deployment to a wide array of hosting options, including static hosting.

Shiny is a framework that makes it easy to build interactive web applications. Shiny was introduced 10 years ago as an R package. In his 10th anniversary keynote speech, Joe Cheng announced Shiny for Python at the 2022 RStudio Conference. Python programmers can now try out Shiny to create interactive data-driven web applications. Shiny comes as an alternative to other frameworks, like Dash, or Streamlit.

⚠️

Shiny for Python is currently in Alpha. It may be unstable, and the API may change. We’re excited to hear your feedback, but please don’t use it for production applications just yet!

Similarly to R Shiny applications, Shiny for Python can be deployed using RStudio Connect, Shiny Server Open Source, and Shinyapps.io. Alternative hosting options – that the Hosting Data Apps website is dedicated to – require the Python Shiny app to run inside a container. In this post, we review how to use Docker to containerize a Shiny for Python app.

Shiny app template

We follow the Get started guide (see also the install guide). You can install Shiny with pip or conda. Here we will use pip. The following commands generate the app file that we will use:

pip install shiny
shiny create app
shiny run --reload app/app.py

# INFO:     Will watch for changes in these directories: ['/Users/Username/app']
# INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
# INFO:     Started reloader process [57404] using StatReload
# INFO:     Started server process [57406]
# INFO:     Waiting for application startup.
# INFO:     Application startup complete.

Go to http://127.0.0.1:8000 in your browser to try the app displaying a slider and a text output returning double of the slider input value (N=n of course):

The simplest Shiny for Python example.

Use Ctrl+C to quit the app.

Example app with plot

We will use the app with plot example. Open the app/app.py file in a text editor and copy the following contents into it:

from shiny import App, render, ui
import numpy as np
import matplotlib.pyplot as plt

app_ui = ui.page_fluid(
    ui.h2("Histogram with Shiny for Python!"),
    ui.layout_sidebar(
        ui.panel_sidebar(
            ui.input_slider("n", "N", 0, 100, 20),
        ),
        ui.panel_main(
            ui.output_plot("plot"),
        ),
    ),
)


def server(input, output, session):
    @output
    @render.plot(alt="A histogram")
    def plot():
        np.random.seed(19680801)
        x = 100 + 15 * np.random.randn(437)
        plt.hist(x, input.n(), density=True)


app = App(app_ui, server, debug=True)

Create the file app/requirements.txt in the same directory as the app.py with the following contents:

shiny>=0.2.7
numpy>=1.23.3
matplotlib>=3.6.0

Now use pip install --no-cache-dir --upgrade -r app/requirements.txt to install the remaining packages. Then load the app again with shiny run --reload app/app.py and visit http://127.0.0.1:8000 in your browser again.

You'll see the new app with a plot that looks very similar to the classical R hello Shiny app:

Shiny for Python app with the plot.

The Dockerfile

If you look at the printout after launching the app, you'll notice that Shiny is using Uvicorn under the hood. This is a common way of containerizing apps using FastAPI deployments.

Let's see what goes into the Dockerfile:

use the official python:3.9 parent image
create the /home/app folder and set an app user with appropriate non-root permissions
install requirements before copying the app – this is to best utilize caching when still iterating on the app
copy the rest of the app folder, i.e. the app itself
expose the 8080 port and define the uvicorn command

FROM python:3.9

# Add user an change working directory and user
RUN addgroup --system app && adduser --system --ingroup app app
WORKDIR /home/app
RUN chown app:app -R /home/app
USER app

# Install requirements
COPY basic/requirements.txt .
RUN pip install --no-cache-dir --upgrade -r requirements.txt

# Copy the app
COPY basic .

# Run app on port 8080
EXPOSE 8080
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]

Next, we can build the image. Define your own image name and tag:

export IMAGE=analythium/python-shiny:0.1

docker build -t $IMAGE .

Test the app running inside the container with docker run:

docker run -p 8080:8080 $IMAGE .

Go to http://127.0.0.1:8080 in your browser and check.

Push to the docker registry with docker push $IMAGE.

Shinylive

Shinylive is an experimental feature (Shiny + WebAssembly) that allows applications to run entirely in a web browser, without the need for a separate server running Python.

⚠️

Shinylive is more experimental than Shiny for Python itself. Not all Python packages are supported by the underlying Pyodide project, so not every app would compile easily.

The point of Shinylive is not really to be served via Docker, but rather as static assets. Still, there might be cases when containerizing Shinylive seems like a good idea. When for example all the rest of the stack is using Docker and we don't want a file server besides that.

If this did not deter you, let's create the simplest Shiny app again inside the live folder:

shiny create live

Add a live/requirements.txt file with the following contents

Shinylive will be installed on its own, no need to include it, just use your requirements from a non-live app:

# live/requirements.txt
shiny

The Dockerfile follows the pattern borrowed from the static R Markdown deployment using a multi-stage Docker build:

install requirements + Shinylive
copy the app
build the Shinylive assets in the site folder
copy the site folder into a minimal image alongside the OpenFaaS watchdog and serve

FROM python:3.9 AS builder
WORKDIR /root
COPY live/requirements.txt .
RUN pip install shinylive
RUN pip install --no-cache-dir --upgrade -r requirements.txt
COPY live app
RUN shinylive export app site

FROM ghcr.io/openfaas/of-watchdog:0.9.6 AS watchdog

FROM alpine:latest
RUN mkdir /app
COPY --from=builder /root/site /app
COPY --from=watchdog /fwatchdog .
ENV mode="static"
ENV static_path="/app"
HEALTHCHECK --interval=3s CMD [ -e /tmp/.lock ] || exit 1
CMD ["./fwatchdog"]

You should be able to build, test, and push the Docker image:

export IMAGE=analythium/python-shiny-live:0.1

# Build
docker build -t $IMAGE .

# Test, visit http://127.0.0.1:8080
docker run -p 8080:8080 $IMAGE

# Push
docker push $IMAGE

The app at http://127.0.0.1:8080 in your browser should look like the one we began with: a slider and a text output showing the double of the slider input value.

Conclusion

We covered how to containerize Shiny for Python applications with dynamic or static Shinylive versions. This newly gained Docker power opens the door for deploying the app to various platforms via the Docker image. These options include Heroku, the DigitalOcean App Platform, Fly.io, Docker Compose, or ShinyProxy. And for the experimental Shinylive apps, just host it anywhere (GitHub pages, Netlify, etc.) as static files.

Deploying a single instance of a Shiny app, however, is not the same as deploying multiple instances. Load balancing between these instances of the same app could prove difficult. We'll revisit the pitfalls of scaling Shiny apps in a subsequent post. Get notified about new posts by signing up for the newsletter.

We Need to Talk About Docker Registries

Peter Solymos — Wed, 09 Nov 2022 07:22:11 +0000

By: Peter Solymos

A Docker registry stores Docker images, this is where we push images to and pull images from. Why do we need a registry? What registries are out there? How can we work with them? Read on to find out.

Many Docker tutorials begin with docker pull as the first command to cover. The docker pull command is followed by the "tag" of a Docker image that we want to be pulled. Say that we wanted the latest Alpine Linux image (I picked this due to its minimal size):

docker pull alpine:latest

# latest: Pulling from library/alpine
# 530afca65e2e: Pull complete
# Digest: sha256:7580ece7963bfa...
# Status: Downloaded newer image for alpine:latest
# docker.io/library/alpine:latest

But where does this image come from?

What is a Docker registry?

Where does the Docker command line interface (CLI for short) pulls the images from? The answer is on the last line of the output. docker.io is the URL for a so called Docker registry. In this case, it refers to the default registry, Docker Hub, applied when you don't specify a registry URL as part of the pull command.

Docker Hub is a service provided by Docker for finding and sharing container images.

The registry URL is usually followed by the user or organization ID, the repository name, the image name and the tag: <registry>/<user>/<repository>/<image>:<tag>.

In short, a Docker registry stores Docker images. This is where we push images to and pull images from:

Docker architecture / © Analythium

Read more about how the different parts of the Docker system relate to each other in this introductory post:

Why do you need a registry?

You might think that you only need a registry because you need to pull your parent images (like Ubuntu, Alpine, r-base from the Rocker project etc.) from somewhere. Then you can build your Docker images locally using a Dockerfile, and run them. Why bother with a registry at all?

You can even use the docker save and docker load commands to save to a compressed tar file:

docker pull alpine:latest

docker save -o alpine-latest.tar alpine:latest

ls -sh alpine-latest.tar
# 11368 alpine-latest.tar

Then take this tar file, copy it to another server and load it:

docker load --input alpine-latest.tar

now imagine that you are managing more than two servers, or you want to share the Docker image with others (to use it or to serve as a parent image). The save/copy/load workflow becomes cumbersome.

In this case, using a registry might be a much better idea. Luckily, there are many options to choose from, and you can even host your own registry.

Roll your own registry

If you want a registry hosted on your machine, just pull the registry image. The next command will pull the registry image, and run the similarly named container in the background on port 5000:

docker run -d \
  -p 5000:5000 \
  --restart=always \
  --name registry registry:2

Tag the Alpine image with the URL of your local registry, localhost:5000, and push the image:

docker tag alpine:latest localhost:5000/my-alpine

docker push localhost:5000/my-alpine

Remove the images so the docker images will not list these anymore:

docker image remove alpine:latest
docker image remove localhost:5000/my-alpine

Pull the image from the local registry and list it with docker images to verify that it is there again:

docker pull localhost:5000/my-alpine

Stop and remove the registry container. The -v option makes sure to remove anonymous volumes associated with the container which is often used to mount a volume from your hard drive into the container where the images are stored:

docker container stop registry && \
  docker container rm -v registry

If you want your own registry to be accessed over a network, then you need to think about security and access control.

Setting up transport layer security (TLS) for HTTPS and user authentication are all advanced topics, so implement those at your own risk.

For now, we will continue with a hosted registry, as most people do.

Log into a registry

When you work with private registries or private images, you need to log in with the docker login command. For Docker Hub, just type docker login. For all other registries, type in the registry URL as well, e.g. docker login ghcr.io.

The Docker CLI then will prompt you for your username and password (or access token).

You can log in programmatically by providing your username and the password through standard input:

export CR_PAT=YOUR_TOKEN
echo $CR_PAT | docker login ghcr.io -u USER --password-stdin

As an alternative, the password can be provided via a file, this approach won't leave trace of your token in your history:

cat ~/my_password.txt | docker login -u USER --password-stdin

Use one of these approaches to log into any public or private repository for which you have credentials. The credentials will be stored locally: in $HOME/.docker/config.json on Linux or %USERPROFILE%/.docker/config.json on Windows. After login, there is no need to re-authenticate until you log out with docker logout.

Note that docker login requires users to use sudo or be root in most cases.

It is always a good idea to use a token instead of your password. Tokens can have limited scope (i.e. only for pulling images), and can be revoked at any time without it impacting other areas of your life.

Commonly used registries

There are many registries out there besides Docker Hub. Here is a non-exhaustive list of options.

GitHub container registry (the ghcr.io URL from above) is available as part of GitHub Packages for free and paid plans, even for private repositories under the free plan. This registry requires authentication using your GitHub token.

An alternative to GitHub is GitLab (registry.gitlab.com), which has provided registry support for its free (public and private) repositories long before GitHub. The docs are also much better in my opinion, and the whole experience is tightly integrated with GitLab's CICD pipelines. This registry also needs login with a token.

Heroku comes with a Docker registry (registry.heroku.com) where the Docker-based deploys push the images to.

Of course, every major cloud provider offers a Docker container registry that is often integrated with their other offerings. Latency should be minimal due to network proximity to the servers:

Other common alternatives for container registries, Helm charts, etc., include:

Although these services are called "container registry", but strictly speaking they store container images.

Summary

We reviewed what container registries are, how to access them, and what are the commonly used services out there that you can leverage for your workflow.

You have now a good understanding of where the images go after push and where the images come from after pull.

CICD Patterns with GitHub Actions and Docker

Peter Solymos — Sat, 04 Dec 2021 04:28:48 +0000

By: Peter Solymos

Start with easy integrations and add more serious testing to your CICD pipeline as your application matures.

CICD is a shorthand to refer to continuous integration and continuous delivery/deployment. CICD includes automated building, testing and deployment of applications. It connects development and operation activities and teams responsible for these stages in application life cycles.

CICD often builds on version control (i.e. GitHub, GitLab, Bitbucket), and hosted CICD services (GitHub actions, GitLab pipelines, Bitbucket pipelines, and many others like Travis, CircleCI, etc.). The workflow looks like this:

Commit your changes to the git remote
Changes on the remote trigger the CICD pipeline
If the steps in the pipeline are successful a new delivery or deployment is initiated

The difference between continuous delivery and continuous deployment is that delivery usually means that all the artifacts are delivered that are required for the deployment, but deployment happens in a separate step. Continuous deployment means that a new version is deployed without manual intervention. Think like your Shiny app is included in a Docker image (delivered) vs. the new image is deployed as a running container.

However, when it comes to Docker-based delivery and deployments, there are multiple ways how CICD can be set up. In this post, I review some of the common patterns I have seen in Docker-related CICD workflows.

Why should you care?

Why is this whole CICD thing important? On one side, it just makes your life easier by automating some of the steps that would otherwise require you to run tests, copy files, push/pull images, etc. Making sure that no human error occurred along the way. By removing the human element and relying on great tooling, your chances of making silly mistakes are hugely eliminated.

On the other side, platform-as-a-service (PaaS) offerings often include some form of integration with major hosted git provides, such as GitHub and GitLab (this latter can be self-hosted too). Therefore, it is a good idea to take advantage of these integrations to reap the benefits of automation.

Four CICD patterns to work with Docker and PaaS

Non-git-based delivery and deployment

By non-git based I mean that the source code management and the delivery/deployment ideas of your activities are independent. You use git to manage your codebase (versioning). From the same code base, you make sure that the right artifacts(files, images) are built, tested, delivered, and deployed.

Local image build

In this most basic scenario, you version your codebase. You build Docker images locally. You push the images to a Docker registry. When you deploy the image, you trigger the pull yourself (via command line or user interface) which then initiates a new deployment.

Local build with independent source code management

This is not a very common pattern in production. It is most likely that you start with something like this when you are just experimenting.

An example would be how to deploy Docker container images to virtual machines or Kubernetes clusters using OpenFaaS. In this case, the faas-cli is used to build images locally, push these into a registry, then instruct the servers to pull the updated images and deploy them.

PaaS-based image build

In this scenario, the version control is still independent of the deployment process in the sense that GitHub won't instruct your PaaS provider about what to do. It is your responsibility. Once you push a new version of your app, PaaS takes care of the build and deployment processes.

Build and deployment taken care of by the PaaS

An example of this scenario is deploying apps to Heroku from the command line interface (CLI). The twist here is Heroku CLI sets up a new remote on Heroku's servers. So in a way, git instructs the PaaS about the deployment. But you also keep your other remote (GitHub, etc.) to version your code. When you push a new git commit (the event), the Heroku remote will kick off the Docker build process and the deployment of the new version for you.

Enter GitOps

GitOps is an operational framework that takes DevOps best practices used for application development such as version control, collaboration, compliance, and CI/CD, and applies them to infrastructure automation. – GitLab Docs

This means that the remote git platform is responsible for all downstream events. Secure communication between different service provides (GitHub and your PaaS) is set up using access tokens (revocable "secrets").

The advantage of this setup is that a lot more steps can be automated within the same workflow, like testing etc. This will make the process more robust.

Git-triggered and PaaS-based image build

This setup is very similar to the previous, PaaS-based image build. The difference is that the build process is triggered by the code hosting platform, GitHub for example. Pushing the new commit, often to a certain pre-configured branch, will result in a webhook alerting the PaaS provider. The PaaS servers then launch the deployment process.

Git commit triggers the build process in the PaaS

GitHub will only deliver the news to the PaaS, but no GitHub actions are involved here. Examples for this scenario include the GitHub integration with the DigitalOcean App Platform. Setup is usually super simple, but the ease of setup is at the expense of full control over how exactly the process is taking place. If you need more control then see the next option.

Git-triggered and GitHub-based image build

The last scenario circles back to the 1st option, but instead of everything happening locally, the whole build/deployment process is happening on remote servers. The git push event kicks off the GitHub action (or similar) CICD process. The Docker image build happens inside the GitHub actions runners.

Build and deploy with GitHub actions

This setup and the required GitHub actions steps give you the most control of how you want things built and delivered. For example, you can specify if you want to cache the Docker image layers. This is also ideal to make sure new deployment only happens if all steps are successful. E.g. testing your code might find errors in unit tests. You don't want to release a new version until the issues are dealt with.

Here are a few examples:

Heroku deployment with GitGub actions
DigitalOcean app platform with image caching example
CICD with the Fly.io platform with a Shiny example workflow
Functions and microservices with OpenFaaS and GitHub actions

When there is a command-line tool to interact with the PaaS, you can most likely set up a GitHub actions workflow with that.

Conclusions

CICD practices are important for a robust and automated developer experience. Knowing that the workflows are in order gives you peace of mind. Deploying containerized applications adds a few extra steps to your workflows. As a result, there are multiple ways of approaching it.

Some workflows are simpler, others are more complex. You can start simple and add complexity as your needs evolve. Try easy integrations when prototyping, add more serious testing and GitHub actions as your application matures. Once you get the hang of it, CICD becomes second nature. Tooling is getting better every day.

How to Pick the Right Hosting Option for Your Shiny App

Peter Solymos — Mon, 15 Nov 2021 06:29:39 +0000

By: Peter Solymos

You mastered Shiny, your app is production-ready. Here are the different ways of hosting it. This post helps you pick one that best suits your needs!

The Hosting Data Apps website recently celebrated its 6-months anniversary. During this time I have written 40 posts, almost all about Shiny hosting options. Some of these posts reviewed particular hosting options, such as Shinyapps.io, Shiny Server, Heroku, and ShinyProxy.

A lot has been said about the hosting options themselves, but what about the needs of the developers and the users? The goal of this post is to address this shortcoming by summarizing the reviews.

How not to do a review

You might have seen various websites listing comparisons titled "A vs. B". I was wondering if doing something like that would make sense for the Hosting Data site. Think like "Which is better: Shinyapps or Heroku?".

This would instantly give me 36 more articles to write. Plenty of ammunition for the next 9 months by recycling the previous content. Easy-peasy, if all I wanted was to boost the page ranking.

Instead, I decided that these A vs. B posts are useless. They fail to ask the most important question: "which option is better **for what"? This is where developers and the users of the app come into the picture with their **specific needs and constraints.

You could say, for example, that "I want to host my portfolio for free and I don't care about a custom domain name". Or a nonprofit might say, "we want to host our apps at low cost, we want custom domains, and we want to be able to handle surge traffic, but we don't want to maintain any servers".

These are really specific criteria, and if you ask me, I might say Shinyapps.io is best for you and Heroku with a Docker-based deployment is best for this organization. But how would you or I make such a decision?

The 3 step process to make the decision

If you have been developing Shiny apps, you might already have your preferred way of deployment. But as your needs evolve, you will identify additional requirements and might find that your go-to option is not the best anymore. Then you'll do some research and find the next option.

If you are not yet familiar with Shiny hosting options, you still need to make an informed choice at some point, so that you are not wasting your time and effort on something that will not serve you well over the long run. Here is the 3 step process that you can follow to help you with this decision.

Before you begin take a piece of paper.

1. Start with the why

Why do you want the app or apps deployed? Are you building a portfolio to boost your career? Are you deploying useful apps for stakeholders or clients of your organization? Are you trying to sell an app as a software-as-a-service (SaaS) offering?

Write down your answer.

Getting clear on the why is the most important question. You might even realize that you don't need to host your Shiny app. For example, your app might be used on a laptop as a GUI to analyze data by non-specialists in the field without internet or cell coverage. In this case, no need to move on to step 2, because all you need is to run Shiny locally.

However, if your answer makes it clear to you that your users will be accessing the app over the Internet, move on to step 2.

2. List your requirements

Answering the Why question will probably reveal important details about your motivations, your audience, the number of apps you are going to host, etc. The answer will also bring you closer to identifying the requirements that you'll need.

For example, do you need a custom domain, how many users are you expecting, do you need authentication or app-level authorization? Do you want to host a single app or do you need to host many apps? Will you host non-Shiny apps?

Write these down too.

The following table lists the important features for many different Shiny hosting options. The table lists tiers offered by the same company as a separate option. Use this table to find the options that meet your requirements. For now, just ignore the columns inside the blue rectangle.

If you crave a more interactive experience, I made a filterable version:

Here is the link to the page where you can filter the options.

Once you filter the table according to your requirements, you'll see a list of your ideal hosting options. Put these in the file or onto the paper too.

3. Identify your constraints

The last step involves identifying your constraints:

What is your budget?
What is your current skill level?
How much time do you have time?

Recognizing these constraints will guide you toward an optimal solution. This is the point where the columns inside the blue rectangle come in.

The "Total Cost" of ownership (USD/year) covers licensing fees and operating costs for the "Number of Apps" listed in the table. Prices range quite a bit from free to the tens of thousands. Price increases with performance and with the availability of enterprise features, such as custom domains and authentication.

PaaS means platform-as-a-service, i.e. it is a fully managed system without you having to worry about the underlying infrastructure. This also means less control over the infrastructure, i.e. when it comes to choosing the data region where your app is served from.

Unlimited app hours are more common for self-hosted options or paid PaaS plans including a single app. The need to host multiple apps will involve some compromises. The ability to host non-Shiny apps (Dahs, Streamlit, etc.) is a feature for RStudio Connect and the Docker-based options.

Time as a constraint will depend on how far your current skill level is from the level needed for a specific hosting option. You also have to consider that some options are fully managed PaaS offerings, others you have to manage, or learn how to use Docker.

If you have to develop new skills, it might take longer. If you have to manage your servers, it will take more time to get started and then you are on the hook for maintaining your setup.

Make your selections inside the columns within the blue area.

You are done!

After the 3-step process, you should see only a few or a single option left. Click on the name of the hosting option and the link will take you to the relevant tag page on the Hosting Data Apps website:

Follow the instructions in the tutorials to get started
At the end of each post, you'll find a Further reading section listing additional resources

If there is no option left in the table, then you might need to be more realistic about your expectations or relax some of your constraints. For example, to keep costs low, you could spend more time and invest in skill development. But if you have more room in your budget, you might choose a different path. You can also revise your requirements until you find an acceptable solution.

Options at a glance

The following diagram gives an intuitive overview of the different options. The vertical axis represents the total cost from the table above: free, low, and high cost. The horizontal axis shows a range of skills you need to set up and manage your hosting solution. It can be as simple as pushing a button, or as complex as managing servers or cloud clusters.

The hosting options in this diagram are not separated by tiers but rather shown as spanning over a range. The fill colours identify Docker and non-Docker-based options, the stroke styling indicates the PaaS solutions.

What is next

If you followed the 3-step process to collect all the information you need, it is likely that you have found an option that is best for your needs. Now go ahead and learn more about that option, deploy, and start hosting your app.

Shiny is a very popular interactive data application framework. As a result, new hosting options are popping up every time. As the number of these hosting options grows in the future, I might update this post by adding the new contenders to the table.

Auto deploy Shiny app changes to the DigitalOcean App Platform

Peter Solymos — Sun, 24 Oct 2021 18:06:13 +0000

By: Peter Solymos

Integrate the DigitalOcean App Platform with GitHub for automated deployment of your Shiny app.

I introduced the DigitalOcean App Platform in a previous post and described how to deploy Shiny apps from existing Docker images using the control panel or the command line.

This post continues the App Platform discussion by looking at how to integrate the App Platform with GitHub and set up continuous integration and deployment (CICD).

Prerequisites

If you have not done so before, go to the App Platform landing page and sign up for a DigitalOcean account. Log into your account and go to your dashboard. In the left drawer, click on 'Apps'.

For the analythium/app-platform-shiny repository because you will have to give permissions to access the repo.

Here is what's inside the repo:

.
├── .do
│   ├── app.yaml
│   └── deploy.template.yaml
├── app
│   ├── global.R
│   ├── server.R
│   └── ui.R
├── .gitignore
├── Dockerfile
├── LICENSE
├── README.md
└── renv.lock

The .do folder contains a template file that is needed if you want to deploy the Shiny app template from the button in the README file. The app specification (app.yaml) that was introduced before, but it is largely simplified:

name: app-platform-shiny
services:
- dockerfile_path: Dockerfile
  github:
    branch: main
    deploy_on_push: true
    repo: analythium/app-platform-shiny
  name: app-platform-shiny

The app folder contains multiple files for the Shiny app. The Dockerfile uses the renv package to handle the dependencies specified in the renv.lock file.

The app itself, as you will see, is a relatively simple example with some range slider, a dependency that does not require building from source. It also has a file upload button – I use the upload functionality to test when a large file upload is required. Large file upload is often restricted, Shiny itself has a 5 Mb file size limit.

GitHub integration

Let's set up the GitHub integration. Go to the DigitalOcean control panel In your control panel. Click on 'Launch New App' under the 'App' item in the left side menu:

When you are asked to choose a source, click on the GitHub icon:

Follow the prompts and install the GitHub app for your personal account or the organization of your choice. Select 'All repositories' or 'Only select repositories' as appropriate. Finally, click 'Save':

Deploy from GitHub repository

Now go back to the Apps control panel. Select the repository from the dropdown menu, specify the branch you wish to deploy. If you want to deploy both a development and a production version of the same repository, you can create multiple apps with the same repo but using different branches.

Leave 'autodeploy' checked if you want to trigger new deployments when the code changes, then click 'Next':

Review the app settings inferred from the Dockerfile, i.e. the HTTP port, etc.:

Type in the name of the service, select the data region, then click 'Next:

The final step lets you define the performance of the app and the corresponding pricing. Here I use the smallest size under the 'Basic' plan for $5 US per month:

After a few minutes, you should see the green checks besides the build and deployment log entries. Building the image might take some time, depending on the number of dependencies in your app:

Follow the app URL from the control panel to see the app online served over HTTPS:

Add a custom domain

Adding a custom domain is done via the control panel. Click 'Add Domain':

Follow the prompts and copy the app URL:

Add a CNAME record in your DNS settings on your DNS providers page (Google Domains shown here), make it point to a domain or subdomain of yours:

You will see the custom domain listed in the app settings:

Click on your new custom domain link and check. You might have to wait a bit if your record has not propagated through the name servers:

Auto deploy updates

Next, we will make some changes to the Shiny app and see the app magically update via git. Do it in your forged repo. The small change I made was to edit the app title. Here is the change:

Once you commit the changes and push to GitHub, it triggers the web-flow pipeline. The new deployment is then listed in the deployment history of your app:

Visit the custom URL and see the new version deployed:

Don't forget to delete the app and the CNAME record if you don't need them anymore to avoid accruing hosting charges. Dynamic app hosting is not free on DigitalOcean.

Testing before deployment

One potential issue with this GitHub integration is it does not depend on passing tests before deployment. If you want to test your changes before deployment, here are the steps to follow:

push changes to the development branch,
run automated tests for the development branch, e.g. using GitHub actions,
set up the production branch as a protected branch, so that merging pull requests require passing tests.

Conclusions

When we deployed an exiting Docker image, the image was pulled to the App Platform from Docker Hub. In this case, an immutable image was pushed and pulled.

The GitHub integration described here takes a different approach. It builds the Docker image after changes are pushed to a pre-defined branch in the GitHub repository. The image is built and stored in the DigitalOcean Container Registry.

With this approach, you get a new deployment on every git push event. You can debate how much this automated deployment (CD) can be considered continuous integration and deployment (CICD) without actually having the CI part.

Testing the app before deployment requires additional steps as part of your git workflow. To achieve full CICD experience, you can use the doctl command-line utility in GitHub Actions. As it turns out, this route is a bit more involved and there are several gotchas to address. Thus, I will explain it in a follow-up post.

How to Host Shiny Apps on the DigitalOcean App Platform

Peter Solymos — Sat, 02 Oct 2021 17:08:38 +0000

By: Peter Solymos

The DigitalOcean App Platform is another PaaS offering that allows you to publish Shiny apps without worrying about infrastructure.

DigitalOcean announced the App Platform a year ago as the platform-as-a-service(PaaS) offering that automates infrastructure management. Developers can deploy their code to production in just a few clicks.

The App Platform can publish applications from your GitHub, GitLab or public Git repositories, or publish a container image you have already uploaded to a Docker registry. The platform supports GitOps (git push-based deployment), and vertical and horizontal scaling.

In spirit and most practical aspects, the platform is very similar to Heroku. You can use Buildpacks or Dockerfiles to build applications. There is no native Buildpack for R. But as we saw before, Shiny apps can be containerized. This gives us a way to deploy to the App Platform seamlessly.

This post focuses on how to deploy a Shiny app from an existing Docker image to the App Platform. This is a fully no-code experience because we already built and pushed the image. You will use the r-minimal Alpine Linux-based COVID19 app that we build before.

Prerequisites

Visit the App Platform landing page and sign up for a DigitalOcean account:

Once have an account, log in and go to your dashboard. In the left drawer, click on 'Apps':

Launch your Shiny app from the dashboard

Deploy

Click on the 'Launch Your App' button:

The next screen will present your options. Select Docker Hub:

Next, type in the repository name (analythium/covidapp-shiny) and the tag (minimal), then click 'Next':

On the next screen, leave the Type as Web Services, set the Run command to the CMD command of the Dockerfile that you are deploying (without the square brackets), and change the port to the one exposed by Docker (3838):

On the next screen, you can edit the app's name your data region:

The last step is to select the plan. Non-static apps require a paid plan ($5 US/month and above):

This is where you can set the number of replicates (horizontal scaling). Finally, click the 'Launch' button:

Check the deployment

After a few seconds, you'll see the app deployed on a subdomain: app-name-hash.ondigitalocean.app:

Visit the link to check the app:

Look around in the dashboard to inspect the deployment and runtime logs. You can even access the container through the console. Under the insights tab, you can check CPU and memory consumption:

Add a custom domain

You can add a custom domain using a CNAME record with your DNS provider. Go to your app's settings tab, find the 'Add Domain' button:

Follow the prompts and copy the app's current URL:

Go to your DNS provider (Google Domains shown here) and add a CNAME record, and save:

After some time the custom domain is live and listed under settings:

Visit the new link, note that it is served over secure HTTPS protocol:

Update the app image

You can edit the source image tag in your app's settings. Once the new version is tagged and pushed to the Docker registry, just edit the tag and the app will be redeployed to the App Platform.

Programmatic image deployment

Of course, you can use the doctl command-line tool and a YAML-based app specification file format for deployment. You can find the app specification in the control panel, look under the Settings tab:

Most of the info in the spec file is self-explanatory and relates to all the settings we made manually. Let's grab the YAML file and use it as a starter.

To use the app spec from the command line you have to install the doctl command-line tool and follow the steps described in the article to use the API token to grant doctl access to your DigitalOcean account.

Deploy an app

We'll make some changes to the app specs and deploy a different app, the one that visualizes the correlation between 2 variables. This will use the analythium/correlation:v1 Docker image. Create an empty file called shiny-app.yml and add the following content to it:

alerts:
- rule: DEPLOYMENT_FAILED
- rule: DOMAIN_FAILED
name: analythium-correlation-shiny
region: nyc
services:
- http_port: 3838
  image:
    registry: analythium
    registry_type: DOCKER_HUB
    repository: correlation
    tag: v1
  instance_count: 1
  instance_size_slug: basic-xxs
  name: shiny-app
  routes:
  - path: /
  run_command: R -e "shiny::runApp('/home/app',port=3838,host='0.0.0.0')"
  source_dir: /

Let's use some doctl apps commands now (these are the commands specific to the App Platform). First, we validate the YAML specification:

doctl apps spec validate shiny-app.yml

We either get an error message or the file content. Once the app spec passed inspection, we can use it to create an app:

doctl apps create --spec shiny-app.yml

We'll see the newly created app info. Take note of the ID. You can get the ID later using the doctl apps list command to list all the apps. This will give the ID, and the Default Ingress, which is the assigned subdomain (https://analythium-correlation-shiny-vil3a.ondigitalocean.app in this case). If the ingress field is empty, it means the deployment is in progress or it has failed.

Use the app ID to see the deployment and runtime logs (edit the APP_ID value according to your console output):

export APP_ID="5c187c9a-5d1a-4988-8848-82905d0f0302"

doctl apps logs $APP_ID --type deploy
doctl apps logs $APP_ID --type run

If all went well, you should see the app after visiting the ingress URL:

Update the app

You can change the YAML spec file, e.g. change the services.image.tag property to tag: v2 and save the file. Then run the following command:

doctl apps update $APP_ID --spec shiny-app.yml

This will update the app in a few minutes:

Delete the apps

Once you are done playing with the new deployment and you don't need it anymore, just click the 'Actions' button, select 'Destroy' from the drop-down menu. Or use the app ID and doctl apps delete $APP_ID to get rid of the apps. Also, remove the CNAME record from your DNS if you have set up a custom domain.

Conclusions

The DigitalOcean App Platform makes it straightforward to deploy Shiny apps without ever opening your terminal. But if you prefer, the command-line option only needs a YAML file and a single command.

Docker once again proved to be a really useful piece of technology. We saw how immutable container images can be moved around the internet without issues. We'll cover the GitHub and App Platform integration in some of the next posts.

ShinyProxy 1-click App

Peter Solymos — Thu, 02 Sep 2021 20:28:10 +0000

By: Peter Solymos

ShinyProxy is a great way to deploy containerized Shiny apps to production and it is only 1 click away.

ShinyProxy is one of the most popular options to host any type of containerized web application, including but not limited to Shiny apps. It provides authentication, authorization, application isolation, scalability, and a lot more with extensive documentation.

ShinyProxy is free and open-source. Although free here means that you are free to do anything with the software within the broad limits of the Apache 2 license. But you will have to manage your server and pay for hosting it.

Is there a simple setup process?

Setting up ShinyProxy can be a bit intimidating if this is the first time you are venturing into this realm. Tutorials about the installation process, configuration, and custom domains are great. But you might wonder if there is an easier way to fast forward the process.

This is the reason Analythium Solutions published the ShinyProxy 1-click app in the DigitalOcean marketplace. We have been successfully using the setup reflected in the tutorials and the 1-click app for some years with our consulting clients. Automating the system setup is beneficial for our own purposes. Releasing it as a marketplace app, however, means that others can benefit from our learnings too.

The ShinyProxy 1-click app info page

This post is a screenshot-based demo of the setup process explaining how you can reclaim an hour of your day. There is also a video version:

An error occurred.

Try watching this video on www.youtube.com, or enable JavaScript if it is disabled in your browser.

ShinyProxy 1-click app tutorial video

What is the Marketplace?

The DigitalOcean Marketplace is a collection of so-called 1-click apps, which are pre-configured machine images that can be used to spin up virtual machines in minutes.

The apps are maintained by DigitalOcean or different vendors. Apps include WordPress for blogging, RStudio image for computing applications, our ShinyProxy image, and a lot more.

Get started

DigitalOcean has a simple and intuitive browser-based dashboard and prices are predictable pricing.

Signup

When you create a DigitalOcean account, you can choose to sign up with an email address or with Google or GitHub single sign-on (SSO) following the prompts after clicking the 'Sign Up' button. Once you sign up, you can set up two-factor authentication to protect your account. You can consider using this referral link and get $100 free credits on sign-up.

Login

To log into your account, click on the 'Sign In' button in the top right corner. This should take you to a form where you can sign in using email and password, or use Google or GitHub SSO. Once you pass login, you'll be prompted to type in a 6-digit verification code that you receive to the email address that you have used for signup.

Add your SSH key

Ideally, you should add your SSH key to your DigitalOcean account before creating the creating resources (virtual machines, etc.). This way your SSH key can be deployed to the new servers and accessing those through the command line will be straightforward.

To set up an SSH key, follow this guide. Once your SSH key is generated you will see two files (these fine names might be different depending on how you generated them), one public, one private. These two files are cryptographically matched, like a key and a lock.

Open the public key file (e.g. .ssh/id_rsa.pub) in a text editor, copy the content. Now go to your DigitalOcean Dashboard, under Settings find Security. Click 'Add SSH Key'. Paste here the contents of the public key file. Add a name, that will remind you which one is yours.

Spin up a "droplet"

Now you signed up and logged in to your account and see your dashboard. The top navigation has a search bar to quickly find your resources, a 'Create' button with a dropdown listing the most common actions. This is where you can find an option to create a new Droplet. What is a "droplet"? A droplet is a small part of the ocean, and this is how DigitalOcean refers to its virtual machines.

The user dashboard with the Create button

Click the 'Create' button and select the 'Droplets - create cloud servers' option. This will take you to a page where you can set the specifications for the new Droplet. You can create the new Droplet as a fresh Linux server (you can select many different Linux distributions, including Ubuntu, Debian, etc., and different versions), or you can use one of your saved images to replicate a given setting.

Search and select the ShinyProxy 1-Click app. This is a pre-configured image on Ubuntu 20.04.

Select ShinyProxy from the Marketplace as your image

After the image is selected, you can choose a plan based on hardware configuration and pricing (see Droplet pricing here).

Next, you can pick your data region, I picked Toronto (Canada) here, feel free to choose what's closest to you.

Finally, you can add additional monitoring (recommender: this allows collecting more refined monitoring data and is required for alerts).

Authentication can be based on a root password or using SSH keys. If you select SSH key-based authentication, password-based authentication will be turned off by default.

Finally, you can specify how many Droplets you wish to create with the same specifications, you can edit the hostname (this is how the new Droplet will show up in the dashboard), or add backup for 20% of the Droplet price.

Set monitoring, security, and replication levels

If everything looks good, you can click the 'Create Droplet' button at the bottom and wait for your Droplet to be provisioned.

Click 'Create Droplet'

Once the progress bar reaches 100% you can copy the IP address and visit the droplet in the browser or in the terminal.

Droplet creation in progress

In the meantime you can study the 1-click app information on what to do next once the virtual machine is up.

ShinyProxy 1-click app description

Click on the droplet name to see the relevant dashboard. The green dot indicates a healthy instance, but you might have to wait another minute before you see CPU and memory usage info. Click on the 'ipv4' address, it will copy the address to your clipboard.

Droplet about the new instance

API Creation

In addition to creating a Droplet from the ShinyProxy 1-Click App via the control panel, you can also use the DigitalOcean API.

As an example, to create a 4GB ShinyProxy Droplet in the SFO2 region, you can use the following curl command. You’ll need to either save your API access token to an environment variable or substitute it into the command below.

curl -X POST -H 'Content-Type: application/json' \
     -H 'Authorization: Bearer '$TOKEN'' -d \
    '{"name":"choose_a_name","region":"sfo2","size":"s-2vcpu-4gb","image":"analythium-shinyproxy-20-04"}' \
    "https://api.digitalocean.com/v2/droplets"

Log in using the ShinyProxy UI

It takes some time for all the services, including Docker and ShinyProxy to come online. Once your new droplet with the ShinyProxy 1-Click app is up and running, you can visit your droplet's IP address. Use admin/password or user/password as user name and password to log into your ShinyProxy instance.

ShinyProxy login page

Once you are in, you'll see two demo R Shiny applications. Try them to make sure Docker is running fine.

ShinyProxy listing the available applications

Now you can modify the configuration to deploy your own apps. This will require you to access the droplet through a secure shell (SSH).

Log in through SSH

Secure shell or ssh is a command-line tool and also a cryptographic network protocol for operating network services securely over an unsecured network.

SSH provides a secure channel connecting an SSH client application with an SSH server (the standard port for SSH is 22).

You'll be able to log into a new Droplet if your SSH key was added on droplet creation:

ssh -i c:/.ssh/id_rsa root@142.93.156.135

This will use the SSH private key (c:/.ssh/id_rsa) to let you into the server as the root user (superuser).

I used the full path to tell ssh where to find the private key (using the -i flag) that will be matched with the public key already on the server. You'll be asked to add the server to your list of trusted hosts on the first login, just answer yes.

Message of the day after log in through SSH

Edit /etc/shinyproxy/application.yml to configure your instance. Pay special attention to authentication: it is set to simple. You should change user names and passwords, possibly the authentication type.

Pull Docker images and add those to the configuration file to deploy your Shiny apps.

Then restart ShinyProxy for the changes to take effect using sudo service shinyproxy restart.

All this is explained in the following posts:

You can also go ahead and set up a custom domain name and HTTPS:

Don't forget to destroy (and not just power off) your droplet if you don't need it any longer to avoid any unwanted changes.

Conclusions

The ShinyProxy 1-click app is a machine image that helps you get your production server up and running in no time. The usual setup process can take anywhere between 30 minutes to an hour. With the 1-click app, it takes only a few minutes. It is still valuable to understand the details, but the app can be a real-time saver in the long run.

Securing ShinyProxy with Caddy Server

Peter Solymos — Mon, 30 Aug 2021 14:13:44 +0000

By: Peter Solymos

Use Caddy server to obtain TLS certificates for your custom domain and to serve Shiny apps securely with ShinyProxy.

In a previous post, I explained how to add a custom domain and HTTPS to your ShinyProxy server using Nginx, and how to set up certificate auto-renewals with Certbot. That was a really long post and the setup included many steps.

A while back, I also used Caddy server to secure a Shiny Server instance. That process was quite straightforward with a lot fewer moving parts. Let's see if we can do the same for ShinyProxy, as it is pictured below.

ShinyProxy setup with Caddy server.

Prerequisites

Start a Ubuntu 20.04 virtual machine and follow instructions from the introductory ShinyProxy post to have the server available on http://$HOST:8080 with the two demo applications.

For a Let's Encrypt certificate, you need a fully registered domain name and an email address. I use the example.com domain here, you have to substitute your domain name. Add an A record with example.com pointing to your server's public IP address.

Install Caddy

Add some keys and update the apt sources, then install Caddy:

apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo apt-key add -
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list

sudo apt update
sudo apt install caddy

Add Caddyfile /etc/caddy/Caddyfile with the following content, replace your email in the global configuration block (some challenges and Let's Encrypt notifications require the email, but this block is optional):

{
        email your.name@example.com
}

example.com {
        reverse_proxy 127.0.0.1:8080
}

Restart Caddy with systemctl reload caddy.

Set the firewall

You can firewall off everything except for the SSH, HTTP, and HTTPS ports:

ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http
ufw allow https
ufw --force enable

That is it. Now you can visit https://$HOST to see the ShinyProxy login page with the secure lock icon:

If you decide to destroy your virtual machine, do not forget to remove the DNS record for your custom domain to prevent a hostile subdomain takeover.

Conclusions

The brevity of this post should be convincing enough that adding a custom domain and HTTPS with Caddy is easy as a breeze. Caddy will take care of the certificate renewals.

Remember that it is your job to make sure your app is not exposing sensitive information and that you follow Docker best practices to minimize risk to your users.

Custom Domain and Security for ShinyProxy with Nginx

Peter Solymos — Mon, 23 Aug 2021 15:44:53 +0000

By: Peter Solymos

Here are the final steps to make your ShinyProxy instance secure and production-ready.

This is the last post in this ShinyProxy series where I describe how to configure ShinyProxy with Nginx using HTTPS, set up certificate renewals, and a custom domain name.

Configuring ShinyProxy is a complex topic, that is the reason I have devoted quite a few posts to it including installation, configuration, and maintenance:

Let's see what else is to be done to have your very own, production-ready, and secure ShinyProxy instance, just like as pictured below.

ShinyProxy setup with Nginx.

Prerequisites

Start a Ubuntu 20.04 virtual machine and follow instructions from the introductory ShinyProxy post to have the server available on http://$HOST:8080 with the two demo applications.

If you are short on time and need a virtual machine with ShinyProxy already installed, try the ShinyProxy 1-click app from the Digitalocean Marketplace for $5 a month. I explain the setup in this short video. The 1-click machine image has Nginx configured, so feel free to jump ahead to the Firewall settings section.

Run ShinyProxy on port 80

Your ShinyProxy instance is running on port 8080 (http://$HOST:8080):

This is common practice in development mode. 8080 is an alternative to port 80 (HTTP). When you type $HOST in the browser, it is going to take you to http://$HOST:80 implicitly. Unlike port 80, this 8080 needs an explicit port override to request in the browser.

Port 8080 is there to remind you of the fact that you need to decide how you want to host ShinyProxy. I explain how to use Nginx as a reverse proxy to serve ShinyProxy on port 80.

Install Nginx

Nginx (pronounced "engine X") is a web server that can also be used as a reverse proxy or load balancer. A large portion of the Internet is running on Nginx, so it is a good idea to get some exposure. Let's install Nginx:

apt-get install nginx

Set up a reverse proxy

A reverse proxy is a server that is placed in front of other servers and forwards browser requests to those servers. In our case, Nginx forwards the request made to port 80 to ShinyProxy listening on port 8080.

You can read more about the difference between a (forward) proxy and a reverse proxy setup here. For our purposes, the important security consideration is that Nginx as a reverse proxy ensures that no client ever communicates directly with the ShinyProxy server.

Edit the config file (/etc/nginx/sites-enabled/default) with vim or nano to have the following content:

server {
        listen 80 default_server;
        listen [::]:80 default_server;

    ## Add here your custom domain
    # server_name example.com www.example.com;
    server_name _;

    location / {
        proxy_pass          http://127.0.0.1:8080/;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 600s;

            proxy_redirect    off;
            proxy_set_header  Host             $http_host;
            proxy_set_header  X-Real-IP        $remote_addr;
            proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;
            proxy_set_header  X-Forwarded-Protocol $scheme;
        }
}

Save the file and restart Nginx with service nginx restart. After this, Nginx will send all traffic to ShinyProxy which is running on port 8080. You can visit http://$HOST/ to access ShinyProxy:

Firewall settings

We will use a firewall to filter all the ports except for the SSH (22), HTTP (80) and HTTPS (443) ports.

The server behind the firewall will use other ports. For example, ShinyProxy itself will connect to the ports on the Docker host that are mapped to the individual containers. It is not recommended to make these ports accessible to the outside world.

For this reason, we install ufw, the Uncomplicated Firewall, to allow incoming traffic only on the above-mentioned ports:

apt install ufw

ufw default deny incoming
ufw default allow outgoing

ufw allow ssh
ufw allow http
ufw allow https

ufw enable # answer y

Now check ufw status:

Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
22/tcp (v6)                ALLOW       Anywhere (v6)
80/tcp (v6)                ALLOW       Anywhere (v6)
443/tcp (v6)               ALLOW       Anywhere (v6)

Configure your custom domain

For a Let's Encrypt certificate, you need a fully registered domain name. I use the example.com domain. You have to substitute your domain name. If you want your server hosted on an apex domain name (example.com), it is recommended to set up a www subdomain as well. Both of the following domain name system (DNS) records need to be set up for your server:

an A record with example.com pointing to your server's public IP address,
an A record with www.example.com pointing to your server's public IP address.

In the /etc/nginx/sites-available/default file, find the line server_name _; and change it to server_name example.com www.example.com; (see comments in the snippet above).

If you are using a subdomain, e.g. shinyproxy.example.com, you only need a single DNS record set up and the single subdomain added to the Nginx default configuration. This is what I am doing in my example using Google Domains:

Restart Nginx to enable your changes by systemctl restart nginx.

Visit your domain to see the login page:

Set up HTTPS

The image above showed the lock icon crossed which indicates we are accessing the custom domain over a non-secure HTTP connection. To have a secure connection over HTTPS, we need to set up Transport Layer Security (TLS).

TLS is a protocol that encrypts traffic between the client and the server. It relies on a set of trusted third-party Certificate Authorities (CA) to establish the authenticity of certificates. One such non-profit Certificate Authority is called Let's Encrypt, which is what we used in the example.

Install Certbot

Let’s Encrypt certificates expire after 90 days, so it is a good idea to use software tools for auto-renewal. We use Certbot to keep certificates up-to-date.

Add repository for up to date Certbot version:

add-apt-repository ppa:certbot/certbot

You'll need to press ENTER to accept.

Install Certbot's Nginx package with apt:

apt install python3-certbot-nginx

Certbot is now ready to use.

Obtaining a TLS certificate

Run this command after changing the domain names:

certbot --nginx -d example.com -d www.example.com

Be careful with capitalization. Browsers might not be case sensitive but Nginx and Certbot want things nice and clean and matching DNS settings.

If this is your first time running Certbot, you will be prompted to enter an email address and agree to the terms of service. Providing a valid email address is encouraged so that you can get notified about expiring certificates or any other issues that might arise.

After doing so, Certbot will communicate with the Let's Encrypt server, then run a challenge to verify that you control the domain that you are requesting a certificate for.

When prompted whether or not to redirect HTTP traffic to HTTPS (option 1) or remove HTTP access (option 2), I usually pick the second option. This is why there is no need to firewall off port 80, see https://letsencrypt.org/docs/allow-port-80/ for an explanation.

If you pick option 1, make sure to read the ShinyProxy security documentation and forward request headers.

Verifying auto-renewal

Run this command for a dry run: sudo certbot renew --dry-run.

If you reload the custom domain in your browser, it should redirect to the HTTPS, notice the lock icon is not crossed anymore:

Remember that it is your job to follow Docker best practices and not use admin privileges inside the containers. You should also be careful that your app is not exposing sensitive information.

If you want to stop and destroy your virtual server, do not forget to remove the DNS record for your custom domain and subdomain to prevent a hostile subdomain takeover.

Conclusions

You are in full control when it comes to ShinyProxy. You can do whatever you like, but you also need to understand how to configure your server and be willing to expand your knowledge base as needed.

Well done! You just learned how to add your custom domain to the ShinyProxy server. This will certainly help your personal or corporate branding.

But your users want also their data to be safe. That's why HTTPS is so important. Security measures for your site can always be improved, and security should always be top of your mind starting from your Shiny app to your firewall settings.

The Quickest Way to Add New Apps to ShinyProxy

Peter Solymos — Mon, 16 Aug 2021 14:33:21 +0000

By: Peter Solymos

Once your ShinyProxy server is up and running, you need to maintain and update it. Last time we looked at how to update existing apps. This section explains how to add new ones hassle-free.

ShinyProxy serves containerized web applications, like Shiny apps, without limiting the number of concurrent users or application hours. It is free, open-source, and comes with free enterprise features, such as authentication and app-level authorization.

In the previous posts, I introduced ShinyProxy and its configuration, then reviewed how to update existing apps without disrupting the users. But once in a while, you want to add new apps to ShinyProxy. This is when you have to update the configuration and do a few more steps. Let's see what are these steps and how to make the process more efficient.

Prerequisites

You can find the configuration file application.yml for this tutorial in the analythium/shiny-correlation GitHub repository.

To follow along, set up a virtual machine running Docker and ShinyProxy following this post on any cloud provider using Ubuntu 20.04. Or you can use the ShinyProxy 1-click app from the Digitalocean Marketplace:

Create a droplet (virtual machine) using the ShinyProxy marketplace image

Once you know the host URL or IPv4 address, set the $HOST environment variable and log in as root user using ssh:

export HOST="178.128.235.125"

ssh root@$HOST

Steps to add new apps

The ShinyProxy configuration is defined by the application.yml file in the /etc/shinyproxy folder. This file lists the apps, therefore adding new apps involve the following steps:

update the application.yml to include the new apps
pull the new Docker images to the host with docker pull
restart the ShinyProxy service with service shinyproxy restart

Steps 2 and 3 happen on the server. But step 1 can be achieved in a few different ways. Let's see what those are.

Edit configuration on the server

The simplest approach is to edit the YAML file directly on the server. Open the file with nano or vim, remove unwanted applications (I removed both demo apps from the /etc/shinyproxy/application.yml), and add the new apps. I show the final application.yml file here for reference:

proxy:
  title: Shiny Proxy
  logo-url: https://hub.analythium.io/assets/logo/logo.png
  landing-page: /
  favicon-path: favicon.ico
  heartbeat-rate: 10000
  heartbeat-timeout: 60000
  port: 8080
  authentication: simple
  admin-groups: admins
  users:
  - name: admin
    password: password
    groups: admins
  - name: user
    password: password
    groups: users
  docker:
    url: http://localhost:2375
    port-range-start: 20000
  specs:
  - id: cor2d
    display-name: Correlation in 2D
    description: App with 2D kernel density estimate
    container-cmd: ["R", "-e", "shiny::runApp('/home/app')"]
    container-image: analythium/correlation:v1
    logo-url: https://hub.analythium.io/assets/apps/cor2d.png
    access-groups: []
  - id: cor3d
    display-name: Correlation in 3D
    description: App with kernel density estimate with 3D RGL plot
    container-cmd: ["R", "-e", "shiny::runApp('/home/app')"]
    container-image: analythium/correlation:v2
    logo-url: https://hub.analythium.io/assets/apps/cor3d.png
    access-groups: [admins, users]

logging:
  file:
    name: shinyproxy.log

The two new apps are the same as what you saw when discussing how to update existing apps. The access-groups: [] value means that all authenticated users can access the 2D version of the apps.

Pull the new Docker images:

docker pull analythium/correlation:v1
docker pull analythium/correlation:v2

If you are using a private Docker registry, you have to log in. It is best to generate a token with restricted scopes (i.e. pull-only privileges). You can use this token instead of your password, and it is easy to revoke the token any time without having to change your password at all the places you have used it. This is a one-time thing, the username and token will be saved and used later when login is required. Save the token into a file (the double space at the beginning of the line prevents the token to end up in bash history):

  echo your_token > ./token.txt

Now you can pass the token via standard input, use your username and registryname:

cat ./token.txt | docker login --username username --password-stdin registryname

Finally, restart ShinyProxy:

service shinyproxy restart

If you were logged into the ShinyProxy server, now you should see 502 Bad Gateway until ShinyProxy restarts, then you'll be asked to log in again.

This is what will happen with any of the logged-in users on the server. To avoid such disruptions, you can schedule the updates to times when there are no users logged in, or when the disruption is minimal and advertised to users in advance. You can also use Redis in-memory database for session persistence.

The approach described here is a perfectly functional way of updating ShinyProxy configuration. But once the file is changed, it is harder going back unless you saved a backup copy somewhere. This is why using version control could be helpful.

Use git to edit and update the configuration

Edit the application.yml on your local machine and git push the changes to a remote repository. When you first update the configuration on the server, git clone the project you have under version control:

git clone https://github.com/analythium/shiny-correlation.git

Next time, you can just cd into the directory and git pull the changes:

cd shiny-correlation
git pull

Copy the YAML configuration file into the /etc/shinyproxy folder:

cp application.yml /etc/shinyproxy

Pull the new images and restart ShinyProxy as in the previous section:

docker pull analythium/correlation:v1
docker pull analythium/correlation:v2

service shinyproxy restart

This approach gets around the versioning issue we had before. But typing the new image names one by one seems tedious. Let's see if we can find some efficiencies.

Push configuration changes over SSH

I am going to explain a workflow similar to the previous one but instead of pulling changes, I am going to use ssh and scp to transfer the file to the server and run the docker pull and ShinyProxy restart commands.

Edit the application.yml and commit the changes so you have the full history. Once your application.yml is ready, cd into the local directory where the configuration file is located and download the following script file:

curl -s https://raw.githubusercontent.com/analythium/shinyproxy-1-click/master/digitalocean/setup.sh -o setup.sh

The general usage of the freshly downloaded setup.sh is:

bash setup.sh -i ~/.ssh/id_rsa -s root@ip_address -f application.yml

The following command-line arguments need to be passed to the setup.sh script:

-i: path to your ssh key
-s: user name (root\) and the IP address, e.g. user@178.128.235.125
-f: path and file name of the ShinyProxy config, e.g. /path/to/application-new.yml.

The script then takes care of the steps in the following order:

Copies the updated YAML to the server into the /etc/shinyproxy folder
pulls the Docker images listed in the YAML file: updates the ones already pulled before, and the ones newly added too
restarts the ShinyProxy service

This is the command for our actual YAML file inside the git repository:

bash setup.sh -i ~/.ssh/id_rsa -s root@${HOST} -f application.yml

You can see an output similar to this one:

[INFO] Copying application.yml to host
[INFO] Updating docker images according to application.yaml
v1: Pulling from analythium/correlation
420047682034: Pulling fs layer

...

7bdb658c82cf: Pull complete
Digest: sha256:175663b0be8c97743f7a5bd1960c59d3f09536cd762a0f6f0a5744ee55ddb7ce
Status: Downloaded newer image for analythium/correlation:v1
docker.io/analythium/correlation:v1
v2: Pulling from analythium/correlation
420047682034: Already exists

...

b65c57a60c68: Pull complete
Digest: sha256:4eaa98afe32b88d4cce7ce5436956ecb38e94c776bee97244a8fc28cde8e415f
Status: Downloaded newer image for analythium/correlation:v2
docker.io/analythium/correlation:v2
[INFO] Restarting ShinyProxy
[INFO] Done

The script parses the YAML configuration and finds the container-image entries. This is done by the following script in the background:

#!/bin/bash
db=$(grep 'container-image:' $1 | sed 's/[^:]*://' | sed 's/^[[:space:]]*//g')
while IFS= read -r line; do docker pull $line; done <<< "$db"

Check your ShinyProxy server, you should see the two new apps added:

ShinyProxy listing the 2 new images after the updates

Don't forget to shut down the server if you don't need it anymore.

Conclusions

We reviewed 3 ways of updating ShinyProxy. The 3rd script-based update of the application.yml file is quite efficient if you have all the command line tools handy. Changes can be version-controlled, and you are one line away from an updated ShinyProxy server without navigating files and directories on the host machine. The script even scans the YAML file for all the Docker images that need to be pulled. The only thing to remember is to perform these updates at a time when it causes minimal disruption to users.

Update Existing Shiny Apps in ShinyProxy

Peter Solymos — Mon, 09 Aug 2021 18:50:54 +0000

By: Peter Solymos

ShinyProxy runs containerized applications while providing enterprise features for free. Learn how to update apps without disrupting your users.

ShinyProxy is one of the best self-hosting options out there when you need enterprise features, like authentication and app-level authorization, for your app. In this post, I explain how to manage existing applications. All this without restarting your ShinyProxy server and thus keeping all your users connected. There are a few important considerations to remember when working with Docker images.

You can find the code for this tutorial in the analythium/shiny-correlation GitHub repository.

What to consider when updating apps

Remember how the apps are listed in the application.yml file in the /etc/shinyproxy folder:

  - id: 02_hello
    display-name: Demo Shiny App
    description: App with sliders and file upload
    container-cmd: ["R", "-e", "shiny::runApp('/home/app')"]
    container-image: analythium/shinyproxy-demo:latest
    access-groups: [admins]

There are two ways of updating the apps in ShinyProxy. You can update only the Docker image. This means that the container-image tag and everything else in the config YAML file stays the same. Because the application.yml file remains the same, there is no need to restart the ShinyProxy service on the server.

If you change anything in the application.yml configuration file, you'll have to restart the ShinyProxy service for these changes to take effect. This might include replacing a container-image tag, adding new apps, or touching any of the general configurations.

Deploying and managing ShinyProxy can get complex when many apps are used, especially when the configuration of ShinyProxy is often updated. When restarting a running ShinyProxy instance (in order to update its configuration), users will face a disconnect from their running applications. – ShinyProxy Docs

Restarting ShinyProxy would mean that all current users would lose their Shiny sessions and logged-in users would be logged out. This is a disruption one can avoid by managing the Docker images carefully. You can also schedule the updates to times when there are no users logged in, or when the disruption is minimal and advertised to users in advance.

Let's focus on the first scenario. The rest of the post explains how you can manage your image tags to minimize disruption to your users.

Add an app to ShinyProxy

The analythium/shiny-correlation GitHub repository contains a simple Shiny app. This app is a bivariate version of the Hello Shiny histogram example. We will build this app locally, then deploy it to the ShinyProxy server.

Build the Docker image

Let's build the Docker image without a tag, using the -f flag to specify which Dockerfile to use:

docker build -f Dockerfile-v1 -t analythium/correlation .

The build process should be pretty fast because I specified a parent image that already contains all the dependencies. Here is the Dockerfile-v1:

FROM analythium/shinyproxy-demo:latest
RUN rm -rf /home/app/*
COPY ./app-v1.R ./app.R

If you list the Docker images with docker image ls, you see something interesting. The image has a tag called :latest, we'll come back to this in a bit:

REPOSITORY                   TAG                 IMAGE ID
analythium/correlation       latest              90732201668c

You can test the image by running the docker run -p 4000:3838 analythium/correlation command and visiting http://localhost:4000.

Change the sample size and the correlation and watch how the 2-dimensional estimates of the intensity are changing:

Shiny app: scatterplot and 2D density of 2 correlated variables

Add the app to ShinyProxy

Follow the ShinyProxy setup instructions or use the ShinyProxy 1-click app from the Digitalocean Marketplace to set up your ShinyProxy server.

Push the image to the registry (docker push analythium/correlation:latest), add it to the ShinyProxy configuration, pull the image on the host machine, and restart the ShinyProxy service with service shinyproxy restart.

Update an existing app

When the Docker image itself changes, but its name (including the tag) stays the same, you won't have to restart ShinyProxy. This can happen when you push a new version of the Docker image to your Docker registry, e.g. after updating the analythium/correlation image.

Update the Docker image

The version 2 update of the Shiny app includes a 3D representation of the distribution using the wonderful rgl R package. Say, we want to tag the new image as :v2 because we'd like to keep both versions. We use Dockerfile-v2:

FROM analythium/shinyproxy-demo:latest
USER root
RUN install2.r -r http://cran.rstudio.com/ rgl
RUN rm -rf /home/app/*
COPY ./app-v2.R ./app.R
USER app

This Dockerfile copies version 2 of the Shiny app into the image and installs the rgl package.

Let's build the new image:

docker build -f Dockerfile-v2 -t analythium/correlation:v2 .

Now print out the list of Docker images with docker image ls:

REPOSITORY                   TAG                 IMAGE ID
analythium/correlation       v2                  9d98df8ea853
analythium/correlation       latest              90732201668c

We have a new image tagged as :v2 but the previous image stayed the :latest. This is counter-intuitive behaviour because we would think that the :latest tag would point to the image that was built last. Instead, it means "the last build/tag that ran without a specific tag/version specified".

If you push a new image with a tag which is neither empty nor ‘latest’, :latest will not be affected or created. – Vladislav Supalov

Image tagging and versioning

You need to be careful when omitting the image tag. Here is what you can do to version the images and to also keep the :latest tag consistent with the intuitive notion of the latest image.

Let's tag the previously created (version 1) image as :v1 using the docker tag command:

docker tag analythium/correlation analythium/correlation:v1

List and verify, :latest is now an alias for :v1 because the image IDs match:

REPOSITORY                   TAG                 IMAGE ID
analythium/correlation       v2                  9d98df8ea853
analythium/correlation       latest              90732201668c
analythium/correlation       v1                  90732201668c

Now rebuild the image without the tag (this should be quick due to caching), this will make the new image the :latest. Then tag the latest image as the new version:

docker build -f Dockerfile-v2 -t analythium/correlation .
docker tag analythium/correlation analythium/correlation:v2

List and verify again, the :latest image ID matches :v2:

REPOSITORY                   TAG                 IMAGE ID
analythium/correlation       latest              9d98df8ea853
analythium/correlation       v2                  9d98df8ea853
analythium/correlation       v1                  90732201668c

Test the new image with docker run -p 4000:3838 analythium/correlation and visit http://localhost:4000/ to play with the 3D rendering of the correlated variables:

Push these images to the registry:

docker push analythium/correlation
docker push analythium/correlation:v1
docker push analythium/correlation:v2

Update the app on the ShinyProxy server

After this little Docker detour, we are on track to update the app in ShinyProxy because the latest tag will mean version 2:

ssh into the ShinyProxy server as the root user
pull the new version of the image with docker pull analythium/correlation:latest

When you update the image, there is no need to restart the Docker or ShinyProxy services. The next time a user starts the application on your ShinyProxy installation, the new image will be used to launch a container.

Versioning your images is considered good practice. It makes a lot of sense so that you can roll back changes when something unexpected happens.

If you tag your images carefully and test the apps locally, you can use the :latest image tag in your ShinyProxy configuration without much trouble. You can also roll back to :v1 if anything unexpected comes up – you'll have to change the config and restart the service in this case.

Update multiple existing apps

If you have multiple apps, you will have to pull the latest version for each. You can use a script for that, or use the following 1-liner to update all the Docker images on your host:

docker images |grep -v REPOSITORY|awk '{print $1":"$2}'|xargs -L1 docker pull

Sometimes, you might want to remove dangling images with docker image prune -f. A dangling image is one that is not tagged and is not referenced by any container, i.e. intermediate layers, etc., which might not be used by the latest images. These can accumulate over time and take up space.

Add a Cron job

You can set up a Cron job to periodically update the images on the server. Run crontab -e as root to have access to the Cron utility. Pick an editor (e.g. nano) if you haven't done so already and then add these lines to the bottom of the file and save it:

# Cleanup at 3:00am every Sunday
0 3 * * 0 docker image prune -f

# Update all images at 1:00am every day
0 1 * * * docker images |grep -v REPOSITORY|awk '{print $1":"$2}'|xargs -L1 docker pull

Check the settings using crontab -l.

Cron jobs represent a polling type of update, which means we are regularly checking for updates. On one hand, if changes to the images are infrequent, there is no need for constant polling. On the other hand, setting Cron intervals too large might lead to missing important updates.

Pick the frequency of updates with this in mind. Webhooks for existing apps and images are considered a better alternative to polling, although webhooks require a bit more work that I am not covering here.

Conclusions

Updating existing apps for your ShinyProxy server is straightforward. You are only one command away from updating all the already deployed apps at once – no disruption to your users. The next user who opens the app will see the updated version. However, be careful with tagging and versioning your images to avoid unwanted surprises.

Forem: Peter Solymos

Manage Dependencies with the deps R Package for Docker Containers

Reproducibility

Package-based development

The middle ground

Tags

Using the deps package

Docker workflow

Conclusions

Further reading

Containerizing Shiny for Python and Shinylive Applications

Shiny app template

Example app with plot

The Dockerfile

Shinylive

Conclusion

Further readings

We Need to Talk About Docker Registries

What is a Docker registry?

Why do you need a registry?

Roll your own registry

Log into a registry

Commonly used registries

Summary

Further readings

CICD Patterns with GitHub Actions and Docker

Why should you care?

Non-git-based delivery and deployment

Local image build

PaaS-based image build

Enter GitOps

Git-triggered and PaaS-based image build

Git-triggered and GitHub-based image build

Conclusions

How to Pick the Right Hosting Option for Your Shiny App

How not to do a review

The 3 step process to make the decision

1. Start with the why

2. List your requirements

3. Identify your constraints

You are done!

Options at a glance

What is next

Auto deploy Shiny app changes to the DigitalOcean App Platform

Prerequisites

GitHub integration

Deploy from GitHub repository

Add a custom domain

Auto deploy updates

Testing before deployment

Conclusions

Further reading

How to Host Shiny Apps on the DigitalOcean App Platform

Prerequisites

Launch your Shiny app from the dashboard

Deploy

Check the deployment

Add a custom domain

Update the app image

Programmatic image deployment

Deploy an app

Update the app

Delete the apps

Conclusions

Further reading

ShinyProxy 1-click App

Is there a simple setup process?

An error occurred.

What is the Marketplace?

Get started

Signup

Login

Add your SSH key

Spin up a "droplet"

API Creation

Log in using the ShinyProxy UI

Log in through SSH

Conclusions

Further readings

Securing ShinyProxy with Caddy Server