Sean's Corner

Rclone On Serverless GCP - Cloud Run [Nope]

  • Jul 17, 2020
  • 0 views

In this first article of a series on running Rclone on serverless platforms on Google Cloud Platform, where we’ll try running Cloud Run on many unexpected services.

Spolier alert: Rclone on Cloud Run doesn’t work.

Table of Contents

Why on Serverless GCP?

Migrating large amounts of data between Google Drive accounts can be pretty difficult: there’s the 100GB/day server side copy quota, and a 750GB/day upload quota. Rclone deals with all these problems relatively easily, but you’ll still have to run it somewhere, whether it be on your own computer (and get hit by ISP’s data cap), or on a rented VPS (sometimes very expensive).

Since serverless is now a thing, why not run it on GCP? High bandwidth to Google infrastructure, completely free ingress, free egress to any Google service, and only get billed for the CPU & RAM you use!

Why (& Why Not) Cloud Run?

Well it’s the first thing I thought of: a serverless platform that can run any Docker container you tell it to. You’re only billed the CPU seconds you actually run Rclone, and there’s even a free 180,000 CPU seconds quota every month.

However, upon further inspection, it turned out Cloud Run is really unsuitable for this: it’s designed for scaling short-lived containers that handles HTTP requests. So there are practical restrictions like:

  • Maximum 900s run time for each container
  • Must respond to HTTP health probes
  • When the service is scaled down to 0, the container is killed with SIGKILL

And because of these restrictions (especially the third one), it didn’t really work, since Cloud Run would just scale the service to 0 after the request finishes. There might be ways to get around this, but in practice this is just not the place to run Rclone.

The Failed Attempt

Building the Rclone Image

Since the container needs to respond to HTTP requests, nginx is chosen as the base image. Also rclone.conf is copied directly into the container because I don’t really want to deal with GCP Secret Manager (& Cloud Run’s env don’t really support multi-line).

FROM nginx:alpine
COPY entrypoint.sh /entrypoint.sh
COPY rclone /usr/bin/rclone
COPY rclone.conf /rclone.conf
RUN chmod +x /entrypoint.sh && chmod +x /usr/bin/rclone
ENTRYPOINT ["/entrypoint.sh"]

For the entrypoint, nginx would be started in the background, and we’ll run rclone immediately after.

#!/bin/sh
# entrypoint.sh

set -e
nginx &
rclone --config="/rclone.conf" $RCLONE_COMMAND

Rclone is fetched, unzipped, and then we’ll build and push the image.

curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip rclone-current-linux-amd64.zip
cd rclone-*-linux-amd64
cp rclone ..
cd ..

docker build . -t gcr.io/<project-id>/rclone
docker push gcr.io/<project-id>/rclone

Setting up Cloud Run

  • Container image URL: gcr.io/<project-id>/rclone
  • Container port: 80
  • Memory allocated: 256MiB - Later bumped to 512MiB
  • CPU allocated: 1
  • Request timeout: 900 seconds - It’s gonna get killed anyways
  • Maximum number of instances: 1

Environment variable:

  • RCLONE_COMMAND: copy src: dest: --transfers=64 --drive-stop-on-upload-limit --stats 1s --stats-one-line

Here --drive-stop-on-upload-limit is used, since by default, rclone will just keep trying until it can upload again.

Results

After the service is live, by manually visiting the endpoint, a container should be spun up & serves the request.

Immediately after that, rclone starts to run. Then (nearly) immediately, the responce finishes and the service is scaled to 0, killing the container (and therefore rclone).

Sigh. And therefore there’s no need to setup Cloud Scheduler to fetch this endpoint on Cron.

Retrospective

It could have been fun if it worked. But I didn’t read the scaling docs, and made this whole thing a waste of time.

Welp, just realized GAE can also run any container you want it to. Guess that’s next.

Search