Global EC2 Monitoring Implementation Proposal

Uma Shankar Pradhan Published on : 04 January 2021 4 minutes

Background information: Rclone is a command-line / UI Based program to manage files on cloud storage. It is a feature-rich alternative to cloud vendors’ web storage interfaces. Over 40 cloud storage products support rclone, including S3 object stores, business & … Continue reading

Background information:

Rclone is a command-line / UI Based program to manage files on cloud storage. It is a feature-rich alternative to cloud vendors’ web storage interfaces. Over 40 cloud storage products support rclone, including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.

Rclone mounts any local, cloud, or virtual filesystem as a disk on Windows, macOS, Linux, and FreeBSD, and also serves these over SFTP, HTTP, WebDAV, FTP, and DLNA.

Rclone is mature, open-source software initially inspired by rsync and written in Go. The friendly support community is familiar with various use cases. Official Ubuntu, Debian, Fedora, Brew, and Chocolatey repos. include rclone. For the latest version downloading from rclone.org is recommended.

What can rclone do for you?

Rclone helps you:

  • Backup (and encrypt) files to cloud storage
  • Restore (and decrypt) files from cloud storage
  • Mirror cloud data to other cloud services or locally
  • Migrate data to cloud or between cloud storage vendors
  • Mount multiple, encrypted, cached or diverse cloud storage as a disk
  • Analyze and account for data held on cloud storage using lsf, ljson, size, ncdu
  • Union file systems together to present multiple local and/or cloud file systems as one

The Feature of MultiCloud (Rclone):

  • Transfers
    • MD5, SHA1 hashes are checked at all times for file integrity
    • Timestamps are preserved on files
    • Operations can be restarted at any time
    • Can be to and from network, eg two different cloud providers
    • Can use multi-threaded downloads to local disk
  • Copy new or changed files to cloud storage
  • Sync (one way) to make a directory identical
  • Move files to cloud storage deleting the local after verification
  • Check hashes and for missing/extra files
  • Mount your cloud storage as a network disk
  • Serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna
  • Experimental Web based GUI

Technical Benefits:

  • Various data collection methods and protocols.
  • Flexible and extendable data gathering.
  • Flexible trigger expressions.
  • Remote command execution to recover services down issues.
  • Flexible notifications

Supported Providers

(There are many others, built on standard protocols such as WebDAV or S3, that work out of the box.)

List of Top Providers
Amazon S3

1Fichier

Alibaba Cloud (Aliyun) Object Storage System (OSS)
Amazon Drive (See note)
Backblaze B2

Box

Ceph

Citrix ShareFile

C14

DigitalOcean Spaces

Dreamhost

Dropbox
FTP
Google Cloud Storage
Google Drive

Google Photos

HTTP

Hubic
Jottacloud

IBM COS S3

Koofr
Mail.ru Cloud

Memset Memstore

Mega

Memory

Microsoft Azure Blob Storage

Microsoft OneDrive

Minio

Nextcloud
OVH
OpenDrive

OpenStack Swift

Oracle Cloud Storage
ownCloud

pCloud

premiumize.me

put.io
QingStor

Rackspace Cloud Files

rsync.net
Scaleway
Seafile
SFTP

StackPath

SugarSync

Tardigrade

Tencent Cloud Object Storage (COS)

Wasabi

WebDAV

Yandex Disk
The local filesystem


Requirements for proposal:

NamePlatform/Container (Microservices)CPU/Memory
RcloneAWS/Linux/Ubuntu/DockerVirtual Appliance

 

SoftwareVersion
RcloneLatest

 

Step in Microservice (Docker):

Dockerfile

FROM alpine:latest

# Define environment variables

ARG RCLONE_VERSION=current

ARG ARCH=amd64

 

RUN apk –no-cache add \

  ca-certificates \

  curl \

  fuse \

  unzip

 

RUN cd /tmp && \

  wget https://downloads.rclone.org/rclone-${RCLONE_VERSION}-linux-${ARCH}.zip && \

  unzip /tmp/rclone-${RCLONE_VERSION}-linux-${ARCH}.zip && \

  mv -v /tmp/rclone-*-linux-${ARCH}/rclone /usr/bin && \

  rm -r /tmp/rclone* && \

  apk del \

  curl \

  unzip

 

# Define mountable directories.

VOLUME [“/config”]

VOLUME [“/media”]

 

EXPOSE 5572/tcp

 

ENTRYPOINT [“sh”, “-c”, “/usr/bin/rclone rcd –rc-web-gui –config=/config/rclone.conf –rc-user=uma148 –rc-pass=password –rc-addr=0.0.0.0:557                        2 –rc-serve”]

Installation Step in VisualHost:

sudo apt update && sudo apt upgrade -y

 

Now, install Rclone

sudo apt install rclone

sudo snap install rclone

rclone config

At first, we have to make a new “remote”. Enter “n”.

Now, provide a name for the new connection. In my case, it’s going to be “myGoogleDrive”.

It’s time to select which service you want to use. In our case, we need Google Drive. Enter “7”.

As of the “client_id” and “client_secret”, leave them blank.

Next up, enter “y” to use auto-configuration.

The tool will pop up the default browser and ask you for logging into your account. Log into your Google account.

 

When asked, click “Allow”.

At this point, you can go back to the Rclone console. Enter “y”.

 

 

Written by: Uma Shankar Pradhan

Uma Shankar has 9+ years of experience in EC2, VPC, S3, Glacier, IAM, SES, SNS, Cloud Watch, EBS, EFS, Elastic Load Balancer, Auto Scaling, RDS, Dynamo DB, AWS CLI, Cloud Front, Cloud Formation, Cloud watch, Route53, Amazon Aurora & AWS. He is an integral member of the Muvi DevOps team.

Add your comment

Leave a Reply

Your email address will not be published.

Try Muvi One Free
For 14 Days

No Credit Card Required

Free Trial