Skip to main content

Deploy bucketeer in Google Cloud, high level idea and infrastructure components

· 3 min read
Hien Vu Ngoc
Hien Vu Ngoc
Software Engineer

Recently, we have released the Bucketeer Lite to easily run Bucketeer anywhere with Docker Compose. However, the standard Bucketeer provides a more robust and scalable solution for more demanding purposes, e.g. withstand high traffic, large data volume, multi-region deployment, etc.

In this blog post, we will share the idea how we deploy Bucketeer in Google Cloud using Terraform, Helm and PipeCD. Terraform scripts will be shared in the future.

Structure​

bucketeer-gcp-structure

This is the general structure of our infrastructure in Google Cloud and also represents the Terraform modules that we will share in the future. Let's break down each component!

VPC Network and DNS​

We create a VPC network with public and private networks. Cloud DNS managed zone is used for public DNS to allow access to Bucketeer web and API services from the internet.

Private DNS is used to create communication between Redis and Cloud SQL with the Bucketeer services in the GKE cluster.

Google Kubernetes Engine (GKE)​

GKE cluster is where Bucketeer services run, linked to the VPC network with subnet configured. For more information about Bucketeer Kubernetes configuration and helm chart, we can check in the Bucketeer repository

Secrets for TLS certificate, OAUTH keys, Service Token, Go Server SDK API Key are also created and stored in here.

Redis and Cloud SQL​

Google Redis MemoryStore is deployed as instance or cluster, depending on the environment (development or production) or traffic needs. We use private IP to connect to the GKE cluster.

MySQL instance is deployed for Bucketeer to store its data. Private IP is also used to connect to the GKE cluster.

Google Pub/Sub and Google BigQuery​

Google Pub/Sub infrastructure is maintained by Google, we just need to create topics for Bucketeer services to use.

Google BigQuery infrastructure is also maintained by Google, data table is created for Bucketeer to store analytics data.

Deploy Bucketeer​

After setting up the infrastructure and prepare the secrets, we can deploy Bucketeer using Helm. In this step, we need to replace values in the Helm chart to match our infrastructure configurations.

To automate the deployment, we install piped in the GKE cluster to deploy Bucketeer. To learn more about how to use PipeCD, please check the PipeCD documentation.

Conclusion​

That's the high-level idea and components of how Bucketeer is deployed in Google Cloud, it's more complex than Bucketeer Lite but provides a more robust and scalable solution.

We will share the Terraform scripts, detailed steps to generate certificates, setup secrets, piped configuration, ingress, ... as well as some notes and mistake we can learn from in the future. stay tuned!