Subpages
- Certification
- Deploying a Landing Zone on Google Cloud
- Google Cloud CICD
- Google Cloud Code
- Google Cloud Deploy
- Google Cloud Developer Guide
- Google Cloud Onboarding
- Google Cloud Run
- New Google Cloud Domain and Account
- Serverless Websites on Google Cloud
This blog/wiki is not an officially supported Google product
Todo:
review CFT https://cloud.google.com/foundation-toolkit and checkout the bootstrap section for organization onboarding https://github.com/terraform-google-modules/terraform-example-foundation/tree/master/0-bootstrap
review Deployment Manager templates https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/tree/master/dm/templates as a precursor to Cloud Deploy
enable the GKE vertical pod autoscaler https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler and https://cloud.google.com/kubernetes-engine/docs/concepts/verticalpodautoscaler
https://cloud.google.com/stackdriver/docs/solutions/gke/observing
https://cloud.google.com/blog/topics/inside-google-cloud/whats-new-google-cloud
Community
Submitting Tutorials
https://cloud.google.com/community/tutorials/write
example https://cloud.google.com/community/tutorials/managing-gcp-projects-with-terraform
Google Cloud
Google Cloud Finops and Zero cost cloud at FinOps#GoogleCloudFinOps
Google Cloud Operations
Migrating to Google Cloud from AWS
GCP Identity Aware Proxy and AWS SSM Systems Manager
Nov 2021 https://cloud.google.com/free/docs/aws-azure-gcp-service-comparison| https://cloud.google.com/free/docs/what-makes-google-cloud-platform-different
Service | GCP | AWS |
---|---|---|
private access to services | Private Service Connect | AWS VPC Endpoints |
CI/CD | Cloud Build | AWS CodeBuild, AWS CodeDeploy, AWS CodePipeline |
Multi-cloud | Anthos | AWS Outposts |
Multi-cloud | Anthos attached clusters | |
Multi-cloud | Anthos on bare metal | |
Multi-cloud | Anthos clusters on AWS | |
Multi-cloud | Anthos clusters on VMware | |
Multi-cloud | Anthos Config Management | Chef Automate AWS OpsWorks |
Multi-cloud | Config Connector | AWS Controllers for Kubernetes |
Multi-cloud | Container-Optimized OS | |
Multi-cloud | Hybrid Connectivity | AWS Direct Connect |
Service mesh | Anthos Service Mesh | AWS App Mesh |
Service mesh | Cloud Router | Amazon VPC Transit Gateway |
Service mesh | Istio on Google Kubernetes Engine | Istio on Amazon EKS |
Service mesh | Traffic Director | AWS App Mesh |
Core compute | Cloud GPUs | Amazon Elastic Compute Cloud (EC2) P3 |
Core compute | Compute Engine | Amazon Elastic Compute Cloud (EC2) |
Core compute | Compute Engine Autoscaler | AWS Autoscaling |
Core compute | OS Login | Amazon EC2 Instance Connect |
Core compute | Persistent Disk | Amazon Elastic Block Store (EBS) |
Core compute | SSH from the browser | AWS EC2 Instance Connect |
Dedicated VMs | Sole-tenant nodes | Amazon EC2 Dedicated Host |
Infrastructure modernization | SAP on Google Cloud | SAP on AWS |
FaaS | Cloud Functions | AWS Lambda |
PaaS | App Engine | AWS Elastic Beanstalk |
VMware connectivity | VMware Engine | VMware Cloud on AWS |
CaaS | Google Kubernetes Engine | Amazon Elastic Kubernetes Service (EKS), Amazon Elastic Container Service (ECS) |
Container registry | Artifact Registry | Amazon Elastic Container Registry (ECR) |
Container Security | Binary Authorization | |
Containers without infrastructure | Cloud Run | AWS Fargate, AWS Lambda, AWS App Runner |
Business intelligence | Looker | Amazon QuickSight |
Data discovery and metadata management | Data Catalog | AWS Glue Data Catalog |
Data Integration / ETL | Cloud Data Fusion | Amazon AppFlow, Amazon Data Pipeline, AWS Glue |
Data warehouse | BigQuery | Amazon Athena, Amazon Redshift |
Messaging | Pub/Sub | Amazon Simple Notification Service (SNS), Amazon Simple Queueing Service (SQS) |
Messaging | Pub/Sub Lite | Amazon Simple Notification Service, Amazon Simple Queueing Service |
Open source processing | Dataproc | Amazon Elastic MapReduce (EMR), AWS Batch, AWS Glue |
Query service | BigQuery | Amazon Redshift Spectrum |
Stream data ingest | Pub/Sub | Amazon Kinesis |
Stream data processing | Dataflow | Amazon Kinesis |
Workflow orchestration | Cloud Composer | Amazon Data Pipeline, AWS Glue, Managed Workflows for Apache Air |
Document data storage | Firestore | Amazon DocumentDB, AWS DynamoDB, AWS AppSync |
In-memory data store | Memorystore | Amazon ElastiCache |
NoSQL: Indexed | Datastore | Amazon DynamoDB |
NoSQL: Key-value | Cloud Bigtable | Amazon DynamoDB |
RDBMS | Cloud Spanner | Amazon Aurora |
RDBMS | Cloud SQL | Amazon Relational Database Service (RDS), Amazon Aurora |
Relational | Bare Metal Solution | Amazon RDS for Oracle |
Client libraries | Cloud SDK | AWS SDKs |
Cloud development IDE plugin | Cloud Code for IntelliJ | AWS Toolkit for IntelliJ |
Cloud development IDE plugin | Cloud Code for VS Code | AWS Toolkit for Visual Studio Code |
Cloud-based IDE | Cloud Shell | AWS CloudShell |
Command-line interface (CLI) | Cloud SDK | AWS CLI |
Marketplace | Marketplace | AWS Marketplace |
IoT platform | Cloud IoT | AWS IoT Core |
Cloud cost optimization | Recommender | AWS Cost Optimization |
Conversational interface | Dialogflow | Amazon Lex |
ML for structured data | Vertex AI AutoML tabular models | Amazon SageMaker |
ML platform | Vertex AI custom-trained models | Amazon SageMaker |
ML platform | Vertex AI custom training | Amazon SageMaker |
ML platform | Vertex AI AutoML models | Amazon SageMaker Autopilot |
ML platform | Vertex AI | Amazon SageMaker |
ML platform | Deep Learning VM Images | Amazon SageMaker, Amazon EC2 P3 |
ML platform | Vertex AI Workbench | Amazon SageMaker |
ML platform | TensorFlow Enterprise | Tensorflow on AWS |
Natural language processing | Natural Language AI | Amazon Comprehend |
Personalization | Recommendations AI | Amazon Personalize |
Translation | Translation AI | Amazon Translate |
Video intelligence | Video Intelligence API | Amazon Rekognition Video |
Vision: Read and extract text | Vision AI | Amazon Textract |
Vision: Speech-to-text | Speech-to-Text | Amazon Transcribe |
API management | Apigee API Management | Amazon API Gateway |
Cost management | Cost Management | AWS Budgets |
Deployment | Cloud Deployment Manager | AWS CloudFormation |
AI | Video AI | Amazon Rekognition Video |
Encoding | Transcoder API | AWS Media Converter |
Streaming | Video Intelligence Streaming API | Live Streaming on AWS |
Container migration | Migrate for Anthos | AWS App2Container |
Server migration | Migrate for Compute Engine | AWS Server Migration Service |
SQL database migration | Database Migration Service | AWS Database Migration Service |
Storage migration | Storage Transfer Service | AWS Storage Gateway |
Storage migration | Transfer Appliance | AWS Snowball |
CDN | Cloud CDN | Amazon CloudFront |
DDoS firewall | Google Cloud Armor Managed Protection | AWS Shield Basic/Advanced |
Dedicated Interconnect connection | Cloud Interconnect | AWS Direct Connect |
Domains and DNS | Cloud DNS | Amazon Route 53 |
Domains and DNS | Cloud Domains | Amazon Route 53 |
Load balancer | Cloud Load Balancing | AWS Elastic Load Balancing |
Network monitoring | Network Intelligence Center | |
Network monitoring | VPC Flow Logs | Amazon VPC Flow Logs |
Network security | Cloud VPN | AWS Virtual Private Network (VPN) |
Premium networking | Network Service Tiers | |
Service mesh | Traffic Director | AWS App Mesh |
Services | Service Directory | AWS Cloud Map |
Virtual networks | Cloud NAT | Amazon VPC NAT instances |
Virtual networks | Virtual Private Cloud | Amazon Virtual Private Cloud (VPC) |
Web application firewall | Google Cloud Armor | AWS WAF |
Audit logging | Cloud Audit Logs | AWS CloudTrail |
Debugging | Cloud Debugger | AWS X-Ray |
Logging | Cloud Logging | Amazon CloudWatch Logs |
Monitoring | Cloud Monitoring | Amazon CloudWatch |
Performance tracing | Cloud Trace | AWS X-Ray |
Profiling | Cloud Profiler | Amazon CodeGuru Profiler |
Certificate management | Certificate Authority Service | AWS Certificate Manager |
CIAM | Identity Platform | Amazon Cognito |
Container security | Artifact Registry | Amazon Elastic Container Registry (ECR) |
Container security | Container Analysis | Amazon ECR Image Scanning |
Container security | Container Security | Security in Amazon Elastic Container Service (ECS) |
Container security | GKE Sandbox | Amazon EKS Container Sandbox |
Data loss prevention (DLP) | Cloud Data Loss Prevention | Amazon Macie |
Encryption | Confidential Computing | AWS Nitro Enclaves |
Exfiltration prevention | VPC Service Controls | |
Hardware security module (HSM) | Cloud HSM | AWS CloudHSM |
IAM | Cloud Identity | AWS Identity Services |
IAM | Identity and Access Management | Amazon Identity and Access Management |
IAM | Managed Service for Microsoft Active Directory | AWS Managed Microsoft AD |
Resource monitoring | Cloud Asset Inventory | AWS Config |
Resource monitoring | Resource Manager | AWS OpsWorks |
Secret management | Secret Manager | AWS Secrets Manager |
Security administration | Cloud Key Management Service | AWS Key Management Service (KMS) |
Security and risk management | Security Command Center | Amazon Guard Duty, AWS Security Hub |
Zero trust | BeyondCorp Enterprise | |
Build | Cloud Storage for Firebase | AWS Simple Storage Service (S3) |
Build | Firebase Auth | Amazon Cognito |
Build | Firebase Hosting | AWS Simple Storage Service (S3) |
Build | Firebase Realtime Database | AWS DynamoDB + AppSync |
Engage | Firebase A/B Testing | Amazon Pinpoint |
Engage | Firebase Cloud Messaging | Amazon Device Messaging (ADM), Amazon Simple Notification Service (SNS) |
Engage | Firebase Dynamic Links | |
Engage | Firebase In-App Messaging | Amazon Device Messaging (ADM), Amazon Simple Notification Service (SNS) |
Engage | Firebase Remote Config | |
Engage | Google Analytics | |
Event handling | CNCF CloudEvents integrated | AWS EventBridge |
Kubernetes platform | Cloud Run CNCF CloudEvents integrated | AWS Fargate |
Release & monitor | Firebase App Distribution | |
Release & monitor | Firebase Crashlytics | |
Release & monitor | Firebase Performance Monitoring | |
Release & monitor | Firebase Test Lab | AWS Device Farm |
Workflow orchestration | Workflows | AWS Step Functions |
Block storage | Persistent Disk | Amazon Elastic Block Store (EBS) |
File storage | Filestore | Amazon Elastic File System (EFS) |
Infrequently accessed object storage | Cloud Storage Archive | Amazon S3 Glacier |
Object storage | Cloud Storage | AWS Simple Storage Service (S3) |
Terraform on Google Cloud
https://learn.hashicorp.com/collections/terraform/gcp-get-started
Terraform Admin folder and service account via https://cloud.google.com/community/tutorials/managing-gcp-projects-with-terraform
Creating Service Accounts
https://cloud.google.com/iam/docs/creating-managing-service-accounts
admin_root@cloudshell:~ (biometric-ncorg)$ gcloud iam service-accounts create terraform-sa --description="terraform-sa" --display-name="terraform-sa" Created service account [terraform-sa]. admin_root@cloudshell:~ (biometric-ncorg)$ gcloud iam service-accounts list DISPLAY NAME: terraform-sa EMAIL: terraform-sa@biometric-ncorg.iam.gserviceaccount.com DISABLED: False
wait 60 seconds for the service account to provision - use an exponential backoff.
Add roles to the Service Account
https://cloud.google.com/iam/docs/granting-changing-revoking-access
Get roles for a particular project first
admin_root@cloudshell:~ (biometric-ncorg)$ gcloud projects get-iam-policy biometric-ncorg
bindings:
- members:
- user:admin-root nuage-cloud.org
role: roles/owner
https://cloud.google.com/iam/docs/granting-changing-revoking-access#grant-single-role
Get the organization id from the project ancestor
admin_root@cloudshell:~ (biometric-ncorg)$ ORG_ID=$(gcloud projects get-ancestors biometric-ncorg --format='get(id)' | tail -1)
admin_root@cloudshell:~ (biometric-ncorg)$ echo $ORG_ID
4719....
Add a role at the organization level to a service account
admin_root@cloudshell:~ (biometric-ncorg)$ gcloud organizations add-iam-policy-binding $ORG_ID --member=serviceAccount:terraform-sa@biometric-ncorg.iam.gserviceaccount.com --role=roles/resourcemanager.projectCreator
Updated IAM policy for organization [47....].
bindings:
- members:
- domain:nuage-cloud.org
- serviceAccount:terraform-sa@biometric-ncorg.iam.gserviceaccount.com
role: roles/resourcemanager.projectCreator
Add a role at the organization level to a super admin account
Impersonating Service Accounts
Terraform can impersonate a Google Cloud Service Account. The user/caller account must have the roles/iam.serviceAccountTokenCreator role set.
see https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference and https://cloud.google.com/iam/docs/create-short-lived-credentials-direct#gcloud with https://cloud.google.com/iam/docs/creating-managing-service-accounts
admin_root@cloudshell:~ (biometric-ncorg)$ SUPER_ADMIN_EMAIL=admin-root@nuag...g admin_root@cloudshell:~ (biometric-ncorg)$ gcloud organizations add-iam-policy-binding $ORG_ID --member=user:$SUPER_ADMIN_EMAIL --role=roles/iam.serviceAccountTokenCreator Updated IAM policy for organization [47...]. bindings: - members: - user:admin-root@nuage-cloud.org role: roles/iam.serviceAccountTokenCreator
Before
After
Notice the Service Account Token Creator - role
Install gcloud cli
https://github.com/cloud-quickstart/wiki#installing-the-google-cloud-sdk-on-osx
Terraform is provisioned with Google Cloud Shell
Launch with http://shell.cloud.google.com
Use an example https://registry.terraform.io/modules/terraform-google-modules/kubernetes-engine/google/latest
https://www.hashicorp.com/blog/kickstart-terraform-on-gcp-with-google-cloud-shell
https://registry.terraform.io/providers/hashicorp/google/latest/docs
Welcome to Cloud Shell! Type "help" to get started. To set your Cloud Platform project in this session use “gcloud config set project [PROJECT_ID]” cloudshell_open --repo_url "https://github.com/terraform-google-modules/docs-examples.git" --print_file "./motd" --dir "address_basic" --page "editor" --tutorial "./tutorial.md" --open_in_editor "main.tf" --force_new_clone michael@cloudshell:~$ cloudshell_open --repo_url "https://github.com/terraform-google-modules/docs-examples.git" --print_file "./motd" --dir "address_basic" --page "editor" --tutorial "./tutorial.md" --open_in_editor "main.tf" --force_new_clone 2021/12/16 03:08:42 Cloning https://github.com/terraform-google-modules/docs-examples.git into /home/michael/cloudshell_open/docs-examples Cloning into '/home/michael/cloudshell_open/docs-examples'... remote: Enumerating objects: 1906, done. remote: Total 1906 (delta 0), reused 0 (delta 0), pack-reused 1906 Receiving objects: 100% (1906/1906), 447.94 KiB | 7.46 MiB/s, done. Resolving deltas: 100% (1442/1442), done. 2021/12/16 03:08:43 === gcloud auth list gcloud config list project michael@cloudshell:~/cloudshell_open/docs-examples/address_basic$ ls backing_file.tf main.tf motd tutorial.md michael@cloudshell:~/cloudshell_open/docs-examples/address_basic$ export GOOGLE_CLOUD_PROJECT=dev-sphere-335220 michael@cloudshell:~/cloudshell_open/docs-examples/address_basic$ terraform init Initializing the backend... Initializing provider plugins... - Finding latest version of hashicorp/google... - Finding latest version of hashicorp/random... - Installing hashicorp/google v4.4.0... - Installed hashicorp/google v4.4.0 (self-signed, key ID 34365D9472D7468F) - Installing hashicorp/random v3.1.0... - Installed hashicorp/random v3.1.0 (self-signed, key ID 34365D9472D7468F) Partner and community providers are signed by their developers. If you'd like to know more about provider signing, you can read about it here: https://www.terraform.io/docs/cli/plugins/signing.html Terraform has created a lock file .terraform.lock.hcl to record the provider selections it made above. Include this file in your version control repository so that Terraform can guarantee to make the same selections by default when you run "terraform init" in the future. Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. michael@cloudshell:~/cloudshell_open/docs-examples/address_basic$ michael@cloudshell:~/cloudshell_open/docs-examples/address_basic$ terraform apply Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the followingsymbols: + create Terraform will perform the following actions: # google_compute_address.ip_address will be created + resource "google_compute_address" "ip_address" { + address = (known after apply) + address_type = "EXTERNAL" + creation_timestamp = (known after apply) + id = (known after apply) + name = (known after apply) + network_tier = (known after apply) + project = (known after apply) + purpose = (known after apply) + region = (known after apply) + self_link = (known after apply) + subnetwork = (known after apply) + users = (known after apply) } # random_pet.suffix will be created + resource "random_pet" "suffix" { + id = (known after apply) + length = 2 + separator = "-" } Plan: 2 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes random_pet.suffix: Creating... random_pet.suffix: Creation complete after 0s [id=one-doberman] google_compute_address.ip_address: Creating... ╷ │ Error: Error creating Address: googleapi: Error 403: Compute Engine API has not been used in project 313394869326 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/compute.googleapis.com/overview?project=313394869326 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry. │ Details: │ [ │ { │ "@type": "type.googleapis.com/google.rpc.Help", │ "links": [ │ { │ "description": "Google developers console API activation", │ "url": "https://console.developers.google.com/apis/api/compute.googleapis.com/overview?project=313394869326" │ } │ ] │ }, │ { │ "@type": "type.googleapis.com/google.rpc.ErrorInfo", │ "domain": "googleapis.com", │ "metadata": { │ "consumer": "projects/313394869326", │ "service": "compute.googleapis.com" │ }, │ "reason": "SERVICE_DISABLED" │ } │ ] │ , accessNotConfigured │ │ on main.tf line 1, in resource "google_compute_address" "ip_address": │ 1: resource "google_compute_address" "ip_address" { Forgot to enable compute - run... michael@cloudshell:~/cloudshell_open/docs-examples/address_basic$ gcloud config set project dev-sphere-335220 Updated property [core/project]. michael@cloudshell:~/cloudshell_open/docs-examples/address_basic (dev-sphere-335220)$ gcloud services enable compute.googleapis.com terraform apply google_compute_address.ip_address: Creating... google_compute_address.ip_address: Still creating... [10s elapsed] google_compute_address.ip_address: Creation complete after 13s [id=projects/dev-sphere-335220/regions/us-central1/addresses/my-address-one-doberman] Apply complete! Resources: 1 added, 0 changed, 0 destroyed. michael@cloudshell:~/cloudshell_open/docs-examples/address_basic (dev-sphere-335220)$ cat terraform.tfstate { "version": 4, "terraform_version": "0.15.0", "serial": 9, "lineage": "15b70e84-53ce-6cf7-3518-ed294a4181d2", "outputs": {}, "resources": [ { "mode": "managed", "type": "google_compute_address", "name": "ip_address", "provider": "provider[\"registry.terraform.io/hashicorp/google\"]", "instances": [ { "schema_version": 0, "attributes": { "address": "34.70.49.6", "address_type": "EXTERNAL", "creation_timestamp": "2021-12-15T19:40:55.884-08:00", "description": "", "id": "projects/dev-sphere-335220/regions/us-central1/addresses/my-address-ample-spaniel", "name": "my-address-ample-spaniel", "network": "", "network_tier": "PREMIUM", "prefix_length": 0, "project": "dev-sphere-335220", "purpose": "", "region": "us-central1", "self_link": "https://www.googleapis.com/compute/v1/projects/dev-sphere-335220/regions/us-central1/addresses/my-address-ample-spaniel", "subnetwork": "", "timeouts": null, "users": [] }, "sensitive_attributes": [], "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoyNDAwMDAwMDAwMDAsImRlbGV0ZSI6MjQwMDAwMDAwMDAwfX0=", "dependencies": [ "random_pet.suffix" ] } ] }, { "mode": "managed", "type": "random_pet", "name": "suffix", "provider": "provider[\"registry.terraform.io/hashicorp/random\"]", "instances": [ { "schema_version": 0, "attributes": { "id": "ample-spaniel", "keepers": null, "length": 2, "prefix": null, "separator": "-" }, "sensitive_attributes": [], "private": "bnVsbA==" } ] } ] } terraform destroy google_compute_address.ip_address: Destroying... [id=projects/dev-sphere-335220/regions/us-central1/addresses/my-address-one-doberman] google_compute_address.ip_address: Still destroying... [id=projects/dev-sphere-335220/regions/us-central1/addresses/my-address-one-doberman, 10s elapsed] google_compute_address.ip_address: Destruction complete after 11s random_pet.suffix: Destroying... [id=one-doberman] random_pet.suffix: Destruction complete after 0s Destroy complete! Resources: 2 destroyed. michael@cloudshell:~/cloudshell_open/docs-examples/address_basic (dev-sphere-335220)$ cat terraform.tfstate { "version": 4, "terraform_version": "0.15.0", "serial": 6, "lineage": "15b70e84-53ce-6cf7-3518-ed294a4181d2", "outputs": {}, "resources": [] }
Terraform Example Network
From https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall and https://www.qwiklabs.com/focuses/4375?parent=catalog
# create resource "google_compute_network" "mynetwork" { name = "mynetwork" # resources auto_create_subnetworks = true } # comment out the following # on mynetwork resource "google_compute_firewall" "mynetwork-allow-http-ssh-rdp-icmp" { name = "mynetwork-allow-http-ssh-rdp-icmp" network = google_compute_network.mynetwork.self_link allow { protocol = "tcp" ports = ["22", "80", "3389"] } allow { protocol = "icmp" } source_ranges = ["0.0.0.0/0"] } terraform apply google_compute_network.mynetwork: Creating... google_compute_network.mynetwork: Still creating... [10s elapsed] google_compute_network.mynetwork: Still creating... [20s elapsed] google_compute_network.mynetwork: Still creating... [30s elapsed] google_compute_network.mynetwork: Creation complete after 32s [id=projects/..../global/networks/mynetwork] google_compute_firewall.mynetwork-allow-http-ssh-rdp-icmp: Creating... google_compute_firewall.mynetwork-allow-http-ssh-rdp-icmp: Still creating... [10s elapsed] google_compute_firewall.mynetwork-allow-http-ssh-rdp-icmp: Creation complete after 12s [id=projects/..../global/firewalls/mynetwork-allow-http-ssh-rdp-icmp]
Install Terraform
Install Terraform on Windows
Download 1.1.0 via https://releases.hashicorp.com/terraform/1.1.0/terraform_1.1.0_windows_amd64.zip
Copy the extracted terraform.exe into a dir that is already in your PATH.
$ terraform -version Terraform v1.1.0 on windows_amd64
Install Terraform on OSX
Update Terraform on OSX
terraform --version Terraform v1.0.11 on darwin_amd64 Your version of Terraform is out of date! The latest version is 1.1.3. You can update by downloading from https://www.terraform.io/downloads.html brew tap hashicorp/tap brew install hashicorp/tap/terraform ==> Downloading https://releases.hashicorp.com/terraform/1.1.4/terraform_1.1.4_darwin_amd64.zip ######################################################################## 100.0% ==> Installing terraform from hashicorp/tap Error: Your Command Line Tools are too outdated. Update them from Software Update in System Preferences or run: softwareupdate --all --install --force If that doesn't show you any updates, run: sudo rm -rf /Library/Developer/CommandLineTools sudo xcode-select --install Alternatively, manually download them from: https://developer.apple.com/download/all/. You should download the Command Line Tools for Xcode 13.1. softwareupdate --all --install --force Software Update Tool Finding available software No updates are available.
https://developer.apple.com/download/all/
Continue terraform upgrade
biometric:wse_github michaelobrien$ brew install hashicorp/tap/terraform Running `brew update --preinstall`... ==> Downloading https://releases.hashicorp.com/terraform/1.1.4/terraform_1.1.4_darwin_amd64.zip Already downloaded: /Users/michaelobrien/Library/Caches/Homebrew/downloads/42cd50722c72dfb1aca649a75041a885b0885c39fb6aa6e636c87fb38a16e5d5--terraform_1.1.4_darwin_amd64.zip ==> Installing terraform from hashicorp/tap Error: The `brew link` step did not complete successfully The formula built, but is not symlinked into /usr/local Could not symlink bin/terraform Target /usr/local/bin/terraform already exists. You may want to remove it: rm '/usr/local/bin/terraform' To force the link and overwrite all conflicting files: brew link --overwrite terraform biometric:wse_github michaelobrien$ terraform --version Terraform v1.0.11 on darwin_amd64 Your version of Terraform is out of date! The latest version is 1.1.3. You can update by downloading from https://www.terraform.io/downloads.html biometric:wse_github michaelobrien$ brew link --overwrite terraform Linking /usr/local/Cellar/terraform/1.1.4... 1 symlinks created. biometric:wse_github michaelobrien$ brew reinstall hashicorp/tap/terraform ==> Downloading https://releases.hashicorp.com/terraform/1.1.4/terraform_1.1.4_darwin_amd64.zip Already downloaded: /Users/michaelobrien/Library/Caches/Homebrew/downloads/42cd50722c72dfb1aca649a75041a885b0885c39fb6aa6e636c87fb38a16e5d5--terraform_1.1.4_darwin_amd64.zip ==> Reinstalling hashicorp/tap/terraform biometric:wse_github michaelobrien$ terraform --version Terraform v1.1.4 on darwin_amd64
Deploy Google Cloud Infrastructure using Terraform
https://learn.hashicorp.com/tutorials/terraform/google-cloud-platform-build
Create GCP Project
https://cloud.google.com/sdk/gcloud/reference/projects/create
gcloud projects create refarch-obrien --name="refacrh" --labels=type=dev Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/refarch-obrien]. Waiting for [operations/cp.9137809687161597566] to finish...done. Enabling service [cloudapis.googleapis.com] on project [refarch-obrien]... Operation "operations/acf.p2-535552712251-008967b0-7fd8-43a1-b748-35bc886090c2" finished successfully.
Enable GCE
https://cloud.google.com/sdk/gcloud/reference/services/enable
gcloud services enable compute
GCP Service Account
GCloud Shell
Gcloud shell is an up to 1 hour provisioned debian VM running a single docker container with a persistent volume that completes in 25 sec - https://cloud.google.com/shell/docs/how-cloud-shell-works customize the shell via https://cloud.google.com/shell/docs/configuring-cloud-shell#environment_customization with persistent addons including
bq 2.0.71
java 11.0.2,
terraform 1.1.2,
go 1.17.2
python 2.7.16
python3 3.7.1
mysql client,
tensorflow,
git 2.2.0,
jq 1.5.1,
make 4.2.1,
helm 3.5.0,
kubectl 1.22.3,
mvn 3.6.3
Costs
GKE 4 Node Cluster Costs
The cost for a 4 node GKE cluster has 2 components - the underlying GCE VMs and the GKE Kubernetes service. Daily cost starts at US 3.22 to a week later at 4.12 (0.17/hr) - of which 40% was the GKE service - which is fully discounted (you only pay for the infrastructure - not the control plane)
Google Workspace
Change Workspace Domain and Google Cloud Organization
https://domains.google.com/registrar/obrienlabs.dev#
Workspace checklist - https://cloud.google.com/docs/enterprise/setup-checklist?utm_source=google&utm_medium=email&utm_content=email&utm_campaign=Enterprise-Checklist-Reminder-Email
Premium Tier Organization
In order to get access to Security Command Center https://cloud.google.com/security-command-center/docs/quickstart-security-command-center you will need Premium Tier access https://cloud.google.com/security-command-center/pricing#premium-tier
Standard tier is free but premium tier is 5% of cloud spend with a base of 25k
Operations
List enabled services
gcloud services list --enabled --project dev
List available services - may run into API quota
gcloud services list --available | grep run
Enabling Services
gcloud services enable \ compute.googleapis.com \ iam.googleapis.com \ iamcredentials.googleapis.com \ monitoring.googleapis.com \ logging.googleapis.com \ notebooks.googleapis.com \ aiplatform.googleapis.com \ bigquery.googleapis.com \ artifactregistry.googleapis.com \ cloudbuild.googleapis.com \ container.googleapis.com
Setting default region
Recently one of my organizations ran into an issue where I could only select from a single recent default region.
Resetting the default region
michael@cloudshell:~ (magellan-sbx)$ gcloud config get-value compute/region Your active configuration is: [cloudshell-30108] (unset) michael@cloudshell:~ (magellan-sbx)$ gcloud compute regions list API [compute.googleapis.com] not enabled on project [29..42]. Would you like to enable and retry (this will take a few minutes)? (y/N)? y Enabling service [compute.googleapis.com] on project [298..042]... Operation "operations/acf.p2-298632905042-4d244f1a-0538-4d0e-8f44-c557937befa9" finished successfully. NAME: asia-east1 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: asia-east2 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: asia-northeast1 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: asia-northeast2 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: asia-northeast3 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: asia-south1 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: asia-south2 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: asia-southeast1 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: asia-southeast2 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: australia-southeast1 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: australia-southeast2 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: europe-central2 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: europe-north1 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: europe-west1 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: europe-west2 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: europe-west3 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: europe-west4 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: europe-west6 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: northamerica-northeast1 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: northamerica-northeast2 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: southamerica-east1 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: southamerica-west1 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: us-central1 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: us-east1 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: us-east4 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: us-west1 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: us-west2 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: us-west3 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: NAME: us-west4 CPUS: 0/24 DISKS_GB: 0/4096 ADDRESSES: 0/8 RESERVED_ADDRESSES: 0/8 STATUS: UP TURNDOWN_DATE: michael@cloudshell:~ (magellan-sbx)$ gcloud compute project-info add-metadata \ > --metadata google-compute-default-region=us-east4,google-compute-default-zone=us-east4-a Updated [https://www.googleapis.com/compute/v1/projects/magellan-sbx] gcloud init Your current project has been set to: [magellan-sbx]. Your project default Compute Engine zone has been set to [us-east4-a]. You can change it by running [gcloud config set compute/zone NAME]. Your project default Compute Engine region has been set to [us-east4]. You can change it by running [gcloud config set compute/region NAME]. Created a default .boto configuration file at [/home/michael/.boto]. See this file and [https://cloud.google.com/storage/docs/gsutil/commands/config] for more information about configuring Google Cloud Storage. Your Google Cloud SDK is configured and ready to use! * Commands will reference project `magellan-sbx` by default * Compute Engine commands will use region `us-east4` by default * Compute Engine commands will use zone `us-east4-a` by default Run `gcloud help config` to learn how to change individual settings This gcloud configuration is called [cloudshell-30108]. You can create additional configurations if you work with multiple accounts and/or projects. Run `gcloud topic configurations` to learn more. Some things to try next: * Run `gcloud --help` to see the Cloud Platform services you can interact with. And run `gcloud help COMMAND` to get help on any gcloud command. * Run `gcloud topic --help` to learn about advanced features of the SDK like arg files and output formatting michael@cloudshell:~ (magellan-sbx)$ gcloud config get-value compute/region Your active configuration is: [cloudshell-30108] us-east4
Google Cloud Titan Key
http://g.co/titansecuritykey/help
Jiras
20220305 1058 mbp6 pa.gl | org checklist 8 projects saturates billing account association with project quota | https://support.google.com/code/contact/billing_quota_increase | |
need CLI output of quota increase - to be able to automate account onboarding in the future | |||
need billing account id copy to qutoa request form | Billing account IDs for which additional quota is needed * | ||
need actual quota link for above billing account association https://console.cloud.google.com/iam-admin/quotas?referrer=search&project=pg-logging-prod | |||
Some cloud shell accounts list 50 - others 60h per week | |||
The blog page resets the navigation list on "back" after scrolling through "load more blogs" | for example navigate to https://cloud.google.com/blog/topics/developers-practitioners scroll down to a story 2 months ago hit the back button to continue scolling back button resets the list | ||
20220325 | message should not say "different" billing account on association |
Google Cloud Diagrams
The following site is an extension of the 4 words site https://googlecloudcheatsheet.withgoogle.com/
An open source tool https://github.com/excalidraw/excalidraw was extended to generate deployment (code/scripts - determining?) from your Google Cloud Architecture Diagram https://googlecloudcheatsheet.withgoogle.com/architecture by adding the "open in cloud shell" icon https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2FGoogleCloudPlatform%2Fappinabox_basiclb&cloudshell_print=install.txt&shellonly=true
Examples
https://googlecloudcheatsheet.withgoogle.com/architecture?diagram=SimpleComputeApp statically references https://github.com/GoogleCloudPlatform/deploystack_todo via https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2FGoogleCloudPlatform%2Fdeploystack_todo&cloudshell_print=install.txt&shellonly=true
https://googlecloudcheatsheet.withgoogle.com/architecture?diagram=ThreeTierApp statically references https://github.com/GoogleCloudPlatform/deploystack_basiclb via
downloads the following repo to your cloud shell - as per the standard "open in cloud shell" icon you can directly add to your github readme.md
https://github.com/GoogleCloudPlatform/deploystack_basiclb
the install.txt file instructs you to run the ./install sh script
The script asks you for number of VMs
It may be just me but there is no link between the diagramming tool from excalidraw and the github hosted projects - the deploy buttons in the 5 examples are static links
The diagram does not actually generate a deployment template in go. There is only a link to a static go example project
The issue seems to be that there is no correspondence between the diagram an the deployment - there is only a link to static github sh gcloud deploy commands
Change the diagram and there is no link to the deployment icon (no code or parameter passing) - just a link.
For example, in the diagram there are 2 GCE instances - the deployment script asks for number of instances - change or delete the diagram and the deploy link will work as directed as it is just a link to the existing repo.
TODO INSTALL This process will create the following: * Frontend - Cloud Run Service * Middleware - Cloud Run Service * Backend - Cloud Sql MySQL instance * Cache - Cloud Memorystore * Secrets - Cloud Secret Manager |
I will have a discussion with the blog author
How to save/access diagram persistence
Google Cloud Diagramming Icons has a very good list of presentation/(ppt) templates : https://cloud.google.com/icons
Lucidchart is excellent and exports to visio for compliance
https://lucidscale.com/ at $2600/year will generate GCP diagrams from code - on top of your normal manual lucidchart account
https://www.gliffy.com/blog/gcp-architecture-diagrams | https://www.cloudockit.com/the-12-most-used-google-cloud-diagrams-explained/
Automated | https://www.hava.io/gcp-infrastructure-diagrams
Google Cloud Events
Google Cloud Ground School May 18-19 2022 https://events.sada.com/ground-school/
Links
Google Web Toolkit from 2013 : http://eclipsejpa.blogspot.com/2013/02/google-web-toolkit-and-google-app.html
Android Development from 2011: http://eclipsejpa.blogspot.com/2011/10/android-development.html
borg https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43438.pdf
https://cloud.google.com/compute/docs/instances/connecting-to-windows
https://medium.com/google-cloud/parallel-executions-with-google-workflows-3a16f8fee0eb
https://cloud.google.com/anthos-config-management/docs/tutorials/landing-zone
https://cloud.google.com/blog/products/devops-sre/google-cloud-deploy-now-ga
https://cloud.google.com/foundation-toolkit/
https://cloud.google.com/architecture/estimate-gke-costs-early-using-github
DevOps
While studying for the PCDE
https://sites.google.com/corp/google.com/devops-library/home
https://cloud.google.com/architecture/devops/devops-tech-architecture?hl=en
https://cloud.google.com/devops
The devops-sre blog https://cloud.google.com/blog/products/devops-sre/google-cloud-deploy-now-ga
https://cloud.google.com/docs/ci-cd
Training
https://masteringtechnicalsales.com/
1 Comment
Michael O'Brien
check local wsl for gcloud install