Page tree
Skip to end of metadata
Go to start of metadata

Subpages


This blog/wiki is not an officially supported Google product

Todo:

review CFT https://cloud.google.com/foundation-toolkit and checkout the bootstrap section for organization onboarding https://github.com/terraform-google-modules/terraform-example-foundation/tree/master/0-bootstrap

review Deployment Manager templates https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/tree/master/dm/templates  as a precursor to Cloud Deploy

enable the GKE vertical pod autoscaler https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler and https://cloud.google.com/kubernetes-engine/docs/concepts/verticalpodautoscaler

https://cloud.google.com/stackdriver/docs/solutions/gke/observing

https://cloud.google.com/blog/topics/inside-google-cloud/whats-new-google-cloud


Community

Submitting Tutorials

https://cloud.google.com/community/tutorials/write

example https://cloud.google.com/community/tutorials/managing-gcp-projects-with-terraform

Google Cloud

Google Cloud Finops and Zero cost cloud at FinOps#GoogleCloudFinOps

Google Cloud Operations




Migrating to Google Cloud from AWS

GCP Identity Aware Proxy and AWS SSM Systems Manager


Nov 2021  https://cloud.google.com/free/docs/aws-azure-gcp-service-comparison| https://cloud.google.com/free/docs/what-makes-google-cloud-platform-different

ServiceGCPAWS
private access to servicesPrivate Service ConnectAWS VPC Endpoints
CI/CDCloud BuildAWS CodeBuild,
AWS CodeDeploy,
AWS CodePipeline
Multi-cloudAnthosAWS Outposts
Multi-cloudAnthos attached clusters
Multi-cloudAnthos on bare metal
Multi-cloudAnthos clusters on AWS
Multi-cloudAnthos clusters on VMware
Multi-cloudAnthos Config ManagementChef Automate AWS OpsWorks
Multi-cloudConfig ConnectorAWS Controllers for Kubernetes
Multi-cloudContainer-Optimized OS
Multi-cloudHybrid ConnectivityAWS Direct Connect
Service meshAnthos Service MeshAWS App Mesh
Service meshCloud RouterAmazon VPC
Transit Gateway
Service meshIstio on Google Kubernetes EngineIstio on Amazon EKS
Service meshTraffic DirectorAWS App Mesh
Core computeCloud GPUsAmazon Elastic Compute Cloud (EC2) P3
Core computeCompute EngineAmazon Elastic Compute Cloud (EC2)
Core computeCompute Engine AutoscalerAWS Autoscaling
Core computeOS LoginAmazon EC2 Instance Connect
Core computePersistent DiskAmazon Elastic Block Store (EBS)
Core computeSSH from the browserAWS EC2 Instance Connect
Dedicated VMsSole-tenant nodesAmazon EC2 Dedicated Host
Infrastructure modernizationSAP on Google CloudSAP on AWS
FaaSCloud FunctionsAWS Lambda
PaaSApp EngineAWS Elastic Beanstalk
VMware connectivityVMware EngineVMware Cloud on AWS
CaaSGoogle Kubernetes EngineAmazon Elastic Kubernetes Service (EKS),
Amazon Elastic Container Service (ECS)
Container registryArtifact RegistryAmazon Elastic Container Registry (ECR)
Container SecurityBinary Authorization
Containers without infrastructureCloud RunAWS Fargate, AWS Lambda, AWS App Runner
Business intelligenceLookerAmazon QuickSight
Data discovery and metadata managementData CatalogAWS Glue Data Catalog
Data Integration / ETLCloud Data FusionAmazon AppFlow, Amazon Data Pipeline, AWS Glue
Data warehouseBigQueryAmazon Athena, Amazon Redshift
MessagingPub/SubAmazon Simple Notification Service (SNS),
Amazon Simple Queueing Service (SQS)
MessagingPub/Sub LiteAmazon Simple Notification Service,
Amazon Simple Queueing Service
Open source processingDataprocAmazon Elastic MapReduce (EMR),
AWS Batch,
AWS Glue
Query serviceBigQueryAmazon Redshift Spectrum
Stream data ingestPub/SubAmazon Kinesis
Stream data processingDataflowAmazon Kinesis
Workflow orchestrationCloud ComposerAmazon Data Pipeline, AWS Glue,
Managed Workflows for Apache Air
Document data storageFirestoreAmazon DocumentDB, AWS DynamoDB,
AWS AppSync
In-memory data storeMemorystoreAmazon ElastiCache
NoSQL: IndexedDatastoreAmazon DynamoDB
NoSQL: Key-valueCloud BigtableAmazon DynamoDB
RDBMSCloud SpannerAmazon Aurora
RDBMSCloud SQLAmazon Relational Database Service (RDS),
Amazon Aurora
RelationalBare Metal SolutionAmazon RDS for Oracle
Client librariesCloud SDKAWS SDKs
Cloud development IDE pluginCloud Code for IntelliJAWS Toolkit for IntelliJ
Cloud development IDE pluginCloud Code for VS CodeAWS Toolkit for Visual Studio Code
Cloud-based IDECloud ShellAWS CloudShell
Command-line interface (CLI)Cloud SDKAWS CLI
MarketplaceMarketplaceAWS Marketplace
IoT platformCloud IoTAWS IoT Core
Cloud cost optimizationRecommenderAWS Cost Optimization
Conversational interfaceDialogflowAmazon Lex
ML for structured dataVertex AI AutoML tabular modelsAmazon SageMaker
ML platformVertex AI custom-trained modelsAmazon SageMaker
ML platformVertex AI custom trainingAmazon SageMaker
ML platformVertex AI AutoML modelsAmazon SageMaker Autopilot
ML platformVertex AIAmazon SageMaker
ML platformDeep Learning VM ImagesAmazon SageMaker, Amazon EC2 P3
ML platformVertex AI WorkbenchAmazon SageMaker
ML platformTensorFlow EnterpriseTensorflow on AWS
Natural language processingNatural Language AIAmazon Comprehend
PersonalizationRecommendations AIAmazon Personalize
TranslationTranslation AIAmazon Translate
Video intelligenceVideo Intelligence APIAmazon Rekognition Video
Vision: Read and extract textVision AIAmazon Textract
Vision: Speech-to-textSpeech-to-TextAmazon Transcribe
API managementApigee API ManagementAmazon API Gateway
Cost managementCost ManagementAWS Budgets
DeploymentCloud Deployment ManagerAWS CloudFormation
AIVideo AIAmazon Rekognition Video
EncodingTranscoder APIAWS Media Converter
StreamingVideo Intelligence Streaming APILive Streaming on AWS
Container migrationMigrate for AnthosAWS App2Container
Server migrationMigrate for Compute EngineAWS Server Migration Service
SQL database migrationDatabase Migration ServiceAWS Database Migration Service
Storage migrationStorage Transfer ServiceAWS Storage Gateway
Storage migrationTransfer ApplianceAWS Snowball
CDNCloud CDNAmazon CloudFront
DDoS firewallGoogle Cloud Armor Managed ProtectionAWS Shield Basic/Advanced
Dedicated Interconnect connectionCloud InterconnectAWS Direct Connect
Domains and DNSCloud DNSAmazon Route 53
Domains and DNSCloud DomainsAmazon Route 53
Load balancerCloud Load BalancingAWS Elastic Load Balancing
Network monitoringNetwork Intelligence Center
Network monitoringVPC Flow LogsAmazon VPC Flow Logs
Network securityCloud VPNAWS Virtual Private Network (VPN)
Premium networkingNetwork Service Tiers
Service meshTraffic DirectorAWS App Mesh
ServicesService DirectoryAWS Cloud Map
Virtual networksCloud NATAmazon VPC NAT instances
Virtual networksVirtual Private CloudAmazon Virtual Private Cloud (VPC)
Web application firewallGoogle Cloud ArmorAWS WAF
Audit loggingCloud Audit LogsAWS CloudTrail
DebuggingCloud DebuggerAWS X-Ray
LoggingCloud LoggingAmazon CloudWatch Logs
MonitoringCloud MonitoringAmazon CloudWatch
Performance tracingCloud TraceAWS X-Ray
ProfilingCloud ProfilerAmazon CodeGuru Profiler
Certificate managementCertificate Authority ServiceAWS Certificate Manager
CIAMIdentity PlatformAmazon Cognito
Container securityArtifact RegistryAmazon Elastic Container Registry (ECR)
Container securityContainer AnalysisAmazon ECR Image Scanning
Container securityContainer SecuritySecurity in Amazon Elastic Container Service (ECS)
Container securityGKE SandboxAmazon EKS Container Sandbox
Data loss prevention (DLP)Cloud Data Loss PreventionAmazon Macie
EncryptionConfidential ComputingAWS Nitro Enclaves
Exfiltration preventionVPC Service Controls
Hardware security module (HSM)Cloud HSMAWS CloudHSM
IAMCloud IdentityAWS Identity Services
IAMIdentity and Access ManagementAmazon Identity and Access Management
IAMManaged Service for Microsoft Active DirectoryAWS Managed Microsoft AD
Resource monitoringCloud Asset InventoryAWS Config
Resource monitoringResource ManagerAWS OpsWorks
Secret managementSecret ManagerAWS Secrets Manager
Security administrationCloud Key Management ServiceAWS Key Management Service (KMS)
Security and risk managementSecurity Command CenterAmazon Guard Duty, AWS Security Hub
Zero trustBeyondCorp Enterprise
BuildCloud Storage for FirebaseAWS Simple Storage Service (S3)
BuildFirebase AuthAmazon Cognito
BuildFirebase HostingAWS Simple Storage Service (S3)
BuildFirebase Realtime DatabaseAWS DynamoDB + AppSync
EngageFirebase A/B TestingAmazon Pinpoint
EngageFirebase Cloud MessagingAmazon Device Messaging (ADM),
Amazon Simple Notification Service (SNS)
EngageFirebase Dynamic Links
EngageFirebase In-App MessagingAmazon Device Messaging (ADM),
Amazon Simple Notification Service (SNS)
EngageFirebase Remote Config
EngageGoogle Analytics
Event handling

Eventarc

CNCF CloudEvents integrated

AWS EventBridge
Kubernetes platform

Cloud Run

Knative

CNCF CloudEvents integrated

AWS Fargate
Release & monitorFirebase App Distribution
Release & monitorFirebase Crashlytics
Release & monitorFirebase Performance Monitoring
Release & monitorFirebase Test LabAWS Device Farm
Workflow orchestrationWorkflowsAWS Step Functions
Block storagePersistent DiskAmazon Elastic Block Store (EBS)
File storageFilestoreAmazon Elastic File System (EFS)
Infrequently accessed object storageCloud Storage ArchiveAmazon S3 Glacier
Object storageCloud StorageAWS Simple Storage Service (S3)


Terraform on Google Cloud

https://learn.hashicorp.com/collections/terraform/gcp-get-started

Terraform Admin folder and service account via https://cloud.google.com/community/tutorials/managing-gcp-projects-with-terraform

Creating Service Accounts

https://cloud.google.com/iam/docs/creating-managing-service-accounts


admin_root@cloudshell:~ (biometric-ncorg)$ gcloud iam service-accounts create terraform-sa --description="terraform-sa" --display-name="terraform-sa"
Created service account [terraform-sa].
admin_root@cloudshell:~ (biometric-ncorg)$ gcloud iam service-accounts list
DISPLAY NAME: terraform-sa
EMAIL: terraform-sa@biometric-ncorg.iam.gserviceaccount.com
DISABLED: False


wait 60 seconds for the service account to provision - use an exponential backoff.


Add roles to the Service Account

https://cloud.google.com/iam/docs/granting-changing-revoking-access

Get roles for a particular project first

admin_root@cloudshell:~ (biometric-ncorg)$ gcloud projects get-iam-policy biometric-ncorg
bindings:
- members:
  - user:admin-root nuage-cloud.org
  role: roles/owner

https://cloud.google.com/iam/docs/granting-changing-revoking-access#grant-single-role


Get the organization id from the project ancestor


admin_root@cloudshell:~ (biometric-ncorg)$ ORG_ID=$(gcloud projects get-ancestors biometric-ncorg --format='get(id)' | tail -1)
admin_root@cloudshell:~ (biometric-ncorg)$ echo $ORG_ID
4719....


Add a role at the organization level to a service account


admin_root@cloudshell:~ (biometric-ncorg)$ gcloud organizations add-iam-policy-binding $ORG_ID --member=serviceAccount:terraform-sa@biometric-ncorg.iam.gserviceaccount.com --role=roles/resourcemanager.projectCreator
Updated IAM policy for organization [47....].
bindings:
- members:
  - domain:nuage-cloud.org
  - serviceAccount:terraform-sa@biometric-ncorg.iam.gserviceaccount.com
  role: roles/resourcemanager.projectCreator


Add a role at the organization level to a super admin account



Impersonating Service Accounts

Terraform can impersonate a Google Cloud Service Account.  The user/caller account must have the roles/iam.serviceAccountTokenCreator role set.

see https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference and https://cloud.google.com/iam/docs/create-short-lived-credentials-direct#gcloud with https://cloud.google.com/iam/docs/creating-managing-service-accounts


admin_root@cloudshell:~ (biometric-ncorg)$ SUPER_ADMIN_EMAIL=admin-root@nuag...g
admin_root@cloudshell:~ (biometric-ncorg)$ gcloud organizations add-iam-policy-binding $ORG_ID --member=user:$SUPER_ADMIN_EMAIL --role=roles/iam.serviceAccountTokenCreator
Updated IAM policy for organization [47...].
bindings:
- members:
  - user:admin-root@nuage-cloud.org
  role: roles/iam.serviceAccountTokenCreator

Before

After

Notice the Service Account Token Creator - role

Install gcloud cli

https://github.com/cloud-quickstart/wiki#installing-the-google-cloud-sdk-on-osx

Terraform is provisioned with Google Cloud Shell

Launch with http://shell.cloud.google.com 

Use an example https://registry.terraform.io/modules/terraform-google-modules/kubernetes-engine/google/latest

https://www.hashicorp.com/blog/kickstart-terraform-on-gcp-with-google-cloud-shell

https://registry.terraform.io/providers/hashicorp/google/latest/docs


Welcome to Cloud Shell! Type "help" to get started.
To set your Cloud Platform project in this session use “gcloud config set project [PROJECT_ID]”
cloudshell_open --repo_url "https://github.com/terraform-google-modules/docs-examples.git" --print_file "./motd" --dir "address_basic" --page "editor" --tutorial "./tutorial.md" --open_in_editor "main.tf" --force_new_clone
michael@cloudshell:~$ cloudshell_open --repo_url "https://github.com/terraform-google-modules/docs-examples.git" --print_file "./motd" --dir "address_basic" --page "editor" --tutorial "./tutorial.md" --open_in_editor "main.tf" --force_new_clone
2021/12/16 03:08:42 Cloning https://github.com/terraform-google-modules/docs-examples.git into /home/michael/cloudshell_open/docs-examples
Cloning into '/home/michael/cloudshell_open/docs-examples'...
remote: Enumerating objects: 1906, done.
remote: Total 1906 (delta 0), reused 0 (delta 0), pack-reused 1906
Receiving objects: 100% (1906/1906), 447.94 KiB | 7.46 MiB/s, done.
Resolving deltas: 100% (1442/1442), done.
2021/12/16 03:08:43 ===

gcloud auth list
gcloud config list project

michael@cloudshell:~/cloudshell_open/docs-examples/address_basic$ ls
backing_file.tf  main.tf  motd  tutorial.md
michael@cloudshell:~/cloudshell_open/docs-examples/address_basic$ export GOOGLE_CLOUD_PROJECT=dev-sphere-335220
michael@cloudshell:~/cloudshell_open/docs-examples/address_basic$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/google...
- Finding latest version of hashicorp/random...
- Installing hashicorp/google v4.4.0...
- Installed hashicorp/google v4.4.0 (self-signed, key ID 34365D9472D7468F)
- Installing hashicorp/random v3.1.0...
- Installed hashicorp/random v3.1.0 (self-signed, key ID 34365D9472D7468F)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
michael@cloudshell:~/cloudshell_open/docs-examples/address_basic$

michael@cloudshell:~/cloudshell_open/docs-examples/address_basic$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the followingsymbols:
  + create

Terraform will perform the following actions:

  # google_compute_address.ip_address will be created
  + resource "google_compute_address" "ip_address" {
      + address            = (known after apply)
      + address_type       = "EXTERNAL"
      + creation_timestamp = (known after apply)
      + id                 = (known after apply)
      + name               = (known after apply)
      + network_tier       = (known after apply)
      + project            = (known after apply)
      + purpose            = (known after apply)
      + region             = (known after apply)
      + self_link          = (known after apply)
      + subnetwork         = (known after apply)
      + users              = (known after apply)
    }

  # random_pet.suffix will be created
  + resource "random_pet" "suffix" {
      + id        = (known after apply)
      + length    = 2
      + separator = "-"
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

random_pet.suffix: Creating...
random_pet.suffix: Creation complete after 0s [id=one-doberman]
google_compute_address.ip_address: Creating...
╷
│ Error: Error creating Address: googleapi: Error 403: Compute Engine API has not been used in project 313394869326 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/compute.googleapis.com/overview?project=313394869326 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/compute.googleapis.com/overview?project=313394869326"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/313394869326",
│       "service": "compute.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│
│   on main.tf line 1, in resource "google_compute_address" "ip_address":
│    1: resource "google_compute_address" "ip_address" {

Forgot to enable compute - run...

michael@cloudshell:~/cloudshell_open/docs-examples/address_basic$ gcloud config set project dev-sphere-335220
Updated property [core/project].
michael@cloudshell:~/cloudshell_open/docs-examples/address_basic (dev-sphere-335220)$ gcloud services enable compute.googleapis.com

terraform apply

google_compute_address.ip_address: Creating...
google_compute_address.ip_address: Still creating... [10s elapsed]
google_compute_address.ip_address: Creation complete after 13s [id=projects/dev-sphere-335220/regions/us-central1/addresses/my-address-one-doberman]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.


michael@cloudshell:~/cloudshell_open/docs-examples/address_basic (dev-sphere-335220)$ cat terraform.tfstate
{
  "version": 4,
  "terraform_version": "0.15.0",
  "serial": 9,
  "lineage": "15b70e84-53ce-6cf7-3518-ed294a4181d2",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "google_compute_address",
      "name": "ip_address",
      "provider": "provider[\"registry.terraform.io/hashicorp/google\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "address": "34.70.49.6",
            "address_type": "EXTERNAL",
            "creation_timestamp": "2021-12-15T19:40:55.884-08:00",
            "description": "",
            "id": "projects/dev-sphere-335220/regions/us-central1/addresses/my-address-ample-spaniel",
            "name": "my-address-ample-spaniel",
            "network": "",
            "network_tier": "PREMIUM",
            "prefix_length": 0,
            "project": "dev-sphere-335220",
            "purpose": "",
            "region": "us-central1",
            "self_link": "https://www.googleapis.com/compute/v1/projects/dev-sphere-335220/regions/us-central1/addresses/my-address-ample-spaniel",
            "subnetwork": "",
            "timeouts": null,
            "users": []
          },
          "sensitive_attributes": [],
          "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoyNDAwMDAwMDAwMDAsImRlbGV0ZSI6MjQwMDAwMDAwMDAwfX0=",
          "dependencies": [
            "random_pet.suffix"
          ]
        }
      ]
    },
    {
      "mode": "managed",
      "type": "random_pet",
      "name": "suffix",
      "provider": "provider[\"registry.terraform.io/hashicorp/random\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "ample-spaniel",
            "keepers": null,
            "length": 2,
            "prefix": null,
            "separator": "-"
          },
          "sensitive_attributes": [],
          "private": "bnVsbA=="
        }
      ]
    }
  ]
}
 terraform destroy
google_compute_address.ip_address: Destroying... [id=projects/dev-sphere-335220/regions/us-central1/addresses/my-address-one-doberman]
google_compute_address.ip_address: Still destroying... [id=projects/dev-sphere-335220/regions/us-central1/addresses/my-address-one-doberman, 10s elapsed]
google_compute_address.ip_address: Destruction complete after 11s
random_pet.suffix: Destroying... [id=one-doberman]
random_pet.suffix: Destruction complete after 0s

Destroy complete! Resources: 2 destroyed.  
michael@cloudshell:~/cloudshell_open/docs-examples/address_basic (dev-sphere-335220)$ cat terraform.tfstate
{
  "version": 4,
  "terraform_version": "0.15.0",
  "serial": 6,
  "lineage": "15b70e84-53ce-6cf7-3518-ed294a4181d2",
  "outputs": {},
  "resources": []
}
 


Terraform Example Network

From https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall and https://www.qwiklabs.com/focuses/4375?parent=catalog


# create
resource "google_compute_network" "mynetwork" {
    name = "mynetwork"
    # resources
    auto_create_subnetworks = true
}

# comment out the following
# on mynetwork
resource "google_compute_firewall" "mynetwork-allow-http-ssh-rdp-icmp" {
   name = "mynetwork-allow-http-ssh-rdp-icmp"
   network = google_compute_network.mynetwork.self_link
   allow {
        protocol = "tcp"
        ports = ["22", "80", "3389"]
    }
    allow {
        protocol = "icmp"
    }
    source_ranges = ["0.0.0.0/0"]
}

terraform apply
google_compute_network.mynetwork: Creating...
google_compute_network.mynetwork: Still creating... [10s elapsed]
google_compute_network.mynetwork: Still creating... [20s elapsed]
google_compute_network.mynetwork: Still creating... [30s elapsed]
google_compute_network.mynetwork: Creation complete after 32s [id=projects/..../global/networks/mynetwork]
google_compute_firewall.mynetwork-allow-http-ssh-rdp-icmp: Creating...
google_compute_firewall.mynetwork-allow-http-ssh-rdp-icmp: Still creating... [10s elapsed]
google_compute_firewall.mynetwork-allow-http-ssh-rdp-icmp: Creation complete after 12s [id=projects/..../global/firewalls/mynetwork-allow-http-ssh-rdp-icmp]


Install Terraform

Install Terraform on Windows

Download 1.1.0 via https://releases.hashicorp.com/terraform/1.1.0/terraform_1.1.0_windows_amd64.zip

Copy the extracted terraform.exe into a dir that is already in your PATH.

$ terraform -version
Terraform v1.1.0
on windows_amd64


Install Terraform on OSX

Update Terraform on OSX

terraform --version
Terraform v1.0.11
on darwin_amd64

Your version of Terraform is out of date! The latest version
is 1.1.3. You can update by downloading from https://www.terraform.io/downloads.html

brew tap hashicorp/tap

brew install hashicorp/tap/terraform
==> Downloading https://releases.hashicorp.com/terraform/1.1.4/terraform_1.1.4_darwin_amd64.zip
######################################################################## 100.0%
==> Installing terraform from hashicorp/tap
Error: Your Command Line Tools are too outdated.
Update them from Software Update in System Preferences or run:
  softwareupdate --all --install --force

If that doesn't show you any updates, run:
  sudo rm -rf /Library/Developer/CommandLineTools
  sudo xcode-select --install

Alternatively, manually download them from:
  https://developer.apple.com/download/all/.
You should download the Command Line Tools for Xcode 13.1.

softwareupdate --all --install --force
Software Update Tool
Finding available software
No updates are available.

https://developer.apple.com/download/all/


Continue terraform upgrade

biometric:wse_github michaelobrien$ brew install hashicorp/tap/terraform
Running `brew update --preinstall`...
==> Downloading https://releases.hashicorp.com/terraform/1.1.4/terraform_1.1.4_darwin_amd64.zip
Already downloaded: /Users/michaelobrien/Library/Caches/Homebrew/downloads/42cd50722c72dfb1aca649a75041a885b0885c39fb6aa6e636c87fb38a16e5d5--terraform_1.1.4_darwin_amd64.zip
==> Installing terraform from hashicorp/tap
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/terraform
Target /usr/local/bin/terraform
already exists. You may want to remove it:
  rm '/usr/local/bin/terraform'
To force the link and overwrite all conflicting files:
  brew link --overwrite terraform
biometric:wse_github michaelobrien$ terraform --version
Terraform v1.0.11
on darwin_amd64
Your version of Terraform is out of date! The latest version
is 1.1.3. You can update by downloading from https://www.terraform.io/downloads.html
biometric:wse_github michaelobrien$ brew link --overwrite terraform
Linking /usr/local/Cellar/terraform/1.1.4... 1 symlinks created.
biometric:wse_github michaelobrien$ brew reinstall hashicorp/tap/terraform
==> Downloading https://releases.hashicorp.com/terraform/1.1.4/terraform_1.1.4_darwin_amd64.zip
Already downloaded: /Users/michaelobrien/Library/Caches/Homebrew/downloads/42cd50722c72dfb1aca649a75041a885b0885c39fb6aa6e636c87fb38a16e5d5--terraform_1.1.4_darwin_amd64.zip
==> Reinstalling hashicorp/tap/terraform 

biometric:wse_github michaelobrien$ terraform --version
Terraform v1.1.4
on darwin_amd64


Deploy Google Cloud Infrastructure using Terraform

https://learn.hashicorp.com/tutorials/terraform/google-cloud-platform-build

Create GCP Project

https://cloud.google.com/sdk/gcloud/reference/projects/create

gcloud projects create refarch-obrien --name="refacrh" --labels=type=dev
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/refarch-obrien].
Waiting for [operations/cp.9137809687161597566] to finish...done.
Enabling service [cloudapis.googleapis.com] on project [refarch-obrien]...
Operation "operations/acf.p2-535552712251-008967b0-7fd8-43a1-b748-35bc886090c2" finished successfully.


Enable GCE

https://cloud.google.com/sdk/gcloud/reference/services/enable

gcloud services enable compute


GCP Service Account


GCloud Shell

Gcloud shell is an up to 1 hour provisioned debian VM running a single docker container with a persistent volume that completes in 25 sec -  https://cloud.google.com/shell/docs/how-cloud-shell-works customize the shell via https://cloud.google.com/shell/docs/configuring-cloud-shell#environment_customization with persistent addons including

bq 2.0.71

java 11.0.2,

terraform 1.1.2,

go 1.17.2

python 2.7.16

python3 3.7.1

mysql client,

tensorflow,

git 2.2.0,

jq 1.5.1,

make 4.2.1,

helm 3.5.0,

kubectl 1.22.3,

mvn 3.6.3

Costs

GKE 4 Node Cluster Costs

The cost for a 4 node GKE cluster has 2 components - the underlying GCE VMs and the GKE Kubernetes service.  Daily cost starts at US 3.22 to a week later at 4.12 (0.17/hr) - of which 40% was the GKE service - which is fully discounted (you only pay for the infrastructure - not the control plane)


Google Workspace

Change Workspace Domain and Google Cloud Organization

https://apps.google.com/supportwidget/articlehome?hl=en&article_url=https%3A%2F%2Fsupport.google.com%2Fa%2Fanswer%2F7009324%3Fhl%3Den&assistant_event=welcome&assistant_id=!PzG6dMGRyYTUo6TlVeM9eD60ASX8v_wDkyDjz6HgGR4%3D&product_context=7009324&product_name=UnuFlow&trigger_context=a

https://domains.google.com/registrar/obrienlabs.dev#

Workspace checklist - https://cloud.google.com/docs/enterprise/setup-checklist?utm_source=google&utm_medium=email&utm_content=email&utm_campaign=Enterprise-Checklist-Reminder-Email


Premium Tier Organization

In order to get access to Security Command Center https://cloud.google.com/security-command-center/docs/quickstart-security-command-center you will need Premium Tier access https://cloud.google.com/security-command-center/pricing#premium-tier

Standard tier is free but premium tier is 5% of cloud spend with a base of 25k

Operations

List enabled services

gcloud services list --enabled --project dev

List available services - may run into API quota

gcloud services list --available | grep run

Enabling Services

gcloud services enable \
  compute.googleapis.com \
  iam.googleapis.com \
  iamcredentials.googleapis.com \
  monitoring.googleapis.com \
  logging.googleapis.com \
  notebooks.googleapis.com \
  aiplatform.googleapis.com \
  bigquery.googleapis.com \
  artifactregistry.googleapis.com \
  cloudbuild.googleapis.com \
  container.googleapis.com


Setting default region

Recently one of my organizations ran into an issue where I could only select from a single recent default region.

Resetting the default region


michael@cloudshell:~ (magellan-sbx)$ gcloud config get-value compute/region
Your active configuration is: [cloudshell-30108]
(unset)
michael@cloudshell:~ (magellan-sbx)$ gcloud compute regions list
API [compute.googleapis.com] not enabled on project [29..42]. Would you like to enable and retry (this will take a few minutes)? (y/N)?  y

Enabling service [compute.googleapis.com] on project [298..042]...
Operation "operations/acf.p2-298632905042-4d244f1a-0538-4d0e-8f44-c557937befa9" finished successfully.

NAME: asia-east1
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: asia-east2
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: asia-northeast1
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: asia-northeast2
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: asia-northeast3
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: asia-south1
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: asia-south2
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: asia-southeast1
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: asia-southeast2
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: australia-southeast1
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: australia-southeast2
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: europe-central2
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: europe-north1
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: europe-west1
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: europe-west2
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: europe-west3
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: europe-west4
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: europe-west6
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: northamerica-northeast1
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: northamerica-northeast2
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: southamerica-east1
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: southamerica-west1
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: us-central1
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: us-east1
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: us-east4
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: us-west1
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: us-west2
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: us-west3
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:

NAME: us-west4
CPUS: 0/24
DISKS_GB: 0/4096
ADDRESSES: 0/8
RESERVED_ADDRESSES: 0/8
STATUS: UP
TURNDOWN_DATE:


michael@cloudshell:~ (magellan-sbx)$ gcloud compute project-info add-metadata \
>    --metadata google-compute-default-region=us-east4,google-compute-default-zone=us-east4-a
Updated [https://www.googleapis.com/compute/v1/projects/magellan-sbx]


gcloud init

Your current project has been set to: [magellan-sbx].

Your project default Compute Engine zone has been set to [us-east4-a].
You can change it by running [gcloud config set compute/zone NAME].

Your project default Compute Engine region has been set to [us-east4].
You can change it by running [gcloud config set compute/region NAME].

Created a default .boto configuration file at [/home/michael/.boto]. See this file and
[https://cloud.google.com/storage/docs/gsutil/commands/config] for more
information about configuring Google Cloud Storage.
Your Google Cloud SDK is configured and ready to use!

* Commands will reference project `magellan-sbx` by default
* Compute Engine commands will use region `us-east4` by default
* Compute Engine commands will use zone `us-east4-a` by default

Run `gcloud help config` to learn how to change individual settings

This gcloud configuration is called [cloudshell-30108]. You can create additional configurations if you work with multiple accounts and/or projects.
Run `gcloud topic configurations` to learn more.

Some things to try next:

* Run `gcloud --help` to see the Cloud Platform services you can interact with. And run `gcloud help COMMAND` to get help on any gcloud command.
* Run `gcloud topic --help` to learn about advanced features of the SDK like arg files and output formatting


michael@cloudshell:~ (magellan-sbx)$ gcloud config get-value compute/region
Your active configuration is: [cloudshell-30108]
us-east4


Google Cloud Titan Key

http://g.co/titansecuritykey/help

Jiras






20220305
1058
mbp6
pa.gl
org checklist 8 projects saturates billing account association with project quota

https://support.google.com/code/contact/billing_quota_increase

https://console.cloud.google.com/billing/projects



need CLI output of quota increase - to be able to automate account onboarding in the future


need billing account id copy to qutoa request form

Billing account IDs for which additional quota is needed *



need actual quota link for above billing account association

https://console.cloud.google.com/iam-admin/quotas?referrer=search&project=pg-logging-prod








Some cloud shell accounts list 50 - others 60h per week
cloud identity / workspace domain = 50





The blog page resets the navigation list on "back" after scrolling through "load more blogs"

for example navigate to 

https://cloud.google.com/blog/topics/developers-practitioners

scroll down to a story 2 months ago

https://cloud.google.com/blog/topics/developers-practitioners/introducing-google-cloud-architecture-diagramming-tool

hit the back button to continue scolling

back button resets the list


20220325

message should not say "different" billing account on association

Project “clouddeploy-os” has been moved to a different billing account


Google Cloud Diagrams

The following site is an extension of the 4 words site https://googlecloudcheatsheet.withgoogle.com/

Start with https://cloud.google.com/blog/topics/developers-practitioners/introducing-google-cloud-architecture-diagramming-tool

An open source tool https://github.com/excalidraw/excalidraw was extended to generate deployment (code/scripts - determining?)  from your Google Cloud Architecture Diagram https://googlecloudcheatsheet.withgoogle.com/architecture by adding the "open in cloud shell" icon https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2FGoogleCloudPlatform%2Fappinabox_basiclb&cloudshell_print=install.txt&shellonly=true

Examples

https://googlecloudcheatsheet.withgoogle.com/architecture?diagram=SimpleComputeApp statically references https://github.com/GoogleCloudPlatform/deploystack_todo via https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2FGoogleCloudPlatform%2Fdeploystack_todo&cloudshell_print=install.txt&shellonly=true

https://googlecloudcheatsheet.withgoogle.com/architecture?diagram=ThreeTierApp statically references https://github.com/GoogleCloudPlatform/deploystack_basiclb via 

https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2FGoogleCloudPlatform%2Fdeploystack_todo&cloudshell_print=install.txt&shellonly=true

downloads the following repo to your cloud shell - as per the standard "open in cloud shell" icon you can directly add to your github readme.md

https://github.com/GoogleCloudPlatform/deploystack_basiclb

the install.txt file instructs you to run the ./install sh script

The script asks you for number of VMs


It may be just me but there is no link between the diagramming tool from excalidraw and the github hosted projects - the deploy buttons in the 5 examples are static links

The diagram does not actually generate a deployment template in go.  There is only a link to a static go example project

The issue seems to be that there is no correspondence between the diagram an the deployment - there is only a link to static github sh gcloud deploy commands

Change the diagram and there is no link to the deployment icon (no code or parameter passing) - just a link.

For example, in the diagram there are 2 GCE instances - the deployment script asks for number of instances - change or delete the diagram and the deploy link will work as directed as it is just a link to the existing repo.



TODO INSTALL
This process will create the following:
        * Frontend - Cloud Run Service
        * Middleware - Cloud Run Service
        * Backend - Cloud Sql MySQL instance
        * Cache - Cloud Memorystore
        * Secrets - Cloud Secret Manager

I will have a discussion with the blog author


How to save/access diagram persistence 


Google Cloud Diagramming Icons has a very good list of presentation/(ppt) templates : https://cloud.google.com/icons

Lucidchart is excellent and exports to visio for compliance

https://lucidscale.com/ at $2600/year will generate GCP diagrams from code - on top of your normal manual lucidchart account

https://www.gliffy.com/blog/gcp-architecture-diagrams | https://www.cloudockit.com/the-12-most-used-google-cloud-diagrams-explained/

Automated | https://www.hava.io/gcp-infrastructure-diagrams


Google Cloud Events


Google Cloud Ground School May 18-19 2022 https://events.sada.com/ground-school/

Links


Google Web Toolkit from 2013 : http://eclipsejpa.blogspot.com/2013/02/google-web-toolkit-and-google-app.html

Android Development from 2011: http://eclipsejpa.blogspot.com/2011/10/android-development.html

borg https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43438.pdf

https://cloud.google.com/compute/docs/instances/connecting-to-windows

https://medium.com/google-cloud/parallel-executions-with-google-workflows-3a16f8fee0eb

https://cloud.google.com/anthos-config-management/docs/tutorials/landing-zone

https://cloud.google.com/blog/products/devops-sre/google-cloud-deploy-now-ga

https://cloud.google.com/foundation-toolkit/

https://cloud.google.com/architecture/estimate-gke-costs-early-using-github

DevOps

While studying for the PCDE

https://sites.google.com/corp/google.com/devops-library/home

https://cloud.google.com/architecture/devops/devops-tech-architecture?hl=en

https://cloud.google.com/devops

The devops-sre blog https://cloud.google.com/blog/products/devops-sre/google-cloud-deploy-now-ga

https://cloud.google.com/docs/ci-cd

Training

https://masteringtechnicalsales.com/



  • No labels

1 Comment

  1. check local wsl for gcloud install