Skip to content

This extension provides a base container to run commands against Kubernetes Engine

License

Notifications You must be signed in to change notification settings

estafette/estafette-extension-gke

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Estafette CI GKE extension

This extension makes it easy to deploy an application to GKE, and automatically handles getting the right credentials, implementing a number of best practices and more.

Usage

To use it you can pass a minimal number of parameters and have it create a deployment for you.

releases:
  dev:
    stages:
      deploy:
        image: extensions/gke:stable
        container:
          repository: extensions
        hosts:
        - gke.estafette.io

But there's many more parameters to control the kind of resource it creates - deployment, cronjob, job, statefulset - and many other things to tune.

Parameters

Global parameters

These parameters apply to any of the kind values.

Parameter Description Allowed values Default value
credentials Is automatically generated from the release name prefixed by gke- string gke-${ESTAFETTE_RELEASE_NAME}
action Controls what action is taken; can take values from Estafette release actions deploy-simple, deploy-canary, deploy-stable, restart-simple, restart-canary, restart-stable, diff-simple, diff-canary, diff-stable, rollback-canary deploy-simple
kind Determines the type of Kubernetes resource to get created deployment, headless-deployment, statefulset, job, cronjob, config, config-to-file deployment
dryrun Controls whether the changes generated by this extension will be applied bool false
app The name used to deploy the application string ${ESTAFETTE_LABEL_APP} if set, ${ESTAFETTE_GIT_NAME} otherwise
namespace Sets the kubernetes namespace to deploy to string empty, but usually set in the credential defaults

Note: the action should preferably not be set directly on the stage, but as actions on the stage, so you can trigger every action from estafette using the same stage:

releases:
  prd:
    actions:
    - name: deploy-canary
    - name: deploy-stable
    - name: rollback-canary
      hideBadge: true
    - name: restart-stable
      hideBadge: true
    stages:
      deploy:
        image: extensions/gke:stable
        kind: deployment

Application container parameters

For any of the kind values except for config and config-to-file these set the values for the main application container with sensible defaults. Try to match your application port and endpoints as much as possible to the defaults so you have to override the bare minimum.

Parameter Description Allowed values Default value
container.repository Path for the image repository minus the last part, for example extensions for extensions/gke image, or gcr.io/<project id> string
container.name The name of the container image, usually ${ESTAFETTE_LABEL_APP} or ${ESTAFETTE_GIT_NAME} string app
container.tag The container image tag, usually the build version string ${ESTAFETTE_BUILD_VERSION}
container.imagePullPolicy The image pull policy for the main container image IfNotPresent or Always IfNotPresent
container.port The port the main container listens on; preferably port 5000 so you don't have to explicitly set it int 5000
container.env A map of environment variable keys and values, passed on to the container map[string]interface{}
container.secretEnv Same as env but the values are stored in a secret instead and referenced with secretKeyRef automatically; no need to base64 encode map[string]interface{}
container.cpu.request The cpu request value; this ensures the application has at least the cpu required to operate under normal circumstances and is used for calculating the autoscaling cpu load string 100m
container.cpu.limit The cpu limit; no need to set, it can lead to cpu throttling, but in case your application turns out to be a noisy neighbour can be set string
container.memory.request The requested memory; setting it lower than the limit can lead to out of memory kill before hitting the limit if the node it runs on is short on memory string 128Mi
container.memory.limit The memory limit; when a container hits this limit it's killed with an out of memory kill; set equal to request for guaranteed Quality of Service string 128Mi
container.liveness.enabled Toggles the liveness probe on the application container, which determines whether the application is still healthy bool true
container.liveness.path Sets the path for the liveness probe string /liveness
container.liveness.port Sets the port for the liveness probe int container.port
container.liveness.delay Sets the number of seconds to wait before running the liveness probe first; increase if it takes longer than 30 seconds for the application to be up and running int 30
container.liveness.timeout Time to wait for a response from the liveness path int 1
container.liveness.period Interval between liveness probes int 10
container.liveness.failureThreshold Number of failures before liveness probe is considered to have failed and the container is killed int 3
container.liveness.successThreshold Number of consecutive successes for liveness probe to be considered successful int 1
container.readiness.enabled Toggles the readiness probe on the application container, which determines whether the application is ready to receive requests bool true
container.readiness.path Sets the path for the readiness probe string /readiness
container.readiness.port Sets the port for the readiness probe int container.port
container.readiness.delay Sets the number of seconds to wait before running the readiness probe first int 0
container.readiness.timeout Time to wait for a response from the readiness path int 1
container.readiness.period Interval between readiness probes int 10
container.readiness.failureThreshold Number of failures before readiness probe is considered to have failed and removed as an endpoint from the service, so it no longer receives requests int 3
container.readiness.successThreshold Number of consecutive successes for readiness probe to be considered successful int 1
container.metrics.scrape Toggles whether Prometheus metrics are exposed and need to be scraped bool true
container.metrics.path The path to the Prometheus metrics endpoint string /metrics
container.metrics.port The port at which the Prometheus metrics are exposed int container.port
container.lifecycle.prestopsleep To reduce the risk of failing requests for terminating pods a prestop sleep is used; disable if the container has no sleep command because there's no os (scratch image) bool true for os: linux, false for os: windows
container.lifecycle.prestopsleepseconds Number of seconds to sleep; 15 to 20 should be enough in the majority of cases int 20
container.containerLifecycle To set custom lifecycle as yaml if this set it will disable the container.lifecycle.prestopsleep, so make sure to set the sleep with this custom hook map[string]interface{} nil
container.additionalports[].name To configure any other ports than the usual http/https ports the application can communicate through string
container.additionalports[].port The port number for an additional port int
container.additionalports[].protocol Can be any of the Kubernetes supported protocols TCP or UDP TCP
container.additionalports[].visibility Can be set differently from the main visibility if it needs to be different (more restrictive for example) see visibility visibility

Deployment parameters

Specific to kind deployment

Parameter Description Allowed values Default value
replicas The number of pods to run int 1
visibility Determines how the application can be reached private, public, public-whitelist, esp, espv2, iap, apigee private
workloadIdentity Enable workload identity to access Google Cloud services from applications running within GKE due to its improved security properties and manageability. bool false
iapOauthClientID Needs a Google OAuth Client ID encoded in base64 when using visibility: iap; has to be created in advance string (base64 encoded)
iapOauthClientSecret Needs a Google OAuth Client Secret encoded in base64 when using visibility: iap; has to be created in advance string (base64 encoded)
espEndpointsProjectID When Google Cloud Endpoints are set up in a centralized project set it's ID with this parameter string
espConfigID When you want to pin the version of the openapi spec uploaded as a Google Cloud Endpoint config it can be set string Takes the latest openapi spec uploaded by this extension
espOpenapiYamlPath Path to openapi.yaml file to use for creating the endpoint config; use separate ones per environment string
espServiceTypeClusterIP Creates ClusterIP service instead of LoadBalancer for ESP. Doesn't destroy already existing LB (if exists), but will point DNS record to new IP bool false
whitelist A list of [CIDRs][https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing] to allow access to the application []string The default configured in the nginx-office controller
progressDeadlineSeconds Sets the number of seconds for Kubernetes to wait for a deployment to lack progress before treating it as a failure int 600
os The operating system to deploy to linux, windows linux
chaosproof Determines whether it's okay to run the application on preemptibles bool false
manifests.files To set additional template files to apply []string
manifests.data To provide extra data to the additional templates beyond what's already set by the extension map[string]interface{}
trustedips To set loadBalancerSourceRanges on the service of type LoadBalancer for visibility: public or visibility: esp or visibility: espv2 []string Cloudflare's origin ip addresses, see https://www.cloudflare.com/ips-v4
labels To set labels that are use on all kubernetes resources map[string]string The labels set in the .estafette.yaml manifest
containerNativeLoadBalancing To use Google Cloud container-native load balancing bool
hosts The public hostnames associated with this application []string
hostsrouteonly Additional hostnames listened to by the app and its ingresses, but not set in DNS records []string
internalhosts The internal hostnames associated with this application []string
internalhostsrouteonly Additional internal hostnames listened to by the app and its ingresses, but not set in DNS records []string
apigeesuffix Suffix for the hostnames when using visibility: apigee string apigee
dns.useCloudflareEstafetteExtension Add annotations used by estafette-cloudflare-dns-extension bool false
dns.useExternalDNS Add annotations used by external-dns bool true
basepath Base path in the ingresses to route to this application string /
autoscale.enabled Enables Horizontal Pod Autoscaler bool true
autoscale.min The minimum replicas set in the HPA int 3
autoscale.max The maximum replicas set in the HPA int 100
autoscale.cpu Target CPU percentage set in the HPA int 80
autoscale.behavior https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior map[string]interface{}
autoscale.safety.enabled Enabled use of estafette-k8s-hpa-scaler as a safety net bool false
autoscale.safety.promquery The Prometheus query to get the request rate to this or a downstream application in the call stack string sum(rate(nginx_http_requests_total{app='%v'}[5m])) by (app)
autoscale.safety.ratio A divider to get from request rate to number of pods; equals the desired requests per pod string 1
autoscale.safety.delta A constant to increase or lower the function minReplicas = Ceiling ( delta + ( promquery / ratio ) ) string
autoscale.safety.scaledownratio Sets the fraction the min replicas is allowed to scale down compared to the last value in order to ease scaling string 1
vpa.enabled Enables Vertical Pod Autoscaler bool false
vpa.updateMode The update mode for VPA "Off", "Initial", "Recreate", "Auto" "Off"
request.ingressbackendprotocol The backend protocol (HTTPS or GRPCS) to be used by the Ingress string HTTPS
request.timeout Maximum time for a response, set at the ingresses and openresty sidecar string 60s
request.maxbodysize Maximum body size for a request, set at the ingresses and openresty sidecar string 128m
request.proxybuffersize Buffer size for proxying, set at the ingresses and openresty sidecar string 4k
request.proxybuffersnumber Number of buffers for proxying, set at the ingresses and openresty sidecar int 4
request.clientbodybuffersize Buffer size for request body, set at the ingresses and openresty sidecar string 8k
request.loadbalance Loadbalancing algorithm used by the ingress controller ewma, round_robin round_robin
request.authsecret Secret name in the form of namespace/secret used for client certificate authentication string
request.verifydepth The validation depth between the provided client certificate and the Certification Authority chain int 3
request.configuration-snippet Configration snippet for all nginx ingresses string
secrets.keys Map of filenames and base64 encoded values stored in a secret, mounted into the application container map[string]interface{}
secrets.mountpath Path to where the secret is mounted string
configs.files Files in the repository to include in a configmap, mounted into the application container []string
configs.data Key/value map to replace any gotemplate placeholders in the config files set with configs.files map[string]interface{}
configs.inline Key/value map to set config files for the configmap without using templates on disk map[string]string
configs.mountpath Path to where the configmap is mounted string
volumemounts[].name Additional volumes to mount into the application container string
volumemounts[].mountpath Path to where the volume is mounted string
volumemounts[].volume Yaml snippet for the volume spec; can be used to mount secrets, configmaps, persistentvolumeclaims, etc map[string]interface{}
certificatesecret If set use a pre-existing secret with TLS certificate instead of automatically creating one from the host and internalhosts using a secret with estafette-letsencrypt-certificate annotations string
allowhttp If the application needs to be available on http, besides the default https bool false
enablePayloadLogging Mounts a host path into the container that's used for an internal Travix payload log shipper bool false
useGoogleCloudCredentials Uses a estafette-gcp-service-account annotated secret to get a service account keyfile and mount it into the application container bool false
disableServiceAccountKeyRotation Keeps the retrieved service account keyfile without rotating it regularly bool true
legacyGoogleCloudServiceAccountKeyFile Base64 encoded keyfile stored in the application secret string
googleCloudCredentialsApp When a shared service account needs to be used set the name of the app the service account is generated for string app
probeService Configures a prometheus probe on the service using blackbox-exporter to check for availability bool false for visibility: esp and visibility: espv2, true otherwise
topologyAwareHints Enables Topology Aware Hints, which provides a mechanism to help keep traffic within the zone it originated fromand reduce the extra costs generated from egress traffic bool true
tolerations Yaml snippets to configure Kubernetes tolerations []yaml snippet
affinity Map of pod and node (anti)affinity config to configure Kubernetes affinity map[string]interface{} Pre-defined PodAntiAffinity and NodeAffinity based on type of OS
injecthttpproxysidecar Indicates whether the openresty sidecar should be injected bool true
initcontainers Yaml snippets to configure Kubernetes init containers []yaml snippet
sidecar deprecated, use sidecars parameter instead
sidecars[].type Can be used to configure a couple of sidecars known by this extension openresty, esp, espv2, cloudsqlproxy, istio
sidecars[].image The full container image path for the sidecar string
sidecars[].env Environment variables passed into the sidecar map[string]interface{}
sidecars[].secretEnv Secret values passed into the sidecar as environment variables by storing them in a secret and using secretKeyRef map[string]interface{}
sidecars[].cpu.request The cpu request value string 50m
sidecars[].cpu.limit The cpu limit value string
sidecars[].memory.request The memory request value string 30m
sidecars[].memory.limit The memory limit value string 50m
sidecars[].healthcheckpath Can be set for the health check through the openresty sidecar towards the main application string container.readiness.path
sidecars[].dbinstanceconnectionname A Cloud SQL connection name to be used in the Cloud SQL proxy sidecar string
sidecars[].sqlproxyport The port the cloud sql proxy listens on int 5432
sidecars[].sqlproxyterminationtimeoutseconds The cloud sql proxy termination timeout int 60
customsidecars Yaml snippets to pass in additional sidecars []yaml snippet
strategytype Configures the upgrade strategy for kind: deployment; augments the Kubernetes strategyType with AtomicUpdate RollingUpdate, Recreate, AtomicUpdate
rollingupdate.maxsurge Maximum percentage of pods to surge during a rolling update string 25%
rollingupdate.maxunavailable Maximum number of unavailable pods during a rolling update string 0
rollingupdate.timeout Timeout during a rolling update before considering it as failed string 5m
defaultOpenrestySidecarImage Allows the default OpenResty sidecar image to be overridden via defaults in kubernetes-engine credentials string estafette/openresty-sidecar@sha256:2aa9f2c8c3f506e0f6cc70871701b5ac81aa0f12e8574c7b8213e4d0379d2ddd
defaultESPSidecarImage Allows the default ESP sidecar image to be overridden via defaults in kubernetes-engine credentials string gcr.io/endpoints-release/endpoints-runtime:1.57.0
defaultESPv2SidecarImage Allows the default ESP v2 sidecar image to be overridden via defaults in kubernetes-engine credentials string gcr.io/endpoints-release/endpoints-runtime:2.29.1
defaultCloudSQLProxySidecarImage Allows the default Cloud SQL proxy sidecar image to be overridden via defaults in kubernetes-engine credentials string eu.gcr.io/cloudsql-docker/gce-proxy:1.35.0
imagePullSecretUser When the application image is stored in a private registry not accessible for the GKE cluster set a username string
imagePullSecretPassword Password for the private registry string

Note: for visibility: esp a release needs access to the openapi spec, so combine with clone: true on the release target, for example:

releases:
  dev:
    clone: true
    stages:
      deploy:
        image: extensions/gke:stable
        visibility: esp
        espOpenapiYamlPath: openapi.dev.yaml

Statefulset parameters

Specific to kind statefulset

Parameter Description Allowed values Default value
storageclass Determines the type of persistent disk created standard standard, pd-standard, pd-ssd
storagesize The size of the persistent disk string 1Gi
storagemountpath The path where the persistent disk is mounted string /data

Cronjob parameters

Specific to kind cronjob

Parameter Description Allowed values Default value
schedule Sets the schedule at which the cronjob spawns a new job string
concurrencypolicy Indicates whether concurrent jobs are allowed or forbidden Allow, Forbid Allow

Cronjob/Job parameters

Specific to kind cronjob and job

Parameter Description Allowed values Default value
completions The amount of times the job needs to complete int 1
parallelism How many jobs to run in parallel int 1
backoffLimit After how many failures to stop retrying int 6
restartPolicy Controls whether a container should be restarted when it stops Always, OnFailure OnFailure

Visibility

How the customers of an application - whether it's an actual user connecting or another service - can communicate with your application is set by the visibility parameter.

Value Description
private A Kubernetes service of type ClusterIP is created and depending on the values for hosts and internalhosts it also sets up ingresses using ingress class nginx-office and/or nginx-internal
public A LoadBalancer Kubernetes service is created which gives direct public access to the application; this is no longer allowed for new deployments, esp should be used in that case to avoid accidental exposure of endpoints like /metrics, /swagger, etc
public-whitelist This uses ingress controller nginx-office but sets a different list of allowed ip addresses than the default by using the whitelist parameter
esp This creates a Google Cloud Endpoint, adds the esp sidecar container to the deployment and exposes it through a LoadBalancer service
espv2 Sames as esp but uses the envoy-based version 2
iap Sets up the application behind Identity Aware Proxy; requires parameters iapOauthClientID and iapOauthClientSecret to be set
apigee Routes requests through the nginx-open ingress controller; requires parameters request.authsecret and request.verifydepth to be set

Note: all of the above set up an internal ingress if parameter internalhosts is set; for esp this cannot be used to connect to the application since internally since it's limited to only a single hostname

About

This extension provides a base container to run commands against Kubernetes Engine

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages