[GTALUG] Text of my slides

[Christopher Browne]

Chris’ Intro to Kubernetes
Christopher Browne
GTALUG, January 2020

The Agenda
A wee intro to k8s (because kubernetes is really long to type!)
Some major components
A progression of service evolution
Chris’ crazy batch jobs, and my challenges
Some useful tools

k8s
Originally a Google project called “Borg”
It’s all about orchestrating the running of services in containers
The “Borg” metaphor…
Robotic services, lots of them, with sub- and sub-sub-services.
Launched automatically.  Relaunched on failure.
Spread across a series of Borg Kubes.  If one Kube blows up, there’s more!
Kubernetes is a Greek word meaning “helmsman” or “pilot”; the one that
steers a ship
It uses Docker (usually); it’s the “opposite side”

Major Components
Starts with a Container system
Docker, containerd, cri-o, rtklet, frakti, AWS Firecracker, gVisor, …
Alternatives claim to be faster or more secure… are they???
Etcd - distributed key/value store for configuration
DNS server - automatically managing hostnames inside the cluster
Kube Proxy - maintains network rules so outside hosts can get at
services inside the cluster

Less Visible Components
Kube API server - used to control node activities
Kube scheduler - launches and destroys Pods
Node controller - monitors each node (am I alive?)
Replication Controller - add/drop Pods based on policy
Endpoint Controller - connects Pods to form Services
Visible or not, the Real Point of Kubernetes is to have the clusters
self-monitoring and self-administering.

A Progression of Sorts of Services
Pods - a set of related containers sharing storage + IP
Multiple containers so you add extra services via extra containers
Use fluentd to aggregate collect logs, forward to ElasticSearch
CronJob so a process runs periodically
Job (run the thing once, more or less)
DaemonSet (so each node has an instance of a service pod)
ReplicationController - obsolesced
ReplicaSet - obsolesced
Deployment defining a ReplicaSet
StatefulSet (needs stable network or storage)


More Sophisticated Services
Operators (think: “system operators”)
PostgreSQL Operator
Deploy multiple replica nodes (sync and async replication)
Run backups using pgBackRest (including to S3)
Automated failover via distributed consensus subsystem
Automated recovery based on replicas or backups
Component for managing users and permissions
Capability to scale up by cloning DB clusters
Sets up pgBouncer for connection pooling
Includes health monitoring
componentshttps://github.com/operator-framework/awesome-operators

Yaml, Yaml Everywhere
Manifests about how the pods in your services are configured, are
written in YAML
Configuration to pass to the pods, also YAML
You are in a maze of twisty YAMLs, all nearly the same…
I think I want an engine for generating YAML way more automatically
(you’ll see why later…)
Well, that’s why they have Helm - http://helm.sh
Lots of systems use the Go language system for templating {{- if
(.Files.Glob “myfile.conf”) }} {{ (.Files.Glob “myfile.conf”).AsConfig
| indent 2 }}

Configuration Pains
Some things are painful…
Twisty maze of YAML, Almost the same…
Environment Variable Overlay  Don’t call a config element PGPORT or
PGHOST or PGDATABASE
It’s kinda like running within a cron job...

My Job Journey
The stuff I build tends to be shell scripts running SQL queries
Not a great fit for being an “endpoint” or a “service”
Best fit appears to be k8s “Jobs”
I have activities split into a series of job steps that pretty much
need to be run serially, one after another
The “best fit” isn’t to run a bunch of k8s Jobs, either
No dependency system provided
No indication that this is an area to expect much evolution
So, I wrote 2 shell scripts as a “batch job controller”

Receives a list of job steps to be run, JOBSTEPS
Verifies that the steps are all valid (abort if not)
Iterates across the steps in JOBSTEPS, in order:
echo “start $step” > $LAUNCHMARKFILE
Poll, waiting for the step to indicate it is done via seeing “done” in
$JOBSTATEFILE
job-controller.sh gets run by 1 container in the Job pod, logging as it goes

Run-controller.sh Job-controller.sh

Batch-job.sh Job-runner.sh
There are a couple dozen containers, one per job step.   Each runs
via: job-runner.sh my-step-name [immediate]
job-launcher function in batch-job.sh:
Grabs configuration, turns it into a set of environment variable values
Wait Loop; watch until $LAUNCHMARKFILE contains  “start my-step-name”,
It’s my turn now :-)
Run the logic for this job step
echo “done” > $JOBSTATEFILE to launch the next step


A little Column A, a Little Column B...
This Job pod has (at the moment) 22 containers
Boy, that’s a lotta YAML!!!
That’s a lotta containers!!!
Some aspects are pretty clean
Each job step gets its own distinct STDOUT logging
It’s easy to identify which step is which
At any given time, only one container is busy; others are lazy
There’s only one Pod
There’s only one container definition, used by all 22 containers
Using one Pod means all the containers have easy access to common data
files, no need to resort to GFS/gluster/NFS to share data
Not terribly parallel, but that’s OK

Helm
Loosely, a package manager for k8s
Implements Charts, that comprise sets of k8s Pods
Provides a way of deploying locally-custom config, that, attached to a
Chart, comprises a software Release
Templates help you mix your configuration with defaults to provide a
complete k8s manifest
Requirements document dependencies (“we need a bunch of other Charts installed”)
Some support for upgrading to newer versions of software

Shell Tools
Because k8s encourages long command lines, so you want any help you can get…
Command completion - built-in
source <(kubectl completion bash)
source <(kubectl completion zsh) github.com/derailed/k9s - curses UI
to manage clustergithub.com/ahmetb/kubectx - CLI help to smoothly
switch between clusters (kubectx) and namespaces
(kubens)github.com/jonmosco/kube-ps1 - k8s prompts for
bash/zshgithub.com/abrochard/kubel - k8s buffer in Emacs

Emacs Toolsgithub.com/abrochard/kubel - k8s buffer in
Emacsgithub.com/chrisbarrett/kubernetes-el - k8s
“porcelain”github.com/abrochard/kubernetes-helm - run Helm commands
from within Emacs
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gtalug.org/pipermail/talk/attachments/20200114/5b8866ed/attachment.html>