GKE - Google Container Engine

Google Container (with K) Engine aka GKE is a container solution from google cloud platform (GCP)

• GCP : https://cloud.google.com/container-engine/
• Kubernetes undeneath

• MS-Azure : https://azure.microsoft.com/en-us/services/container-service/
• Docker swarm
• DC/OS
• Kubenetes 
• Amazon AWS : http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_instances.html
• https://console.aws.amazon.com/iam/

How to start with GKE?

1. I assume you have a cloud.google.com account signed up with your gmail account, need credit card (but $300 credit) for 60 days
2. https://cloud.google.com/container-engine/
3. View My Console 
4. Remember to download "gcloud" https://cloud.google.com/sdk/docs/quickstart-mac-os-x
5. My cluster is called "kubecluster-1"
6. Once i have gcloud init and installed , set up the auth by gcloud auth list      
7. Sample kubectl command shows it works

#kubectl get namespaces

NAME          STATUS    AGE

default       Active    1h

kube-system   Active    1h

voila, kube is up

How to run graylog cluster in kubernetes

Graylog is the splunk equivalent for log collection, indexing , search analysis function.

I am playing around with it little bit and seeing the functions, so i need a quick installed version in 

1. If OS X install minikube https://kubernetes.io/docs/getting-started-guides/minikube/#minikube-features
2. Running graylog cluster : 
• #kubectl run graylog --image=graylog2/allinone --port=9000
3. Forward TCP:9000 to show the graylog UI 
• #kubectl port-forward graylog-2473246842-b6rhg 9000 &
4. To view UI : point your browser to : http://127.0.0.1:9000
• admin/admin is the default username and password
5. To view logs of graylog pods : 
• #kubectl get pods ( this should give an output of running graylog container instance )
• #kubectl logs graylog-2473246842-b6rhg

Reference : http://crispanalytics.com/blog/running-graylog-kubernetes/

Data Analytics / Data Pipeline PANKCAKE

• Data Analytics / Data Pipeline
• PANCAKE stack
• PrestoDB : https://prestodb.io/ : Distributed SQL for big data ( HIVE, HBASE, cassandra , any SQL
• Arrow : Apache arrow ( https://arrow.apache.org/ ) , high performing columnar-in memory
• Nifi : https://blogs.apache.org/nifi/ ( big fat 700MB) Excellent tool for graphing and data routing , transformation, visualization 
• Cassandra : http://cassandra.apache.org/ : Clustered no sql DB , written in java
• Airflow : https://github.com/apache/incubator-airflow : Orchestrator, work flow engine, configure hadoop jobs and direct the scheduler
• Kafka : https://kafka.apache.org/ : High speed messaging 
• ElasticSearch
• Other Tools
• Apache-Spark : 
• TensorFlow
• Algebird
• CoreNLP
• Kibana
• Apache Zeppelin : Multi-purpose Notebook for 
• Data Ingestion
• Data Discovery
• Data Analytics
• Data Visualization and Collaboration
• PIG scripts and Map Reduce scripts 
• http://pipeline.io/ : Optimizing tensor flow for prod
• http://jupyter.org/ : Awesome online tool to have shared notepad, code ,
• PMML : Predictive model markup language  : JPMML : Java PMML ( https://www.ibm.com/developerworks/library/ba-ind-PMML1/ )

Working on a goal to bring this up in simple kubenetes cluster.

minikube ( playgroud for kubernetes )

Exercise Goal : Bring up minikube ( kubernetes ) in local VM

Bringing up minikube (kubenetes) + weave scope ( a monitoring stack ) in local laptop

• Minikube 
• I am on OS X Darwin, I got virtual box installed
• Lets get started https://github.com/kubernetes/minikube
• Install below if not exist, “Requirements"
• xhyve driver, VirtualBox or VMware Fusion installation
• Get minikube latest release
• Follow instructions as in the page , minikube start …
• I got the dashboard on by
• minikube dashboard
• More links https://medium.com/@claudiopro/getting-started-with-kubernetes-via-minikube-ada8c7a29620#.gc87faxvq
• http://blog.kubernetes.io/2016/07/minikube-easily-run-kubernetes-locally.html

#minikube addons list

- registry-creds: disabled

- addon-manager: enabled

- dashboard: enabled

- kube-dns: enabled

- heapster: disabled

- ingress: disabled

I could not successfully bring in weave, i like it better graphically, but all i got it heapster for now

#minikube addons enable heapster 

heapster was successfully enabled

Kubectl Drain

#kubectl drain minikube —force

This node minikube will be drained and all services will be stopped by force, there is an option to provide grace time.

To recover a node which has drained

#kubectl uncordon minikube

Heapster – the influx , graphana, dashboard for kube

#minikube addons open heapster 

This should open a browser and graphana showing pods, cluster

Serverless ( AWS : Lambda / GCP:Cloud Functions / Azure:Functions )

Serverless 

I am writing this as i exploring the infrastructure and landscape of serverless in early 2017.
Lately i see the adoption to serverless is drastic and really see the need for it every month i have to check the bills.

I was telling people that serverless is the solution to big fat bills from cloud providers. In turn what really that means is serverless is the way of standing up applications, databases, ETL jobs , web servers etc... in simple and quick way , where the huge operational cost of standing up virtual machines with operating system is not needed. Indeed the application runs as simple function instance.

AWS : Lamba
GCP : Cloud Functions
Azure : Functions

Are the available famous serverless architectures.

Marriage of containers and serverless

as i write here, there are lots of momentum on container (docker) supporting the serverless infrastructure.


Hypernetes

Hypernetes is a secure, multi-tenant Kubernetes distro. Simply put,

Hypernetes = Bare-metal + Hyper + Kubernetes + KeyStone + Cinder + Neutron.

https://github.com/hyperhq/hypernetes

Hyper.sh
Reference : https://www.hyper.sh/

Tricks using GCP for begginers

Tracking Some gcloud commands

1. I see the application login authentication need your browser to use the google id and password.

This creates a file ~/.kube/config and stores the downloaded user information session in here.

Launch kubectl proxy or #kubectl cluster-info this uses this application login details to function.

gcloud auth application-default login




Kubernetes cluster on GCP for Dummies

Agenda : Bring up kubernetes cluster on Google Cloud Platform

Reference :  https://github.com/kelseyhightower/kubernetes-the-hard-way

1. Bring up https://cloud.google.com, sign in as your google id or create a new google account. Note : Currently $300 is free per account on GCP, *still need credit card
2. Follow the instruction in the reference section https://github.com/kelseyhightower/kubernetes-the-hard-way

Sequence of steps

1. Cloud Infrastructure Provisioning
2. Setting up a CA and TLS Cert Generation
3. Bootstrapping an H/A etcd cluster
4. Bootstrapping an H/A Kubernetes Control Plane
5. Bootstrapping Kubernetes Workers
6. Configuring the Kubernetes Client - Remote Access
7. Managing the Container Network Routes
8. Deploying the Cluster DNS Add-on
9. Smoke Test
10. Cleaning Up

Result
I got the cluster up and working and when i tried to install kube-dashboard, it did not work.
"Unauthorized" error when accessed web UI.

Screens

Some more readings : https://thehftguy.com/2016/06/15/gce-vs-aws-in-2016-why-you-should-never-use-amazon/