How to run graylog cluster in kubernetes

Graylog is the splunk equivalent for log collection, indexing , search analysis function.

I am playing around with it little bit and seeing the functions, so i need a quick installed version in 

1. If OS X install minikube https://kubernetes.io/docs/getting-started-guides/minikube/#minikube-features
2. Running graylog cluster : 
• #kubectl run graylog --image=graylog2/allinone --port=9000
3. Forward TCP:9000 to show the graylog UI 
• #kubectl port-forward graylog-2473246842-b6rhg 9000 &
4. To view UI : point your browser to : http://127.0.0.1:9000
• admin/admin is the default username and password
5. To view logs of graylog pods : 
• #kubectl get pods ( this should give an output of running graylog container instance )
• #kubectl logs graylog-2473246842-b6rhg

Reference : http://crispanalytics.com/blog/running-graylog-kubernetes/

Data Analytics / Data Pipeline PANKCAKE

• Data Analytics / Data Pipeline
• PANCAKE stack
• PrestoDB : https://prestodb.io/ : Distributed SQL for big data ( HIVE, HBASE, cassandra , any SQL
• Arrow : Apache arrow ( https://arrow.apache.org/ ) , high performing columnar-in memory
• Nifi : https://blogs.apache.org/nifi/ ( big fat 700MB) Excellent tool for graphing and data routing , transformation, visualization 
• Cassandra : http://cassandra.apache.org/ : Clustered no sql DB , written in java
• Airflow : https://github.com/apache/incubator-airflow : Orchestrator, work flow engine, configure hadoop jobs and direct the scheduler
• Kafka : https://kafka.apache.org/ : High speed messaging 
• ElasticSearch
• Other Tools
• Apache-Spark : 
• TensorFlow
• Algebird
• CoreNLP
• Kibana
• Apache Zeppelin : Multi-purpose Notebook for 
• Data Ingestion
• Data Discovery
• Data Analytics
• Data Visualization and Collaboration
• PIG scripts and Map Reduce scripts 
• http://pipeline.io/ : Optimizing tensor flow for prod
• http://jupyter.org/ : Awesome online tool to have shared notepad, code ,
• PMML : Predictive model markup language  : JPMML : Java PMML ( https://www.ibm.com/developerworks/library/ba-ind-PMML1/ )

Working on a goal to bring this up in simple kubenetes cluster.

minikube ( playgroud for kubernetes )

Exercise Goal : Bring up minikube ( kubernetes ) in local VM

Bringing up minikube (kubenetes) + weave scope ( a monitoring stack ) in local laptop

• Minikube 
• I am on OS X Darwin, I got virtual box installed
• Lets get started https://github.com/kubernetes/minikube
• Install below if not exist, “Requirements"
• xhyve driver, VirtualBox or VMware Fusion installation
• Get minikube latest release
• Follow instructions as in the page , minikube start …
• I got the dashboard on by
• minikube dashboard
• More links https://medium.com/@claudiopro/getting-started-with-kubernetes-via-minikube-ada8c7a29620#.gc87faxvq
• http://blog.kubernetes.io/2016/07/minikube-easily-run-kubernetes-locally.html

#minikube addons list

- registry-creds: disabled

- addon-manager: enabled

- dashboard: enabled

- kube-dns: enabled

- heapster: disabled

- ingress: disabled

I could not successfully bring in weave, i like it better graphically, but all i got it heapster for now

#minikube addons enable heapster 

heapster was successfully enabled

Kubectl Drain

#kubectl drain minikube —force

This node minikube will be drained and all services will be stopped by force, there is an option to provide grace time.

To recover a node which has drained

#kubectl uncordon minikube

Heapster – the influx , graphana, dashboard for kube

#minikube addons open heapster 

This should open a browser and graphana showing pods, cluster

Serverless ( AWS : Lambda / GCP:Cloud Functions / Azure:Functions )

Serverless 

I am writing this as i exploring the infrastructure and landscape of serverless in early 2017.
Lately i see the adoption to serverless is drastic and really see the need for it every month i have to check the bills.

I was telling people that serverless is the solution to big fat bills from cloud providers. In turn what really that means is serverless is the way of standing up applications, databases, ETL jobs , web servers etc... in simple and quick way , where the huge operational cost of standing up virtual machines with operating system is not needed. Indeed the application runs as simple function instance.

AWS : Lamba
GCP : Cloud Functions
Azure : Functions

Are the available famous serverless architectures.

Marriage of containers and serverless

as i write here, there are lots of momentum on container (docker) supporting the serverless infrastructure.


Hypernetes

Hypernetes is a secure, multi-tenant Kubernetes distro. Simply put,

Hypernetes = Bare-metal + Hyper + Kubernetes + KeyStone + Cinder + Neutron.

https://github.com/hyperhq/hypernetes

Hyper.sh
Reference : https://www.hyper.sh/

Tricks using GCP for begginers

Tracking Some gcloud commands

1. I see the application login authentication need your browser to use the google id and password.

This creates a file ~/.kube/config and stores the downloaded user information session in here.

Launch kubectl proxy or #kubectl cluster-info this uses this application login details to function.

gcloud auth application-default login




Kubernetes cluster on GCP for Dummies

Agenda : Bring up kubernetes cluster on Google Cloud Platform

Reference :  https://github.com/kelseyhightower/kubernetes-the-hard-way

1. Bring up https://cloud.google.com, sign in as your google id or create a new google account. Note : Currently $300 is free per account on GCP, *still need credit card
2. Follow the instruction in the reference section https://github.com/kelseyhightower/kubernetes-the-hard-way

Sequence of steps

1. Cloud Infrastructure Provisioning
2. Setting up a CA and TLS Cert Generation
3. Bootstrapping an H/A etcd cluster
4. Bootstrapping an H/A Kubernetes Control Plane
5. Bootstrapping Kubernetes Workers
6. Configuring the Kubernetes Client - Remote Access
7. Managing the Container Network Routes
8. Deploying the Cluster DNS Add-on
9. Smoke Test
10. Cleaning Up

Result
I got the cluster up and working and when i tried to install kube-dashboard, it did not work.
"Unauthorized" error when accessed web UI.

Screens

Some more readings : https://thehftguy.com/2016/06/15/gce-vs-aws-in-2016-why-you-should-never-use-amazon/

Public Cloud Monitoring and Diagnostic tools

As i was exploring tools on the microsoft azure and actively deploying applications.
Around 300 some VMs and lots of network constructs with VLANs and subnets.

I was looking for a one stop shop / single pane of glass tool for public cloud ( azure ) and local DC server applications , and hybrid monitoring tool.


I see the eco-system has good momentum and lots of thrid party applications claiming the

• https://stackify.com/
• http://www.cerebrata.com/products/azure-diagnostics-manager
• http://cloudmonix.com/ ( lots of visualization graphs and drill down ability )
• https://www.unigma.com/

Some other tools

• https://www.upguard.com/features, I personally like these features 
• Configuration search
• Scalable Anomaly Detection
• Time-lapse Collaboration View
• Visual Difference at its core
• https://www.opsclarity.com/product/features/
• Automated Topology Discovery
• Infrastructure Host map 
• APIs
• Identity and Access : https://jumpcloud.com/daas-product/index
• User Management 
• Device Management 
• SSO
• Active Directory Bridge
• LDAP-aas
• MFA
• https://www.moogsoft.com/product/
• Real-time anomaly detection 
• Closed loop remediation
• Production workbench

cadvisor , influx and grafanna

1.sudo docker run -d -p 8083:8083 -p 8086:8086 --expose 8090 --expose 8099 --name influxsrv tutum/influxdb

2.sudo docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish 8080:8080 --detach=true --link influxsrv:influxsrv --name=cadvisor google/cadvisor:latest -storage_driver_db=influxdb -storage_driver_host=influxsrv:8086

3.sudo docker run -d -p 3000:3000 -e INFLUXDB_HOST=localhost -e INFLUXDB_PORT=8086 -e INFLUXDB_NAME=cadvisor -e INFLUXDB_USER=root -e INFLUXDB_PASS=root --link influxsrv:influxsrv --name grafana grafana/grafana

# git repo for remote TTY over the web # git clone https://github.com/krishnasrinivas/wetty

4. sudo docker run --name term -p 3000 -dt nathanleclaire/wetty

Linux Process Management - S6

http://skarnet.org/software/s6/overview.html

S6 is a  process management tool , built from ground up. Based upon reliable libraries

https://skarnet.org/software/s6/why.html

Purpose : To manage any process inside a operating system , restart if needed.

Some examples of s6 programs meant to be used in run scripts:

• The s6-log program is a long-lived process. It is meant to be executed into by a ./log/run script: it will be supervised, and will process what it reads on its stdin (i.e. the output of the ./run daemon).
• The s6-envdir program is a short-lived process that will update its current environment according to what it reads in a given directory, then execute into the rest of its command line. It is meant to be used in a run script to adjust the environment with which the final daemon will be executed into.
• Similarly, the s6-softlimit program adjusts its resource limits, then executes into the rest of its command line: it is meant to set the resources the final daemon will have access to.
• The s6-applyuidgid program, part of the s6-*uidgid family, drops root privileges before executing into the rest of its command line: it is meant to be used in run scripts that need root privileges when starting but do not need it for the execution of the long-lived process.
• s6-ipcserverd is a daemon that listens to a Unix socket and spawns a program for every connection. It is meant to be supervised, so it should be used in a run script, and it's also meant to be a flexible super-server that you can use for different applications: so it is a building block that may appear in several of your run scripts defining local services.

opendoor

A home flipping startup ,