golang CI

I came across these important tools and marking it here for references, here

• https://build.golang.org : Dashboard
• http://www.golangbootcamp.com/book/tricks_and_tips

Golang tiny test framework

• https://github.com/nbio/st

Golang excellent way to un-marshal data stream ( JSON) into struct

• https://github.com/mitchellh/mapstructure

consul , vault , registrator

All these products are from hashicorp.

consul : a ready made datacenter tool for service discovery ( finding applications coming up on nodes on data center) .

• Each data center MUST run atleast one consul server
• Each node MUST run a consul agent

How to run a simple cluster of consul servers ( as docker containers )

• docker run -d --name consul-node-1 -h consul-node-1 -p 0.0.0.0:53:8600/udp -p 8500:8500 -p 8400:8400 gliderlabs/consul-server -node consul-node-1 -bootstrap -advertise 10.0.0.2 -client 0.0.0.0 
• JOIN_IP="$(docker inspect -f '{{.NetworkSettings.IPAddress}}' consul-node-1)" docker run -d --name consul-node-2 -h consul-node-2 gliderlabs/consul-server -node consul-node-2 -join $JOIN_IP
• docker run -d --name consul-node-3 -h consul-node-3 gliderlabs/consul-server -node consul-node-3 -join $JOIN_IP 
• docker run -d --name consul-node-4 -h consul-node-4 gliderlabs/consul-server -node consul-node-4 -join $JOIN_IP

 docker ps

CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS                                                                                                          NAMES

8b1fb4cd3c09        gliderlabs/consul-server   "/bin/consul agent -s"   16 minutes ago      Up 16 minutes       8300-8302/tcp, 8400/tcp, 8500/tcp, 8301-8302/udp, 8600/tcp, 8600/udp                                           consul-node-3

8c5a59c2f878        gliderlabs/consul-server   "/bin/consul agent -s"   18 minutes ago      Up 18 minutes       8300-8302/tcp, 8400/tcp, 8500/tcp, 8301-8302/udp, 8600/tcp, 8600/udp                                           consul-node-2

cdf06afbe5fc        gliderlabs/consul-server   "/bin/consul agent -s"   20 minutes ago      Up 20 minutes       0.0.0.0:8400->8400/tcp, 8300-8302/tcp, 8600/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp, 0.0.0.0:53->8600/udp   consul-node-1

I am writing this page to collect all my findings and share with team.

Hope it helps, happy reading!!!

I am working on

• setting up hashicorp's consul and vault . 
• Write a vault client as part of PEP

• How to start consul locally in your VM ( i have a ubuntu on my parallels )
  • Each data center MUST run atleast one consul server
  • Each node MUST run a consul agent
  • Running consul-server as a docker container
    • docker run -d --name consul-node-1 -h consul-node-1 -p 0.0.0.0:53:8600/udp -p 8500:8500 -p 8400:8400 gliderlabs/consul-server -node consul-node-1 -bootstrap -advertise 10.0.0.2 -client 0.0.0.0
  • Running consul-server as a standalone binary on your host
    • Get appropriate package download it ( i have it on my /usr/bin) 
  • This starts hashicorp's consul , UI : http://127.0.0.1:8500/ui
  • Running a consul cluster of docker containers

• • • docker run -d --name consul-node-1 -h consul-node-1 -p 0.0.0.0:53:8600/udp -p 8500:8500 -p 8400:8400 gliderlabs/consul-server -node consul-node-1 -bootstrap -advertise 10.0.0.2 -client 0.0.0.0 
    • JOIN_IP="$(docker inspect -f '{{.NetworkSettings.IPAddress}}' consul-node-1)" docker run -d --name consul-node-2 -h consul-node-2 gliderlabs/consul-server -node consul-node-2 -join $JOIN_IP
    • docker run -d --name consul-node-3 -h consul-node-3 gliderlabs/consul-server -node consul-node-3 -join $JOIN_IP 
    • docker run -d --name consul-node-4 -h consul-node-4 gliderlabs/consul-server -node consul-node-4 -join $JOIN_IP

• How to start a vault pointing to above consul
  • vault can be downloaded directly as a packaged binary ( 64-bit )
  • a simple vault configuration could be as below 
    • backend "consul" { address = "127.0.0.1:8500" path = "vault" advertise_addr = "http://127.0.0.1" scheme = "http" } listener "tcp" { address = "0.0.0.0:8200" tls_disable = 1 }
  • starting vault server
    • export VAULT_ADDR=http://127.0.0.1:8200
      • vault status Error checking seal status: Error making API request. URL: GET http://127.0.0.1:8200/v1/sys/seal-status Code: 400. Errors: * server is not yet initialized
        
        ubuntu$ vault init 
        Key 1: 5fea1643a59b41cea982ce684baf2399851d3423483a1deccfb59a22d315875901 
        Key 2: bdf7d80022697c44e9d50c5318cae72926f69008c1aef3b02f82d4f44c438eb502 
        Key 3: ad7e6ca87d8bb3df70549f2062fafd6532ce0c2cbb4543b960eb515fb60fe6b703 
        Key 4: d5127290d248b68774a554053ae11bbdae22b41ed3d185480166d7b3fd53df1904 
        Key 5: c59bc6388daa791ced24c77640d101f1ba1a283aa93a35414e0f5218071fb71b05 
        Initial Root Token: 97a7735a-0b7d-7a40-b51e-779904e85d8b Vault initialized with 5 keys and a key threshold of 3. Please securely distribute the above keys. When the Vault is re-sealed, restarted, or stopped, you must provide at least 3 of these keys to unseal it again. Vault does not store the master key. Without at least 3 keys, your Vault will remain permanently sealed.
  • Writing secrets into vault
    • vault has to be authenticated and unsealed before any operation can be done 
    •  $vault auth 97a7735a-0b7d-7a40-b51e-779904e85d8b
    • $vault write secret value=secret
• References
  • https://www.vaultproject.io/intro/getting-started/first-secret.html
  • https://www.vaultproject.io/docs/http/

hashicorp : consul

Consul: Microservice Enabling Microservices and Reactive Programming from Rick Hightower

Docker and ip tables

Docker and fire-walling

• Docker daemon makes an entry into host iptables everytime when a container wants to expose a port on host
• https://fralef.me/docker-and-iptables.html
• There are open source projects which can wrap this and prevent by making a generic entry for ALLOW of this port.
• Code : docker-fw, dfwfw
• Weave network for docker 
• Excellent docker 

Look at docker-fw or dfwfw : these are nothing but the docker daemon / engine interceptor and it provision some iptables entries in between , the best part is these services run as a docker image in privileged mode and does the magic , also include DNS servers.

Containers : are not VMs , container are of course secured

A recent video debate on container technology and where we are at ( i see this video got recorded on October 2015 )


A clear spell-out of difference between virtual machines and containers , worth a watch
Hear to the myths on containers

• containers are NOT VMs
• containers are NOT secured 

More follow up

• ACI ( App Container Image , something like Amazon AMI ) - coreos comes up with rkt. 
• OCI ( Open Container Initiative )

Open Container Initiative: Home

borg, omega and now the open source version of it kubernetes

Excellent articles i came across recently

How container technologies kicked in by google ( borg, omega and now the open source version of it kubernetes )

• https://www.opencontainers.org/
• https://github.com/opencontainers/runtime-spec/blob/master/ROADMAP.md
• http://queue.acm.org/detail.cfm?id=2898444

Terminal emulator, terminal multiplexer(tmux)

I am working on a mission to integrate a sharing terminal (TTY) , pseudo-tty (PTTY) for my cloud containers.

The game plan is to orchestrate the cloud container to launch with special capability( program ) and access the terminal for debugging from the web browser.


There are bunch of elegant solutions
A nodes based solution https://github.com/chjj/tty.js
But the pain is to dockerize nodejs (https://coding.net/u/duwan/p/docker-web-terminal/git/blob/master/package.json)



More interesting : sharing shell over internet tmate(https://tmate.io)
References
https://www.youtube.com/watch?v=B8lS5Nl_TGI

Docker Monitoring Tools

I came across recently this article on monitoring options for docker world

http://rancher.com/comparing-monitoring-options-for-docker-deployments/

A Quick Summary looks like below

1. Docker Stats
2. CAdvisor ( https://github.com/google/cadvisor)
3. Scout
4. Datadog ( Cloud based )
5. Prometheus - http://prometheus.io 
1. container exporter : https://github.com/docker-infra/container_exporter
2. https://hub.docker.com/r/prom/prometheus/
3. Prometheus Dashboard Builder

Prometheus' main distinguishing features as compared to other monitoring systems are:

• a multi-dimensional data model (timeseries defined by metric name and set of key/value dimensions)
• a flexible query language to leverage this dimensionality
• no dependency on distributed storage; single server nodes are autonomous
• timeseries collection happens via a pull model over HTTP
• pushing timeseries is supported via an intermediary gateway
• targets are discovered via service discovery or static configuration
• multiple modes of graphing and dashboarding support
• federation support coming soon

Inspiring IOT

Came across a neat open source project for IOT platform, a framework to deal with interactions between IOT agents and server.
http://www.kaaproject.org
https://github.com/kaaproject/kaa


KAA Server is a server component written purely in JAVA

• spring MVC
• NoSQL
• Jenkins
• ...

KAA Client

https://github.com/kaaproject/kaa/tree/master/client/client-multi

kodi a personalized media manager

Kodi | Open Source Home Theatre Software


http://cyberrule.blogspot.com/2015/01/tamilkodi-xbmc-add-on-installation-guide.html