Terraform, consul, vault - AWS

I am in a mission to setup my AWS cluster with terraform(similar to chef and puppet) , consul (similar to etcd ,  distributed data store), vault ( a secret store )


BTW, I am on ubuntu linux ( 14.04 trusty )

AWS CLI ?

• http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html

AWS Key-pair?

• http://www.dowdandassociates.com/blog/content/howto-create-an-amazon-ec2-key-pair-using-the-aws-cli/
• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#how-to-generate-your-own-key-and-import-it-to-aws

Terraform?

• https://www.terraform.io

AWS and Terraform?

• A very old dated not working example : http://awsadvent.tumblr.com/post/105835590469/aws-advent-2014-using-terraform-to-build
• A Very hard problem at first to make terraform work
• $aws ec2 --region us-west-2 create-key-pair --key-name terraform --output text 
• This will create a key-pair in us-west-2 region and output will be a public key displayed
• $aws ec2 --region us-west-2 create-key-pair --key-name terraform --output text > terraform.pem
• Create a terraform.pem file out of this THIS DO not work either, because AWS needs only public key and not the private key
• https://alestic.com/2010/10/ec2-ssh-keys/
• With lots of struggle i managed to create certificate and learned a lot
• AWS has changed the way their key-pair works to be compatible with openSSL
• http://docs.aws.amazon.com/cli/latest/reference/ec2/import-key-pair.html
• How to create a local key-pair using openssl and import into AWS correctly?
• #openssl genrsa -out terraform.pem 2048 
• This is a cert , private key
• #openssl rsa -in terraform.pem -pubout > terraform.pub
• Creating a public key , AWS key pair just needs this. But there is a catch this file has unwanted contents (BEGIN PUBLIC KEY , END PUBLIC KEY  etc...)and new lines

• #tr -d '\n' < terraform.pub > terraform.pub1; mv terraform1.pub to terraform.pub
• This is to truncate all new lines, edit it and remove other unwanted text 
• How to load this above key-pair into AWS?
• #aws ec2 import-key-pair --key-name terraform --public-key-material <here goes the copy paste of terraform.pub>
• Remember always the public keys are for the client , in this case here AWS
• How to start the terraform simple application using above key-pairs?
• #terraform apply -var 'key_name=terraform-test' -var 'public_key_path=terraform.pub'
• This takes a lot of time , it created instance and try to connect over SSH to setup terraform playbook.
• 
  

go lang concurrency visually explained

i came across this good blog on concurrency on go

https://blog.codeship.com/visualizing-concurrency-go/

Devops, cloud datacenter , developments June 2016

Recent Developments 


Solutions , training and support
http://container-solutions.com

Stack Community Support
http://mantl.io

Java Heap - clean write up

http://www.ibm.com/developerworks/java/library/j-codetoheap/index.html

golang CI

I came across these important tools and marking it here for references, here

• https://build.golang.org : Dashboard
• http://www.golangbootcamp.com/book/tricks_and_tips

Golang tiny test framework

• https://github.com/nbio/st

Golang excellent way to un-marshal data stream ( JSON) into struct

• https://github.com/mitchellh/mapstructure

consul , vault , registrator

All these products are from hashicorp.

consul : a ready made datacenter tool for service discovery ( finding applications coming up on nodes on data center) .

• Each data center MUST run atleast one consul server
• Each node MUST run a consul agent

How to run a simple cluster of consul servers ( as docker containers )

• docker run -d --name consul-node-1 -h consul-node-1 -p 0.0.0.0:53:8600/udp -p 8500:8500 -p 8400:8400 gliderlabs/consul-server -node consul-node-1 -bootstrap -advertise 10.0.0.2 -client 0.0.0.0 
• JOIN_IP="$(docker inspect -f '{{.NetworkSettings.IPAddress}}' consul-node-1)" docker run -d --name consul-node-2 -h consul-node-2 gliderlabs/consul-server -node consul-node-2 -join $JOIN_IP
• docker run -d --name consul-node-3 -h consul-node-3 gliderlabs/consul-server -node consul-node-3 -join $JOIN_IP 
• docker run -d --name consul-node-4 -h consul-node-4 gliderlabs/consul-server -node consul-node-4 -join $JOIN_IP

 docker ps

CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS                                                                                                          NAMES

8b1fb4cd3c09        gliderlabs/consul-server   "/bin/consul agent -s"   16 minutes ago      Up 16 minutes       8300-8302/tcp, 8400/tcp, 8500/tcp, 8301-8302/udp, 8600/tcp, 8600/udp                                           consul-node-3

8c5a59c2f878        gliderlabs/consul-server   "/bin/consul agent -s"   18 minutes ago      Up 18 minutes       8300-8302/tcp, 8400/tcp, 8500/tcp, 8301-8302/udp, 8600/tcp, 8600/udp                                           consul-node-2

cdf06afbe5fc        gliderlabs/consul-server   "/bin/consul agent -s"   20 minutes ago      Up 20 minutes       0.0.0.0:8400->8400/tcp, 8300-8302/tcp, 8600/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp, 0.0.0.0:53->8600/udp   consul-node-1

I am writing this page to collect all my findings and share with team.

Hope it helps, happy reading!!!

I am working on

• setting up hashicorp's consul and vault . 
• Write a vault client as part of PEP

• How to start consul locally in your VM ( i have a ubuntu on my parallels )
  • Each data center MUST run atleast one consul server
  • Each node MUST run a consul agent
  • Running consul-server as a docker container
    • docker run -d --name consul-node-1 -h consul-node-1 -p 0.0.0.0:53:8600/udp -p 8500:8500 -p 8400:8400 gliderlabs/consul-server -node consul-node-1 -bootstrap -advertise 10.0.0.2 -client 0.0.0.0
  • Running consul-server as a standalone binary on your host
    • Get appropriate package download it ( i have it on my /usr/bin) 
  • This starts hashicorp's consul , UI : http://127.0.0.1:8500/ui
  • Running a consul cluster of docker containers

• • • docker run -d --name consul-node-1 -h consul-node-1 -p 0.0.0.0:53:8600/udp -p 8500:8500 -p 8400:8400 gliderlabs/consul-server -node consul-node-1 -bootstrap -advertise 10.0.0.2 -client 0.0.0.0 
    • JOIN_IP="$(docker inspect -f '{{.NetworkSettings.IPAddress}}' consul-node-1)" docker run -d --name consul-node-2 -h consul-node-2 gliderlabs/consul-server -node consul-node-2 -join $JOIN_IP
    • docker run -d --name consul-node-3 -h consul-node-3 gliderlabs/consul-server -node consul-node-3 -join $JOIN_IP 
    • docker run -d --name consul-node-4 -h consul-node-4 gliderlabs/consul-server -node consul-node-4 -join $JOIN_IP

• How to start a vault pointing to above consul
  • vault can be downloaded directly as a packaged binary ( 64-bit )
  • a simple vault configuration could be as below 
    • backend "consul" { address = "127.0.0.1:8500" path = "vault" advertise_addr = "http://127.0.0.1" scheme = "http" } listener "tcp" { address = "0.0.0.0:8200" tls_disable = 1 }
  • starting vault server
    • export VAULT_ADDR=http://127.0.0.1:8200
      • vault status Error checking seal status: Error making API request. URL: GET http://127.0.0.1:8200/v1/sys/seal-status Code: 400. Errors: * server is not yet initialized
        
        ubuntu$ vault init 
        Key 1: 5fea1643a59b41cea982ce684baf2399851d3423483a1deccfb59a22d315875901 
        Key 2: bdf7d80022697c44e9d50c5318cae72926f69008c1aef3b02f82d4f44c438eb502 
        Key 3: ad7e6ca87d8bb3df70549f2062fafd6532ce0c2cbb4543b960eb515fb60fe6b703 
        Key 4: d5127290d248b68774a554053ae11bbdae22b41ed3d185480166d7b3fd53df1904 
        Key 5: c59bc6388daa791ced24c77640d101f1ba1a283aa93a35414e0f5218071fb71b05 
        Initial Root Token: 97a7735a-0b7d-7a40-b51e-779904e85d8b Vault initialized with 5 keys and a key threshold of 3. Please securely distribute the above keys. When the Vault is re-sealed, restarted, or stopped, you must provide at least 3 of these keys to unseal it again. Vault does not store the master key. Without at least 3 keys, your Vault will remain permanently sealed.
  • Writing secrets into vault
    • vault has to be authenticated and unsealed before any operation can be done 
    •  $vault auth 97a7735a-0b7d-7a40-b51e-779904e85d8b
    • $vault write secret value=secret
• References
  • https://www.vaultproject.io/intro/getting-started/first-secret.html
  • https://www.vaultproject.io/docs/http/

hashicorp : consul

Consul: Microservice Enabling Microservices and Reactive Programming from Rick Hightower

Docker and ip tables

Docker and fire-walling

• Docker daemon makes an entry into host iptables everytime when a container wants to expose a port on host
• https://fralef.me/docker-and-iptables.html
• There are open source projects which can wrap this and prevent by making a generic entry for ALLOW of this port.
• Code : docker-fw, dfwfw
• Weave network for docker 
• Excellent docker 

Look at docker-fw or dfwfw : these are nothing but the docker daemon / engine interceptor and it provision some iptables entries in between , the best part is these services run as a docker image in privileged mode and does the magic , also include DNS servers.

Containers : are not VMs , container are of course secured

A recent video debate on container technology and where we are at ( i see this video got recorded on October 2015 )


A clear spell-out of difference between virtual machines and containers , worth a watch
Hear to the myths on containers

• containers are NOT VMs
• containers are NOT secured 

More follow up

• ACI ( App Container Image , something like Amazon AMI ) - coreos comes up with rkt. 
• OCI ( Open Container Initiative )

Open Container Initiative: Home

borg, omega and now the open source version of it kubernetes

Excellent articles i came across recently

How container technologies kicked in by google ( borg, omega and now the open source version of it kubernetes )

• https://www.opencontainers.org/
• https://github.com/opencontainers/runtime-spec/blob/master/ROADMAP.md
• http://queue.acm.org/detail.cfm?id=2898444