Wednesday, December 9, 2015

Docker Monitoring Tools

I came across recently this article on monitoring options for docker world

http://rancher.com/comparing-monitoring-options-for-docker-deployments/

A Quick Summary looks like below

  1. Docker Stats
  2. CAdvisor ( https://github.com/google/cadvisor)
  3. Scout
  4. Datadog ( Cloud based )
  5. Prometheus - http://prometheus.io 
    1. container exporter : https://github.com/docker-infra/container_exporter
    2. https://hub.docker.com/r/prom/prometheus/
    3. Prometheus Dashboard Builder
Prometheus' main distinguishing features as compared to other monitoring systems are:
  • multi-dimensional data model (timeseries defined by metric name and set of key/value dimensions)
  • flexible query language to leverage this dimensionality
  • no dependency on distributed storage; single server nodes are autonomous
  • timeseries collection happens via a pull model over HTTP
  • pushing timeseries is supported via an intermediary gateway
  • targets are discovered via service discovery or static configuration
  • multiple modes of graphing and dashboarding support
  • federation support coming soon

Inspiring IOT

Came across a neat open source project for IOT platform, a framework to deal with interactions between IOT agents and server.
http://www.kaaproject.org
https://github.com/kaaproject/kaa


KAA Server is a server component written purely in JAVA

  • spring MVC
  • NoSQL
  • Jenkins
  • ...

KAA Client
https://github.com/kaaproject/kaa/tree/master/client/client-multi

Tuesday, November 24, 2015

nodejs based web UI Shell Emulator

* Install nodejs in ubuntu
& sudo apt-get update
& sudo apt-get install nodejs
& sudo apt-get install npm
*git clone https://github.com/chjj/tty.js/
& nodejs install tty.js
 & cd bin
&./tty.js --daemonize --port 3000

References :
https://www.digitalocean.com/community/tutorials/how-to-set-up-a-node-js-application-for-production-on-ubuntu-14-04

GPG Errors in ubuntu

W: GPG error: https://apt.dockerproject.org ubuntu-trusty Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F76221572C52609D

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F76221572C52609D
sudo apt-get update

Tuesday, November 17, 2015

Setting up golang proto-gen-go

Here is the step by step directions:
  1. Download protoc-win32.zip from https://developers.google.com/protocol-buffers/docs/downloads
  2. Unzip and add location of the protoc.exe to your PATH environment variable
  3. Run `protoc --version` from command prompt to verify
  4. Verify the your GOPATH environment variable is set
  5. Run `go get -u github.com/golang/protobuf/protoc-gen-go` from command prompt. This should install the binary to %GOPATH%/bin
  6. Add `%GOPATH%/bin` to your PATH environment variable
  7. Open a new command prompt, navigate to your .proto file, run `protoc --go_out=. *.proto` 
NOTE: if you are running from a text editor or ide, you may need to reboot after modifying your environment variables

RabbitMQ and its nuances

I happened to work on AMQP/AMQPS implementation for one of our large server farms.
One side of the RMQ( Rabbit MQ) is in java and other side is in go lang,

Java RMQ Libraries

  • https://www.rabbitmq.com/api-guide.html
The most noticeable challenge was the amount of thread created when using this library, read below.
I had to set up a Executor

import com.google.common.util.concurrent.ThreadFactoryBuilder;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;


private final ThreadFactory threadFactory = new ThreadFactoryBuilder().setNameFormat("RmqExecutor-pool-%d").setDaemon(true).build();

private ExecutorService rmqExecutorService = Executors.newFixedThreadPool(1,threadFactory);

connection = factory.newConnection(rmqExecutorService);

With above code of creating a ThreadFactory and assigning this ExecutorService into    

Advanced Connection options

Consumer thread pool

Consumer threads (see Receiving below) are automatically allocated in a new ExecutorService thread pool by default. If greater control is required supply an ExecutorService on the newConnection() method, so that this pool of threads is used instead. Here is an example where a larger thread pool is supplied than is normally allocated:
ExecutorService es = Executors.newFixedThreadPool(20);
Connection conn = factory.newConnection(es);
Both Executors and ExecutorService classes are in the java.util.concurrent package.
When the connection is closed a default ExecutorService will be shutdown(), but a user-suppliedExecutorService (like es above) will not be shutdown(). Clients that supply a custom ExecutorService must ensure it is shutdown eventually (by calling its shutdown() method), or else the pool’s threads may prevent JVM termination.
The same executor service may be shared between multiple connections, or serially re-used on re-connection but it cannot be used after it is shutdown().
Use of this feature should only be considered if there is evidence that there is a severe bottleneck in the processing of Consumer callbacks. If there are no Consumer callbacks executed, or very few, the default allocation is more than sufficient. The overhead is initially minimal and the total thread resources allocated are bounded, even if a burst of consumer activity may occasionally occur.

Tuesday, July 14, 2015

GO programming Collection

My recent exploration of writing GO programs, i had to refer below

http://www.goinggo.net
http://play.golang.org/
https://www.ardanlabs.com/hardcore-go
https://godoc.org/github.com/capnm/sysinfo
https://plus.google.com/+WilliamKennedy/posts


Friday, June 5, 2015

Compiling protobuf

How to build protobuf

git clone https://github.com/google/protobuf.git
sudo apt-get update
sudo apt-get install autoconf
sudo apt-get install Libtool
./autogen
./configure
make

Build protobuf jars

protoc --java_out=src/main/java -I../src \
     ../src/google/protobuf/descriptor.proto

Wednesday, May 6, 2015

Panamax


  • Panamax is a open source project from Centurylinklabs
  • It provides an user interface to manage docker containers in a host.
  • As i tried now, it comes as a vagrant box image , which has CoreOS + panamax installed on it, few ports are opened to reach out this VM from host
  • Some good are
    • Panamax has a User Interface it lets you see list of containers co-hosted in this VM
    • See stats of containers - hooked up with google/CAdvisor
    • Search docker.io repo for new images - BUT when tried to Add Image it does not work
  • Lot of bad are
    • There are too many github small projects under centurylinklabs and i am not sure what this guys are trying to do? Looks too amateur to me .
    • Code base is too buggy
    • Open Issues are not answered

Docker aka Container management

I am recently playing more with dockers and containers as my job demand it.

Jotting down some useful details here on my learning for general public and for my self records as well.

What is docker?
A Startup company in San Francisco, California , created a simple linux tool to manage linux containers.

Why one need docker?
Like VMWare's virtualization technology is for virtualization .Docker is the answer to provide next generation light weight virtualization technology. Docker is made to ease things out here.

What is a Container?
Container exist for a long time in linux world, it is a instance of another linux inside a linux operating system.

Why we need a Operating System inside another Operating System?
Virtualization is all about making efficient use of  "COMPUTE", "STORAGE" and "NETWORKING" . As new software applications or upgrade are rolled out every second in data centers for modern needs like (social networking, chatting, video, etc... ), a big chunk of compute power is needed. For ease of management of software applications in data centers a smallest possible workload bundle is needed. For which a host ( LINUX OS) should be capable of running multiple smaller guest LINUX OS ( which sounds like the virtual machine concept , but not : containers are super light weight compared to virtual machines.

Tuesday, January 6, 2015

Making sense of SDN

I came across this three part video in youtube, which was recorded back in August 2012, i see it is interesting and useful.






OpenSSL User Interface

An excellent tool for OpenSSL (working version) tool i came across http://sourceforge.net/projects/opensslui/

  • It has a installable windows UI
  • It uses OpenSSL equivalent command to create and manage CA certs, Keys for PKI



References

Network Policy Management

Way too many times we had come across about the complexities with setting up policies / rules in network equipments following their proprietary approaches.

In the end , there is no one-stop-shop ( a CLI or single pane ) to set up this classic multi-vendor problem.

As a simple rule of thumb, a network service policy , when written as words looks like this
"Input trunk + forwarding policy = output truck"

A centralized control traffic management standards is the Open Flow Protocol and managing policies to be applied with distinct network equipment.

An orchestrator project like openstack , drives this thru initiatives like https://wiki.openstack.org/wiki/Congress

Policy Language

The policy language for Congress is Datalog, which is basically SQL but with a syntax that is closer to traditional programming languages. This declarative language was chosen because its semantics are well-known to a broad range of DevOps, yet its syntax is more terse making it better suited for expressing real-world policies. The grammar is given below.
<policy> ::= <rule>*
<rule> ::= <atom> COLONMINUS <literal> (COMMA <literal>)*
<literal> ::= <atom>
<literal> ::= NOT <atom>
<atom> ::= TABLENAME LPAREN <term> (COMMA <term>)* RPAREN
<term> ::= INTEGER | FLOAT | STRING | VARIABLE  

bee-social