Elasticsearch says it's on a mission to make massive amounts of data
available to businesses. Toward that end, in February the company
debuted release 1.0 of its Elasticsearch search and analytics software
for working with big data. The software is part of the company's
open-source ELK stack that includes Elasticsearch, Logstash (log
management) and Kibana (log visualization) software.
Elasticsearch is competing with Solr, the open-source search
technology from the Apache Software Foundation. Many reviewers, users
and solution providers give both technologies high marks. So it will be
interesting to see if both gain traction, or if one or the other wins
out.
- Logstash is an agent to collect and mine logs : https://github.com/logstash-plugins
- Also see sumologic where they claim they have their log management agent in AWS
Risk Focus is an expert implementation partner of Splunk in the
financial services arena and a big proponent of the value Splunk
delivers. We have implemented Splunk in numerous banks, funds and
clearing/exchange firms across multiple mission-critical use cases.
Yet Splunk is not the only log analyzer out there. In particular,
ELK (elasticsearch + logstash + kibana) is a growing open source offering which purports to have the same functionality as Splunk at virtually no cost.
Is this a valid claim? Do Splunk and ELK provide
similar functionality, stability and technical richness needed by
corporate institutions like banks, asset managers, hedge funds,
exchanges, industry utilities and major technology providers?
Cost of Splunk vs. ELK
Let’s start by taking a look at the overall cost of investing in
Splunk versus ELK. Corporate buyers invariably look at total cost of
ownership (TCO) when making their buying decision because they want to
know the true “all in cost” not just the up front fee.
“Splunk is expensive and ELK is free.”
A web search will turn up hundreds of blog entries claiming Splunk’s
pay-per-gigabyte-indexed pricing model is expensive. Splunk data
indexing charges sound pricey, but the way the pricing actually works is
far cheaper than it first appears. Yes, an ELK license is free but
Splunk is amazingly cheap, too.
Underlying the “Splunk is expensive” claim is the assumption that all
data will be indexed, which is rarely true. A proper implementation
includes an up-front analysis such that only the valuable subset of
a company’s data is indexed.
Midsize and larger companies tend to purchase software and
data licenses at bulk discounted rates. This gives a discount off list
price and provides predictability (no “bait and switch” surprises) after
adoption. For less than the cost of a single skilled FTE in a G10
country you can index a huge amount of log data with Splunk across your
IT infrastructure and earn tremendous efficiency cost savings. We’ve
seen Splunk’s rates dropping over time, so it’s getting even cheaper. If
you just need to dabble, do basic development and testing, or a proof
of concept, Splunk offers a free Enterprise license up to 500MB per day.
The primary concern for sophisticated corporate buyers is
cost-to-value or total cost of ownership (TCO). Data license costs (at
least at the pricing level of Splunk or other log analyzers) tend to be
the least important factor in the TCO equation. The value of cost
savings and new revenue discoveries can provide immense financial
value that dwarf the license costs.
“Once we get ELK going it will be cheaper. The time to learn and get ELK figured out is an investment.”
Both Splunk and ELK can be installed and running quickly at a basic
level with minimal learning time. Efficiency gains from reducing “Team
GREP” activities (which never really identify or resolve underlying
issues well), the actionable intelligence obtained (allowing you to
optimize IT spend very quickly), and operational risk reductions all
argue in favor of installing a log analyzer as soon as possible.
Once installed, the question then becomes
“can Splunk or ELK knowledge can be rolled out through an organization quickly?”
Rapidly enabling multiple users to take advantage of indexing,
correlation and dashboarding capabilities is necessary to generate
business value from a log analyzer.
Splunk is far ahead of ELK in speed of roll out and depth of
coverage. Splunk offers a rich education program, a Professional
Services group and an expansive network of skilled consulting partners.
Getting a team Splunk certified takes less than 1 month. Hiring a
Splunk partner firm to roll out capabilities quickly and build advanced
correlation apps can further shortcut the time-to-value. Splunk has a
large App Store with hundreds of free and paid apps to connect to
standard IT hardware and software platforms. ELK is growing rapidly and
is making similar education efforts, but is years behind Splunk in these
critical areas.
“Beware the hidden costs.”
Compared to Splunk, we believe the investment required to
institutionalize ELK is far more time-consuming and costly in terms of
lost efficiencies and investment dollars than buying Splunk. An ELK
user is likely to find that they must create an entire infrastructure
around knowledge transfer, skill-building and connectors to underlying
log sources before rolling ELK out across the firm.These are hidden but
serious costs of choosing ELK. The incremental cost of a Splunk data
license is much lower then the time and costs of building an in-house
knowledge and support structure from scratch around ELK.
For a small organization with basic needs and a small IT support
group, ELK is certainly a good investment. It’s free from a license
standpoint and rewards DIY approaches. If you’re an IT development shop
ELK is an excellent choice.
For large and sophisticated firms, especially financial institutions,
energy companies, defense firms and the like, Splunk is cheaper in the
long run.
Splunk Support vs. ELK Support
Part of the total value comparison between Splunk and ELK is support, both from the vendor and the surrounding community.
“Wait, ELK support isn’t free?”
If you’re going to implement and run mission-critical IT monitoring
tools, then proper support level agreements (SLAs) and enterprise-level
engineering processes are mandatory.
Splunk is a fully integrated indexing and analytics package with
enterprise-level support from both Splunk, Inc. and the huge Splunk
developer community. Buying a Splunk license provides these critical
support items. Splunk has supported thousands of installations
worldwide.
ELK now offers paid support, SLAs, etc. These services are not
free and essentially push ELK into the “freemium” model. Not a bad move
on ELK’s part, just unproven. ELK paid support doesn’t yet have
experience supporting hundreds of large corporations.
“Is our IP protected?”
Splunk has operated a secure support infrastructure for years.
The ELK open source community is very active with support but there
are no data confidentiality, security or IP protections when sharing an
issue with the ELK community. This lack of IP protection does not pass
stringent financial, healthcare or defense industry requirements.
“What if something goes really wrong?”
ELK = elasticsearch + logstash + kibana. These are three different
businesses (aka “projects”) which have a symbiotic relationship.
Relying on three different open source businesses for one solution
carries significant operational and legal risk.
Splunk, Inc. is a major publicly listed company with deep pockets and
well-defined operations. They are a viable and stable corporate partner
presenting little risk.
“Can I trust the vendor’s partners to do solid work?”
Splunk has a rich network of preferred domain experts like Risk Focus
that must go through intensive training and certification. The
installation and app development work Splunk partners do is generally
reliable and meets mission-critical engineering standards.
ELK consultancies are emerging, but the community does not yet have a
disciplined and well-trained network of implementation partners held
firmly to common standards. ELK has no leverage over independent ELK
implementation consultants to follow their standards. Buyer beware.
We are open source proponents, but these issues merit serious
consideration when dealing with complex regulated institutions like
banks, brokers, asset managers and exchanges.
Splunk Apps vs. ELK Plugins
One of the major areas of value when adopting an enterprise-wide
technology is the ability to easily integrate it with your existing IT
infrastructure and applications.
Integration is normally achieved through pre-built adapters, ETL
components, apps, plugins and the like. The more pre-built adapters
available, the faster a company can roll out the technology and obtain
value without spending significant sums on development, testing and
integration.
The
Splunk App Store
is a rich source of free and paid apps which you can use off the shelf
or customize. Splunk’s app selection covers most generic enterprise
software (Outlook, SQL Server, Oracle, etc.), messaging solutions
(TIBCO), databases, servers, firewalls, security applications, gateways,
etc.
ELK has far fewer pre-built apps (plugins).
Elasticsearch
has a number of them. Most are scattered around different vendors.
There is no comprehensive certification process for ELK plugins. ELK
is in major catch-up mode here and adopting ELK means lots of DIY
development to extend it across the enterprise.
To our knowledge there are no financial services app specialists for
ELK and very few, if any, in most other industry verticals. The ELK
community is growing fast so the landscape of specialists will change,
but for now Splunk has substantially more penetration across industry
verticals.
Set name to 
Save the private key somewhere with a passphrase and then copy the public key text to the clipboard.
