Why should Splunk be your solution?

Maybe you want to use a database, maybe you want to write your own solution. Developers have been doing that for decades, so why try something new?

Why not use a database? Database solutions seem attractive at first. With little data, they are inexpensive, and they initially work. But as the size of your data and users grows, database solutions quickly get very expensive. Oracle didn't become Oracle by charging $19.95. And for all their expenses, relational databases just don't scale in a map-reduce fashion, where you can just plug in new hardware and be up and running in seconds. After the initial honeymoon, you'll soon discover additional problems with a database solution -- your data is dirty, inconsistent, and not homogeneous. Every time you want to fix something you're going to have to rebuild your database? Add new fields, new indexes, etc. All this assumes you first extract all your data into the database scheme.

Maybe a home built solution is the answer? You're a smart developer, how hard can it be to write some records to a disk, read them in and search for them. While you're at it, why not write your own web browser to run your application in, and your own email server as well. The truth is, you're on a small team or by yourself. You're not going to write something that scales to trillions of events, returns results absurdly fast, extracts out dynamic key-values from your data, have a whole statistical and reporting package integrated in, all in a distributed environment. Seriously, we did the work. Enjoy. Be successful. Spend your time making your application as kickass as it can by spending your time on your data and value-add, not on the plumbing!

When shouldn't you use Splunk? If you data is small, if it's format will never change, and if it has no preference for newer data, maybe your should consider using MySQL or storing the data yourself. If your data requires ACID database requirements, such as atomic transactions and multiple record consistency, you might want to consider a relational database. If you data is only binary, such as images or videos, you might want to consider storing the data yourself, although Splunk would be highly appropriate for images and video metadata. For everything else, Splunk is your solution.

So is it really free? If you index less than 500M of data per day, Splunk is free to use, and the license never expires. This limit refers to the amount of new data you can add per day, but you can keep adding more and more data every day, storing as much as you want. You could add 500MB of data per day and eventually have 10T of data, for example, in Splunk. Up until recently that would have been enough to store every twitter tweet ever made, in a splunk instance for free. If you need more than that, you'll need to purchase a license.