There's a revolution going on in Business Intelligence part I

Just out of college, I got a job maintaining an Access database for a legal firm in Wisconsin that was vetting different types of records for legal discovery. As a 'temp worker' registered with the local staffing firm, I spent about 10 months writing queries and importing spreadsheets and scanning documents for a team of paralegals. With that experience, I was able to launch a successful software development career on my arrival to Los Angeles in 1999 and through to present day where I own my own 1-man code 'shop'. My main expertise is and has always been dealing with data, and managing the process from OLTP to OLAP and on to business intelligence, which is where a lot of progress is being made right now. Just in the last few years I have seen a preponderance of many new outfits in this space, and that is mostly a good thing but can be a bit wearying to vet so many similar products. I will be writing a few things around this topic so I am labeling this part I.

I am going to wait on doing a top list of anything, or start comparing products. But what I do want to do today is talk about Shiny. Here is a simple proof of concept that I created, where you can select a number of random 'seeds' and the application will automatically create a histogram of the normal distribution of those random numbers.

Now this might not look like much, but for me it is an absolute revolution. First of all, the speed - R is built for what you call 'medium data' analysis - which fits right between the 'small data' and the current buzz-word 'big data' (at least according to Hadley Wickham, chief scientist at RStudio). And even in the OLAP / medium data space, you run into speed issues. Constantly. Different platforms handle that differently, which I will talk about some other time, but right now let me be very clear - R-backed ShinyApp is way ahead as a service, because R is way ahead as a data crunching platform. The advantages vs. SQL are already very clear to me and the truth is that most businesses, large and small, will probably never get to a NEED for big data. And I can't possibly imagine a scenario where it would NOT benefit you to spend a long time playing with your data and creating models in R, even if you plan on going directly to Hadoop or BigTable or Redshift. It's just the right place to start.

And secondly, the workflow for generating this stuff is really revolutionary. It is certainly your best bet for having complete control over presentation at the same time of not having to render all your parameter selections with massive javascript blocks using something like AMCharts or Highcharts, both of which I have worked with extensively, both of which force you to manage a massive amount of custom javascript code inside your views to the point where it just feels messy almost immediately.

And finally, along with the speed of it, the reliability of the shiny app server is great. I have seen nothing to suggest it won't scale, and I think it will soon become a real competitor to the current leader in the white-labelled embeddable BI space, which IMHO is Mode Analytics.

Next up for this series will be to flesh out a bigger shiny app with some more meaningful data, which means I will have to navigate the date selection functionality as input, and use that to rewrite the output. I'm thinking of a day of the year slider, with maybe some heat maps so you can drag across a timeline and watch the daily micro-swings in denomination distribution which might be an aha for somebody if I hook it up to some sales data.