it is all about the Javascript

So there I was, on an interview with a cool company called Clevertech, with a pleasant lady named Rena, and the question of Javascript came up, and I mentioned that I saw basically most of their dev team was Javascript developers, and I said something like 'I guess they are doing a lot of front-end'... Not sure why I said that, knowing full well that we already live in a world where Javascript is taking over the back end, but you know sometimes things just come out, and I probably won't get that job anyway but at the end of the day I'm never sure which is which in terms of good vs. bad. Quite to the contrary - when I think I've done well, I've often done rather poorly, and when I think I messed up a bit that is often when I get an offer. But either way, JS is taking over the world, and unless I want to hang my hat on being a Rails dev forevermore (please, no?) -  then I had better get with the program. And now I am doing a little 'code project' for a prospective employer, and pardon my French but I love that shit, because no one ever says no once they see my code. I am a poet who writes code, and that means I care about every line.

So anyway I am working my way through a Node.js proof of concept, and seriously I am having more fun than I remember having in a long time (Node + Express + EJS). Coding is fun, anyone will tell you that, that is how we all get hooked; but if you keep your head down long enough you may forget it is art at the end of the day. I can't tell you how excited I am to get started on something new. I've got a lot more interviews coming up and it all just feels really good right now.

It is really strange to be back on the market, so to speak, it has been almost a decade now since I've been looking for full-time work. But I feel really ready right now for whatever comes my way. And I have until next Friday to deliver on a pretty simple POC in Node.js, and I'm telling you, that app is going to fart rainbows by the time I'm done with it. :)

the reverse interview

So after 20 years of doing this, I've learned a few things about the interviewing process. One of those things is the concept of a reverse interview.

For my example, I am going to use the interview I did for a company called Recurly (in San Francisco), they were based in the Mission near where we used to live and I was excited to finally get one of those 'warehouse' dev jobs, like I used to pine for in LA when I used to live east of Hollywood with my view of the downtown skyscrapers. So I get the interview, I talk to Scott, their lead dev, my prospective 'boss', and we hit it off like gangbusters. Seriously. I was super-excited to work for this guy.

And then he says, one last thing - we need our boss to sign off on you. He's going call you tomorrow. He's going to do an online 'code' interview. We'll call Scott's boss Steve. So Steve gets on the phone, hey how you doing, I am going to ask you how to code a binary tree, we need a left method and a right method and some sort of object. Pick a language. We were using a screen-sharing service, it might have been a shared google doc or something. I could type and he could see it. And I'm saying to myself, 'sweet! I just learned about data structures at Stanford - this is going to be easy'.

I had just finished a class about structures and algorithms at Stanford with Cynthia Lee, and although we used C++ for that I decided to write something with Ruby, because that's my go-to. And I decide to write an implementation using recursion, because had just finished that section with Cynthia and I was excited to show off.

ME: Ok, well the first thing we need is a base case.

STEVE: can you start with the left and right methods?

ME: OK...

and it all went downhill from there. Basically, if you don't understand recursion, it can't be explained to you in a few minutes. It's like the concept of limits - you need some time to wrap your brain around it. And it became clear that Steve did not really understand recursion, or worse yet, thought he did but hadn't put it into practice.

I did not get that job, and for a while I was angry because I really liked talking to Scott, but then I realized, that Steve guy is running the ship. And he was a mess. Put simply: he decided to veto a candidate that his own manager had already approved, based solely on his own limited understanding of Ruby, and recursion, and his inability to see his own limits (pun intended).

So that's what the reverse interview is - if you are not paying attention, as an interviewEE, to all of the details available to you in terms of what kind of culture exists at the company, then you are missing a huge opportunity to vet the organization before you are tied down. I believe great organizations are easy to spot - as are uneasy ones. Most of the time all we have to do open our eyes, and pay attention.

EDIT: I was shooting for something like this implementation but I didn't think to rope in the enumerable module.

some needed perspective

So what happened exactly?

Screen Shot 2017-11-03 at 1.09.54 PM.png

Clearly, they appreciated me.

So without going into too many specifics, let me explain some of what happened. Perhaps this kind of insight will be valuable to someone in a similar position.

The Need For a Good CTO

I remember years ago, when the original company I had helped build from scratch was bought with leveraged Wall Street money by the cats over at Topps Digital Services, thinking that perhaps things would change. I had hoped for the better. At that point we were young and scrappy and there were no managers, just AEs and artists and engineers. It was a good time. Then they brought in Tom (the name was changed).

Now Tom is still running that ship, and it's been I think close to 6 years. Our growth has been incredible. We are 20x of our original size with much more growth to come. Tom is currently listed as the CEO, the CEO, and CFO. What is missing? The CTO. Literally we don't have one. Never did. What we had was me, the Engineer.

I built the entire platform, from scratch, 3 times over: the prototype, I did a complete refactor for the move to Rails 3.x, and another complete refactor which was never launched. I think Tom felt challenged by me, so instead of either promoting mo to CTO, or hiring one, he just left it alone. He thought he could do the job. Basically because the Engineer was so good, for 9 years we didn't have a CTO. But we were racking up technical debt, which is a concept that Tom didn't understand. So after giving free hours year after year to keep the TD at bay, I started having children and I didn't have those free hours to give. I demanded a bonus and got it, and I tried to stop putting in unpaid hours, but I couldn't.

The Hard Concepts

Here is why you need a good CTO - sometimes there are concepts that are hard to understand. Concepts that you can't bluff your way through, especially when it comes to trade-offs. Decisions are hard to make. But you still have to make them. We failed because of a lack of leadership. I pitched a BI product (Periscope) that proved to be woefully inadequate for what we needed (white-label embeds), and 4 months into the implementation I sounded the alarm, and we needed to pivot. We either refactor the implementation, or we choose a new product. A or B. I made the case over and over again, and no one could put a finger on the map. I was not empowered to decide, yet no one else could make the decision. Tom was in Europe for most of the year and was pretty much MIA for the whole summer. We lost a whole quarter to a bad product, but because no-one could enable a pivot, we threw good hours after bad. And then it was too late. If you don't have someone in a leadership position that understands, at a minimum:

  1. What a stack is
  2. what technical debt it, and why refactoring is important (hint: it has to do with needs changing)
  3. how all decision makers need to be managed in order to achieve a functional SDLC
  4. how complexity growth and scaling needs to impact infrastructure

then your Engineer is not going to be very successful, no matter how hard he tries.

The Last Straw

On the last call I had with Tom, he complained about a 'bug' from months ago. One that had been fixed, but that had not yet been re-tested by QA. The feedback loop between Dev, the users, and QA was so slow that he had not had an update from them in 3 MONTHS. That is just not an environment conducive to anything productive at all. I carried that stress for as long as I could, because I love working in the BI space and I loved working with people from Facebook, and StubHub, and Netflix. But as long as Tom is where he is, knowing as little as he does about what his engineering team needs, that is no place for me.

and on a lighter note

I spent a week making an album in Austin this October and it was better than anything I could have thought up. Seriously. Sometimes releasing control is the best possible approach. I worked with a producer who is a good friend of mine, Shawn Pander. HE KILLED IT.

with regard to Russian hackers

We have all heard of the boogeyman, right? He's for scaring kids to sleep. Russian hackers: they're for redirecting anger towards the unknown, and obfuscating the real issues with our 2-party system. Hopefully everyone has also heard of Pascal, and what is now known as Pascal's Wager. The question is, should one believe in God?

Basically, believing in God, per Pascal, is the smartest decision to make, because it avoids the only  possible 'infinite loss' which is that you don't believe in God yet he/she is in fact, real.

Basically, believing in God, per Pascal, is the smartest decision to make, because it avoids the only  possible 'infinite loss' which is that you don't believe in God yet he/she is in fact, real.

Great, let's try this with Russian hackers. It's a bit different, because our choice is one of priority, but it still looks similar:

fix hacking firstfix hacking 2nd
fix DNC fraud firstn/atakes care of itself
fix DNC fraud secondwe still have a problemn/a

Look, I get it. I don't like the idea of anyone hacking our anything, especially if it's a little bit true. But to quote GoTThe night is dark and full of terrors...

Seriously, there are a lot of problems with the world today. They all need to be addressed. I agree. It is a question of priority. Here are my premises:

1 - we as a nation are basically able to obsess over only 1 thing at a time

2 - Bernie would have won if the DNC primary had been fair

I don't think either of those things are even up for debate anymore. The ability to avoid Dr. Orange was baked right into the system. It was the DNC that failed us. Russian hackers were not and are not enough to sway the election, even if they do exist. If you grant the truth of those 2 things, THE ONLY DISCUSSION we should be having right now is how to either create a viable third party in this country (Bernie, I am looking at you) - or to somehow change the DNC for the better. If we do neither of those things, we are on track for season 2 of DOTUS™.

Coming to you in 2020.

But by all means, let's keep talking about Russia.

 

EDIT: lol, someone sent me this link on how to watch GoT over VPN.

breaking up with MySQL

Breaking up is hard to do. I have been looking at this for over a month now and I'm seeing the writing on the wall.

We are still running 5.6, and we have run into a lot of issues getting the upgrade. Things like:

  1. Apparently 5.7 is so buggy that my PaaS provider suggests we don't do the upgrade.
  2. running 5.6 queries against 5.7 is a massive issue because the Group By syntax has changed, so just to get there every snippet of SQL that uses group by needs to be refactored.
  3. This is a tangent but I am very concerned about ownership of MySQL by Oracle. I am naturally cynical when it comes to private business, as I am a proponent of open-source software. I feel like businesses have become increasingly orthogonal to consumers, basically brushing them aside as they see fit, and over time it just devolves. I like to see a clear path stretching out into the sunset for the core products that I use, and for MySQL, I think those days are over.
  4. Everyone is jumping ship. I don't know anyone using MySQL anymore. Not that there is a significance there, necessarily, because I don't have that many developer friends, and I am not a blind follower, but it does perhaps suggest a simple truth - we have finally outgrown SQL and relational databases in general as the de facto answer for data storage. It is a more nuanced marketplace now.

So my angle will be to facilitate a move away from MySQL over the next months and years. Sounds like I have a lot of research to do. 

MariaDB?

PostGres?

NoSQL?

I am even looking at R right now to replace or augment our BI so things could get pretty interesting. We will see.

nosql-vs-sql-overview-1.png

There's a revolution going on in Business Intelligence part I

Just out of college, I got a job maintaining an Access database for a legal firm in Wisconsin that was vetting different types of records for legal discovery. As a 'temp worker' registered with the local staffing firm, I spent about 10 months writing queries and importing spreadsheets and scanning documents for a team of paralegals. With that experience, I was able to launch a successful software development career on my arrival to Los Angeles in 1999 and through to present day where I own my own 1-man code 'shop'. My main expertise is and has always been dealing with data, and managing the process from OLTP to OLAP and on to business intelligence, which is where a lot of progress is being made right now. Just in the last few years I have seen a preponderance of many new outfits in this space, and that is mostly a good thing but can be a bit wearying to vet so many similar products. I will be writing a few things around this topic so I am labeling this part I.

I am going to wait on doing a top list of anything, or start comparing products. But what I do want to do today is talk about Shiny. Here is a simple proof of concept that I created, where you can select a number of random 'seeds' and the application will automatically create a histogram of the normal distribution of those random numbers.

Now this might not look like much, but for me it is an absolute revolution. First of all, the speed - R is built for what you call 'medium data' analysis - which fits right between the 'small data' and the current buzz-word 'big data' (at least according to Hadley Wickham, chief scientist at RStudio). And even in the OLAP / medium data space, you run into speed issues. Constantly. Different platforms handle that differently, which I will talk about some other time, but right now let me be very clear - R-backed ShinyApp is way ahead as a service, because R is way ahead as a data crunching platform. The advantages vs. SQL are already very clear to me and the truth is that most businesses, large and small, will probably never get to a NEED for big data. And I can't possibly imagine a scenario where it would NOT benefit you to spend a long time playing with your data and creating models in R, even if you plan on going directly to Hadoop or BigTable or Redshift. It's just the right place to start.

And secondly, the workflow for generating this stuff is really revolutionary. It is certainly your best bet for having complete control over presentation at the same time of not having to render all your parameter selections with massive javascript blocks using something like AMCharts or Highcharts, both of which I have worked with extensively, both of which force you to manage a massive amount of custom javascript code inside your views to the point where it just feels messy almost immediately.

And finally, along with the speed of it, the reliability of the shiny app server is great. I have seen nothing to suggest it won't scale, and I think it will soon become a real competitor to the current leader in the white-labelled embeddable BI space, which IMHO is Mode Analytics.

Next up for this series will be to flesh out a bigger shiny app with some more meaningful data, which means I will have to navigate the date selection functionality as input, and use that to rewrite the output. I'm thinking of a day of the year slider, with maybe some heat maps so you can drag across a timeline and watch the daily micro-swings in denomination distribution which might be an aha for somebody if I hook it up to some sales data.