code

running armory on Mac

So I have been party to a sort of bitcoin saga for months now. I had a miner running for a few months in the summer of 2014, and I ended up with about 1.3 bitcoins. At the time it felt like a bad deal, to make barely one coin over three months, so I just forgot about it. Enter this whole crazy crypto run-up at the end of this year and I started to wonder if that wallet was still doing all right. As it turns out, it's doing fine. It's getting money OUT of the wallet that is proving difficult.

You see, the blockchain is now around 160GB and growing. Armory then runs an index database along side, so I'm not sure if a 256GB SSD is enough to comfortably host a BTC wallet now. Plus you need at least 8GB of RAM and a decent processor; and of course a good internet connection. I had a little Dell Inspiron that I thought would do the trick, but man it was slow. 2 weeks to get the blockchain downloaded. Another few days to run the Armory indexes. And then, because Windows, the McAfee sales popup crashes the app and corrupts the index. Reinstall the OS. Repeat.

Anyway, it sort of became my white whale, until I realized that the hardware was just not good enough.

Now, I have this workhorse laptop that I had situated for running SQL and importing our massive database, it has tons of RAM and and large SSD and it is perfect for finally pulling up this wallet and getting those BTC out of there. I want all of my bitcoin to be ready to sell if the price gets too much higher.

So the blockchain downloaded over night; took barely 12 hours. Armory was difficult however. Here's what you do.

  1. Get version 0.95.1
  2. you will need to have HomeBrew installed.
  3. run xcode-select --install if you haven't done that already

  4. make sure your ssl is up to date: brew upgrade openssl

  5. make this edit to ArmoryUtils.py:
Screen Shot 2017-12-21 at 3.12.33 PM.png

and that is all you need to get Armory running. 

breaking up with MySQL

Breaking up is hard to do. I have been looking at this for over a month now and I'm seeing the writing on the wall.

We are still running 5.6, and we have run into a lot of issues getting the upgrade. Things like:

  1. Apparently 5.7 is so buggy that my PaaS provider suggests we don't do the upgrade.
  2. running 5.6 queries against 5.7 is a massive issue because the Group By syntax has changed, so just to get there every snippet of SQL that uses group by needs to be refactored.
  3. This is a tangent but I am very concerned about ownership of MySQL by Oracle. I am naturally cynical when it comes to private business, as I am a proponent of open-source software. I feel like businesses have become increasingly orthogonal to consumers, basically brushing them aside as they see fit, and over time it just devolves. I like to see a clear path stretching out into the sunset for the core products that I use, and for MySQL, I think those days are over.
  4. Everyone is jumping ship. I don't know anyone using MySQL anymore. Not that there is a significance there, necessarily, because I don't have that many developer friends, and I am not a blind follower, but it does perhaps suggest a simple truth - we have finally outgrown SQL and relational databases in general as the de facto answer for data storage. It is a more nuanced marketplace now.

So my angle will be to facilitate a move away from MySQL over the next months and years. Sounds like I have a lot of research to do. 

MariaDB?

PostGres?

NoSQL?

I am even looking at R right now to replace or augment our BI so things could get pretty interesting. We will see.

nosql-vs-sql-overview-1.png

There's a revolution going on in Business Intelligence part I

Just out of college, I got a job maintaining an Access database for a legal firm in Wisconsin that was vetting different types of records for legal discovery. As a 'temp worker' registered with the local staffing firm, I spent about 10 months writing queries and importing spreadsheets and scanning documents for a team of paralegals. With that experience, I was able to launch a successful software development career on my arrival to Los Angeles in 1999 and through to present day where I own my own 1-man code 'shop'. My main expertise is and has always been dealing with data, and managing the process from OLTP to OLAP and on to business intelligence, which is where a lot of progress is being made right now. Just in the last few years I have seen a preponderance of many new outfits in this space, and that is mostly a good thing but can be a bit wearying to vet so many similar products. I will be writing a few things around this topic so I am labeling this part I.

I am going to wait on doing a top list of anything, or start comparing products. But what I do want to do today is talk about Shiny. Here is a simple proof of concept that I created, where you can select a number of random 'seeds' and the application will automatically create a histogram of the normal distribution of those random numbers.

Now this might not look like much, but for me it is an absolute revolution. First of all, the speed - R is built for what you call 'medium data' analysis - which fits right between the 'small data' and the current buzz-word 'big data' (at least according to Hadley Wickham, chief scientist at RStudio). And even in the OLAP / medium data space, you run into speed issues. Constantly. Different platforms handle that differently, which I will talk about some other time, but right now let me be very clear - R-backed ShinyApp is way ahead as a service, because R is way ahead as a data crunching platform. The advantages vs. SQL are already very clear to me and the truth is that most businesses, large and small, will probably never get to a NEED for big data. And I can't possibly imagine a scenario where it would NOT benefit you to spend a long time playing with your data and creating models in R, even if you plan on going directly to Hadoop or BigTable or Redshift. It's just the right place to start.

And secondly, the workflow for generating this stuff is really revolutionary. It is certainly your best bet for having complete control over presentation at the same time of not having to render all your parameter selections with massive javascript blocks using something like AMCharts or Highcharts, both of which I have worked with extensively, both of which force you to manage a massive amount of custom javascript code inside your views to the point where it just feels messy almost immediately.

And finally, along with the speed of it, the reliability of the shiny app server is great. I have seen nothing to suggest it won't scale, and I think it will soon become a real competitor to the current leader in the white-labelled embeddable BI space, which IMHO is Mode Analytics.

Next up for this series will be to flesh out a bigger shiny app with some more meaningful data, which means I will have to navigate the date selection functionality as input, and use that to rewrite the output. I'm thinking of a day of the year slider, with maybe some heat maps so you can drag across a timeline and watch the daily micro-swings in denomination distribution which might be an aha for somebody if I hook it up to some sales data.

Holy Crap It's March - or How I Made My Lenovo Flex 4 a Ruby Slave

I just have not been writing. I can’t believe it’s March. These last few months have flown by. We had a flood in the basement, which is stressing me out a bit, and I had to postpone doing my taxes this year because Wells Fargo screwed up my paperwork, so that’s a bummer.

And I spent a few weeks prepping my laptop for a trip to San Francisco, and we had to cancel because everyone got sick. But we are heading over this weekend instead, so that’s good.

I guess I’ve just had a lot on my mind. My friend Mike passed away sort of suddenly from cancer, and that is just a tough pill to swallow. I feel super guilty because the last time I saw him he had his guitar out behind The Hotel Cafe, and was playing the blues, I was there for Sally Jaye’s release for Amarillo, and I just felt like he was going to be there forever. And now that was 10 years ago and that is the last memory I have of him. So sad.

And after looking at our finances for last year, we really needed to pull in the belt a bit. So when I was looking at getting a new laptop, for the first time ever, really, I seriously considered getting a Windows laptop instead of a MacBook because the price differential is pretty much out of control.

I settled on a Lenovo Flex 4 / Yoga, which ran me just under $600. The cheapest MacBook with the same specs would have been close to triple that. I just couldn’t justify that expense.

I set it up in about 2 days, and now it is running a Rails 3.2 stack and it is a fully functional work laptop. It’s amazing how far things have come.

So if you want to do your Rails Development on a laptop that is not a Mac, perhaps this list of steps can help you a bit.

1 - Windows 10, or nothing. You need Windows 10, because it is the Anniversary Upgrade for Win 10 that introduced something called WSL (https://blogs.msdn.microsoft.com/wsl/) ‘Windows Subsystem for Linux’, and that is what makes this all possible.

2 - You cannot access localhost from WSL, but the ip 127.0.0.1 works just fine. And Windows properly can access localhost inside WSL. So it goes like this:

Database - lives in windows, accessed by WSL server via 127.0.0.1

Browser, lives in Windows (I use Chrome), accesses WSL server via localhost

server, runs in Ubuntu

Editor, lives in Windows, accesses linux docs w symlink 

3 - Apps. You might be worried about what apps are available, but it seems like a decent environment at this point. I love my Mac apps - Fetch, Sequel Pro, Textmate - but there are perfectly decent replacements available. And for any old Textmate people, like me, you might want to take a look at Atom. The auto-complete works really well and reminds me of my old Visual Studio days inside the Microsoft machine.

I use - 

Cmder (http://cmder.net)

Atom (https://atom.io)

mysql command line client for Windows (https://dev.mysql.com/downloads/installer/)

NOTE for MySQL - the 32-bit installer does indeed install a 64-bit version if you want. That was not documented at all that I could see, and if you are looking for a 64-bit installer you are not going to find one.

Sqlectron (https://sqlectron.github.io)

and I haven’t checked yet but I’m sure I can find a decent ftp tool.

I used this walkthrough as a reference.

and the last caveats are that, for ubuntu, only Rails 3.2 and 4.3 are available; and MySQL is only supported up to 5.6 (at least as of a few weeks ago).

All in all, I am very happy with my experience with Windows 10. It has been pretty great so far.

The steps (roughly):

1 - install WSL

2 - test out bash command, set up aliases

3 - follow rvm instructions for installing on linux (use stable version - https://rvm.io/rvm/install)

4 - rvm install Ruby

5 - gem update --system && gem update

6 - sudo apt install git

7 - gem install bundler

8 - symlink linux docs folder back to windows documents -  (ln -s /mnt/c/YOURUSERACCOUNT/Documents ~/Documents)

9 - set up new key gen (I used the GitHub link as reference: (https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/)

10 -start trying bundle install, use apt-get as needed

for example: sudo apt-get install mysql-client-core-5.6, sudo apt-get install ruby-railties-3.2, sudo apt-get install ruby-dev, sudo apt-get install build-essential g++, sudo apt-get install libmysqlclient-dev will all be necessary

11 - gem install mysql2 -v '0.4.5'

12 - gem install activerecord-mysql2-adapter

and then the occasional sudo apt-get upgrade should keep you running.

good luck!

UTF-8, ASCII, and that horrible thing called FTPS

So I need to rant a bit here, but also I'm hoping some of this info will help some people out because I realized there is not much info out there on this problem.

PART I: FTPS

Once upon a time, they were inventing ways to connect with other servers and distribute information. FTP was an early version of this kind of server to server protocol. It worked great, but was unencrypted. So SFTP was born. It's a great example of the power of open source. The SFTP library is full-featured and very stable. So why would anyone want to use anything else?

Honestly, I don't know the answer to that. But Microsoft decided to build another thing we don't need. Instead of getting on board, they decided to roll their own thing and create FTPS. Now if you run Microsoft Servers and ASP.NET or what have you, it's a non issue. But if you want to talk to anything open source, you are going to have to bridge that gap somehow. And what I found out was that there are almost no options for you.

you've got: the double bag gem

this 4 year old Ruby hack.

and the ftpfxp gem, which I won't even link to because it didn't work at all for me.

So really, slim pickings. First lesson - if anyone asks about FTPS, tell them to run like hell. I was able to get something going with the double bag gem but it is ugly. It occasionally throws errors for no reason. I've tried Implicit, Explicit, and it is just a buggy protocol I guess. so there it is. SFTP just works. FTPS is a mess. How's that for a triumph of open-source over big Billysoft. 

PART II: ASCII characters

Once upon a time, we were figuring out how best to digitally represent characters, and in the beginning we had ASCII. But then Unicode was invented because standardization is our friend and pretty much everyone moved to UTF-8 except for, you guessed it, Microsoft. So here we are again. The data I get from these horrible FTPS servers every so often throws ASCII-8BIT characters at me. Just to piss me off, I think. It will bring the whole ingest to a halt. So here's how I handled that.

step 1, get the data. Sounds easy but if you have any non-UTF-8 characters in there, the download will fail on a call to 'gettextfile'. So just download it as a binary file. You're still going to have problems parsing it, but for now just get the damn thing to download.

step 2, remove the offending characters. There is by definition no way to represent non-UTF-8 characters in strict UTF-8, so just dump them. I used a function like this:

def remove_ascii(filename)
  outfile = ""
  s = File.open(filename, 'r:ascii-8bit') do |o|
    o.each_char do |x|
      begin
        outfile << x.encode(Encoding::UTF_8)
      rescue
      end
    end
  end
  File.delete filename
  output = File.open( filename,"w" )
  output << outfile
  output.close
end

and we are done. I don't feel particularly good about, it's not an elegant solution but I shouldn't be getting those characters in the first  place.

Anyway, good luck out there!

Help El Capitan crushed my developer dreams

It's funny, every time Mac launches a new version, even after having been burned over and over again I still fall right into their trap. Every time, I say to myself they must have figured it out this time. They can't possibly do as badly at this as they did last time... And in the end, for as long as I can remember, every time they launch a new OS, I install too early and I lose days of my life. It is just that simple. What did old shrubby say?

Fool me once, shame on you.. Fool me twice? Can't get fooled again!

Except I did get fooled, again. So I installed El Capitan last night and so far:

  1. It has killed my Exchange account for work (although frankly Microflop should take equal blame).
  2. It killed my git install.
  3. It killed my MySQL install.

So I had to fix it. Here is your answer key:

1 - hello webmail! Outlook webmail is surprisingly nice so I punted on that one.

2 - git just had to be reinstalled from binary at https://git-scm.com/.

3 - mysql - the folders just needed rights again. I was seeing this message:

Warning the user/local/mysql/data directory is not owned by the mysql user

so I just did this:

  • sudo chown -RL root:mysql /usr/local/mysql
  • sudo chown -RL mysql:mysql /usr/local/mysql/data
  • sudo /usr/local/mysql/support-files/mysql.server start

Also, before you can do any of this, you need to disable System Integrity Protection and perhaps also reset your /usr/local settings. Read about that here.

The gist of it is:

restart using command+R and run:

csrutil disable

rebeoot and fix permissions:

sudo chflags norestricted /usr/local && sudo chown $(whoami):admin /usr/local && sudo chown -R $(whoami):admin /usr/local

and that should just about do it.

oh man upgrades are the worst

Really. Sometimes progress just thunks you on the head. I have been supporting this Rails 2.3.x project for about 8 years now and this weekend looks like the one where I upgrade my Ruby version, finally, from 1.8.7 to 2.0.0 because our new host doesn't even support 1.8.7 OR 1.9.3. I had to jump two upgrades. Damn.

The good news is that because I have always been stingy about using gems for anything my upgrade path looks pretty seamless so far. If I can pull this off I will be very proud. Finally getting onto continuous integration. I am leaving the rails version alone though. A framework update might actually kill us. So I'm looking at Rails LTS. And if I can get us on Ruby 2.0.0 we will be current again. For 2 years I've wanted to move to EngineYard, and now we are here. I've got a lot of work to do and I am thrilled.

Here are some links that were helpful:

linux brew

upgrading Ruby

drone.io

and of course, rvm gemsets. Seriously, thank you.

UPDATE: actually, the Rails 2.3.x launch failed. Utterly. This is what happened with the old stack too - it's always the app servers that go. The support just falls off the edge of the world and the Rails platform just has to upgrade. It has to. In this case it was nginx that now needs to play nice with unicorn. So I'm going to 3.2.22 now. Yes, my workload just doubled. Yes, I have about 60 different form_for_remote and link_to_remote and :update => "some_id" calls that have to change. But I needed to clean house on my javascript libraries anyway.