Adventures in programming and data.
-
ASX 200 Market Cap
I recently came across this /r/dataisbeautiful post and was wondering what the equivalent would look like using Australian stocks. I found historical market cap data for the ASX 200 and using D3 created a similar visualisation, experimenting with D3’s transitions, scales, areas and axis. Source code on GitHub.
-
Websockets are easy
I was interested in getting a minimal example working for Websockets and it was surprisingly easy to get a demo working between Node.js and a browser. First install the ws library: Create an index.js file with the contents: Start the Node.js process: In a browser console, send a message to the Node.js process:
-
Re-rendering map layers
I recently was optimising the performance of a Leaflet-based map that rendered TopoJSON layers via Omnivore. The layers were a visualisation using the ABS’s Postal Areas, and while there was only a single TopoJSON file, this resulted in a number of feature layers being displayed, with each bound to their own data. The data for…
-
Templated SQL
I’ve recently been working on a data migration SQL script that performs the same operations against numerous tables. While I could use dynamic SQL – where SQL statements are built as a string and then executed – I don’t particularly like the downsides: It cannot be checked at compile time It is difficult to read,…
-
Leaflet and Google Maps
I’ve recently been developing an application that uses Leaflet to interactivity with a geographic map. One of the business requirements was to use Google Maps as a basemap, since it is pervasively used by our customers. A naive implementation used Leaflet’s tileLayer to render the tiles directly: This approach, while simple, does not conform to…
-
Organise photos by date
I organise my photos by grouping them in a directory, where the directory takes on the name “created_date title” (e.g. 2016-05-16 Birthday Party). As it became tedious to create a directory and manually group photo sets, I wrote a Ruby script to assist with the heavy lifting. After that, it’s just a matter of providing…
-
SecurePay SecureFrame example
I recently had to debug an application that uses SecurePay as an online payment system. Rather than using API integration, the application used SecureFrame, where SecurePay provides the payment page (e.g. via an iframe). I couldn’t find an example in the SecureFrame documentation, so here’s a minimal working example:
-
Resetting password of Devise user from the database
I recently encountered a situation where I was trying to access a Rails application but: I didn’t know the password for the admin user The reset password functionality was not working I had no access to a Rails console Knowing the User model was using Devise, I set a password for a user on a…
-
Spatial Analysis with Python
I’ve recently had to undertake spatial data analysis and one of the more challenging tasks was to identify repeat vs unique vehicle journeys. While QGIS is fantastic for visualising routes through heatmaps and density plots, one of my requirements was to provide quantitative metrics around the proportion of repeat vs unique journeys. To achieve this,…
-
Data Category Spheres of Influence
I recently read the article “What is Master Data Management?” by Guy Holmes in the recent (Q2, 2015) PPDM Foundations Journal. There was one statement he made that thought-provokingly captured the definition of master data: “The key difference [from transactional data] is that master data elements tend to repeat in other data sets, and therefore…