Tuesday, 23 February 2016

Why you need a Developer Diary

After eighteen months of use, I can safely claim how useful a a developer diary is, and why you should start using one if you don't already.

Changes

Using a single file grew too large as expected. Instead each grouping is allocated an individual file. While some of these are small, the ability to navigate easily is not lost. The simplest thing to do here is just point a browser at the directory. Navigation and searching is then built in. Each week I actively split the main file file into separate files or merge to existing content.

Uses

I'm increasingly finding regular needs and uses to refer back to notes. In some cases it is quicker than searching online, simply hit the bookmark and start typing.

Adding notes works great when reading books or watching videos. All content recorded ends up in the same location.

Do It

Keeping a developer diary is without doubt one of the most useful changes to my development process. Encouragingly others are doing similar things too, but under slightly different names.

Wednesday, 17 February 2016

Gaining Ten Extra Hours a Week

For a long time my work life balance has gone through phases. Some weeks I would spend hours after work writing code. This would exceed to well beyond midnight in some cases. This phase was not sustainable but it appeared to be the norm.

My other hobbies such as reading and gaming were also neglected. These too suffered from weeks of focus, followed by quieter periods.

Over the course of a typical year this cycle would generally balance out, but never feel comfortable. There were always areas that lacked focus. One of my common complaints which I have shared with many other developers is around the lack of time to do anything additional to the day to day job. There always seemed to be a sacrifice.

Mornings

The book Soft Skills recommends taking the first hour or two out of your weekly day to focus on important goals. This piece of advice was not going to work for me I thought. I was not a morning person.

Sometime after I discovered an article from the Art Of Manliness (AoM) which gave tips on becoming a early riser.

How To

Instead of simply waking up a whole two hours earlier than usual, AoM offered an incremental solution. Set your alarm ten minutes earlier and live with it for a whole week. Each week reduce by a further ten minutes until you hit your target.

This process took about three weeks to see any real benefit. While this can be considered slow progress the act of doing so was incredibly easy. Incrementally reducing the time also provides time for your body to adjust, meaning it's a lot easier to stick with.

Downsides

The obvious downside here is that the two additional hours you gain in the morning are removed from the end of the day. In other words you'll find yourself going to sleep earlier. Having a good balance between work, hobbies and free time means this trade off is well worth the change.

One area this technique cannot much provide benefit is with children. Developers with children would find it difficult to use an uninterrupted block first thing in the morning. Not being a parent myself means I cannot provide any alternatives.

Benefits

Like most things in life, small changes add up to big things over a long period. Thanks to this change I nearly doubled the amount of blog posts I wrote back in 2015 when compared to 2014. I've read more books and watched more technical videos during this period than ever, while still maintaining a healthy balance in other areas of my life. Give it a go, become a morning person.

Wednesday, 10 February 2016

The N+1 Problem

The N+1 problem is when multiple queries are executed against a persistent store when a reduced amount could serve the same purpose. This degrades performance, uses more memory and can cause complexity to be added to the code that processes the results. Most sources of the problem come from the poor use of ORMs or developers thinking procedurally instead of in terms of how the underlying database operates.

Example

Consider a collection of posts that each contain zero or more comments.

  Post
    Comment
    Comment
  Post
    Comment
    Comment
    Comment
    Comment
    Comment
  Post

To retrieve a selection of ten posts including their comments, one option would be to query all posts then perform a query for each individual posts' comments. This would result in a total of eleven queries. While this solution works it is far from ideal. Disturbingly this solution is easily introduced when developers execute queries against databases using loops or misconfigured ORMs.

Solutions

Solutions to solving the N+1 problem are remarkably straightforward. In the case of manual queries such changes are usually easy to implement.

Single Query

Use a join operation to perform a single query. This one query would pull back all posts and their matching comments. This would be the ideal fix for the example described above.

Query and Stitch

Sometimes there is no clear grouping or relation between sets of data. This is often the case when normalized data needs to be denormalized prior to retrieval. In these cases the query and stitch method can be used.

  • One query to grab master set.
  • Another query to grab the related set.

Then simply match on a key in code. The key would be something that groups the data and is present in both sets or is the result of additional programming logic. Query and stitch is useful for paging or when relational thinking and grouping does not fit. This tends to be the case for REST APIs where data is aggregated or composed from multiple sources, or needs further processing after retrieval.

Despite two queries here, it is often possible to return separate datasets within a single query prior to stitching the data together as a further optimisation and simplification.

ORMs or Tooling

When ORMs are used discovering the N+1 problem is more obscured without logging the underlying queries that are performed. Once an issue is discovered it is usually a case of consulting documentation on what the fix is - often configuration related. Due to this it is worth enabling logging during development so queries can be analysed.

Tuesday, 2 February 2016

Sproc vs ORM vs Inline vs Polyglot

With relational databases the common data access patterns tend to fall into three core options.

  • Direct access via inline SQL
  • Stored procedures using the standard library
  • ORM frameworks or libraries

Individually these have both pros and cons, often leading to heated debate and discussion.

Inline

  • Leaky abstractions.
  • Dangerous in places via SQL injection.
  • Quick and dirty solution.
  • Non testable by default.
  • Useful for integration testing where dynamic input is required and safe.

Stored Procedures (standard library)

  • Can be clunky and low level to use in places.
  • Non testable by default.
  • Allows the use of DB specific features internally.
  • Easy to tune and optimize as long as interface is stable.
  • Developers can optimise the execution of queries.

ORMs

  • Testable by default.
  • Complex, large and difficult to use correctly.
  • Leaky abstractions.
  • Optimisation is harder, especially for DB engineers.
  • Mini or lightweight alternatives exist, with less of the downsides.

Polyglot Persistence

The actual decision of which data access method to use can be a non issue providing a good abstraction is used. Whether you use inline SQL, stored procedures or full blown ORMs is beside the point. Instead of abstracting the implementation detail, focus on the role the object or function has to play. A benefit of this approach is the ability to mix and match data access patterns. Polyglot persistence is gaining more traction where alternate data storage solutions are more appropriate.

N+1

One common flaw that all these data access patterns can have is the N+1 problem.