Tuesday, 29 July 2014

I Need to Stop Misusing Namespaces

At the recent NSBCon one interesting question that came about was how to structure a project. The panel consisting of various speakers had no answer, after all this is dependant upon the project in question. Therefore there is no right or wrong answer.

However one point they were in unison about was splitting the domain and technical implementation of a project apart by the correct use of in namespaces.

This is not the first time I've come across this, but I find myself breaking this principle on a regular basis. For example a typical project I work on looks like the following.

/Controllers
   FooController.cs
   BarController.cs
   BazController.cs
/Models
   FooViewModel.cs
   BarViewModel.cs
   BazViewModel.cs
/Helpers
   FooHelper.cs
   BarHelper.cs
   BazHelper.cs

Problems

  • The namespace reflects a technical implementation detail, and not the problem domain.
  • Using Foo as an example, here the namespace is duplicated within the name of the types, which in turn defeats the point of namespaces.
  • Another issue is that the types can be much longer than they need to be, which is often a criticism of enterprise software development, when the names of objects roll off the screen because they contain so many patterns or conventions.

Solution

Use namespaces for related domain responsibilities. In turn, group together the objects and types that are used together.

An example of a better solution therefore would be:

/Foo
    Controller.cs
    Helper.cs
    ViewModel.cs
/Bar
    Controller.cs
    Helper.cs
    ViewModel.cs
/Baz
    Controller.cs
    Helper.cs
    ViewModel.cs

Benefits

  • Things that change at the same rate, would live logically next to things that also need changes. In other words if I update the FooViewModel, chances are I'll need to update views or controllers.
  • Less typing if you don't suffer a namespace clash!
  • You can still prefix the namespace where required, e.g. Foo.Controller if you have a clash or prefer the readability.
  • Shorter type names!

While this is the ideal way of structuring our applications it's not always possible. Some coding conventions actually encourage the first example, and depending on the configurability of certain frameworks this may prove difficult. That aside, I'll be making a strong push towards structuring my projects correctly going forwards.

Monday, 21 July 2014

SOA Done Badly vs SOA Done Right

I was under the assumption I had been doing SOA for over 3 years. Previously I have had services which did stuff, these talked to other services which did other stuff and so on. We would group services around functionality.

We would break these services down if they got too big to handle. Behind the scenes the services would talk to a single database. When it came to creating an application or system, these front end applications would invoke various services, which in turn invoked other services. This layered style of architecture worked for a while, so everything appeared to be fine.

The overall architecture looked like this:

Bad example of SOA

Over time I began to question the benefit of this style.

  • I spent more time coding than solving business problems
  • I spent more time debugging than solving problems
  • I spent more time fixing broken deploys than solving problems
  • I spent more time writing infrastructure (to glue services together) than solving problems

It turns out this is actually quite a common style to SOA. But there are some serious flaws with this layered approach.

Problems

  • If a service fails, the whole system is pretty much broken.
  • If the database breaks, the whole system is pretty much broken.
  • If a service is slow or breaks SLA, the whole system is pretty much broken.
  • In order to decouple between services, you need complex, costly abstractions.

Solution

A solution to solve these problems is to partition the domain boundaries, vertically. The overall architecture would look like this:

Good example of SOA

Details

  • Each domain could consist of one or more "services".
  • Each with their own choice of data store, e.g. SQL, NOSQL, file system etc...
  • No domain can directly communicate with another domain.
  • If they do need to communicate, then the pub/sub model would be used or an ansyc command could be issued. Think .NET events or the event design pattern but across processes, not objects.

Benefits

  • Each service could fail and the impact would be minimal, simply rollback the deploy or fix the problem. Once the service comes back to life any commands or events that have not yet been processed would be handled. Each service would have their own message queues to persist commands/events.
  • Services could be re-wrote and distributed easily, proving they are highly decoupled.
  • No complex infrastructure, e.g. no need to map between objects because the service owns the whole stack. The same model could be used across partitions for example.
  • Works really well with agile development methodologies, e.g. vertical slicing of stories.

There is more to this introduction. Microservices are a hot new topic. Being defined as "SOA done correctly". Using the second example, within each domain there could be multiple services (or autonomous components) that coexist. The likes of UI composition to enable applications to be created from services is another great benefit. This enables so called "mashups".

I'll expand on these topics over time, but I am certain this method of vertically slicing services based upon business capabilities will be my default approach going forward.

Sunday, 1 June 2014

The Importance of Tools

One of the most influential books I've read on software development has been The Pragmatic Programmer.

One of the key points raised within the book is that of automation and tooling. For example, automating the build process is a very worthwhile undertaking. You should be able to check out some code and execute a script that will set up your machine, compile, test and deploy the code base in question.

The key benefit of automating even trivial tasks such as automatically pulling down the latest code daily is that unlike developers, automating tooling will never perform the task wrong. Nor will they forget to do it. Ultimately this prevents the dreaded "works on my machine" issue.

I've become such a fan of this approach to automating away any manual steps that some of the most used code I've written has been small scripts that execute hundreds of times a day. From a development point of view, the likes of good practices, SOLID, OO etc.. are usually void, such scripts simply get the job done, allowing myself to focus on the more important tasks such as delivering business value else where.

There is not a lot else to say on the subject of tooling. The best tools should be composable, proven solutions where possible. In other words, rather than something that must be configured via a GUI, opt for something that can be automated. Also ensure that you are not re-inventing the wheel unnecessarily. Save your time and energy on creating the custom tooling you can't get "off the shelf".

Thursday, 1 May 2014

Learning Tests

At the last Agile Staffordshire I attended the task was to complete the string calculator with constraints. The group worked in pairs and everything was running smoothly. Until I heard a few guys behind struggling with something.

I'd worked with one of the developers previously, so they called me over to take a look. What he found was pretty shocking - they had found a bug in the .NET framework. The string class of all things. Bugs exist in all code. Bugs in the substring method though are probably rarer given how exhaustively used this particular bit of code is.

The problem was how they expected the method to behave. When creating a substring they were getting confused with how the offests worked.

This is an easy mistake. Different languages or frameworks can have different methods to do similar tasks. I take no shame in not knowing of the top of my head whether the offest of the substring method is an offset of the index, or an offset from the start of the string.

I managed to spot the issue very quickly but never let on. Instead I decided to share a technique which I use regularly to great effect.

Rather than painfully using the debugger and stepping through the code line by line I suggested he write a simple test around the single line of code they were convinced was misbehaving.

After a couple of more tests it was clear how the substring method worked in .NET. Once this was cleared up, we deleted the tests we just wrote and modified the production code to use the correct offset. This whole process took less than sixty seconds.

I explained afterwards that such a technique of writing learning tests or scaffholding tests is incredibly valuable. The feedback cycle here is very quick. Quicker than explaining to another developer what is wrong; Quicker than "Googling" the problem; Quicker than looking at the reference implementation and certainly quicker than using the debugger.

My rules are pretty explicit when dealing with learning tests. They should be short lived. Testing implementation details is often a bad idea, but that is the whole point of such style of testing. Therefore if you do decided to check these tests in tagging them so they are only run as part of CI builds is worthwhile. In other words, just like real world scaffholding, they are temporary. Don't feel bad about writing some tests, only to delete them minutes later.

Learning tests have another nice side effect. They give static languages which have a slower feedback cycle a form of REPL. It's a lot quicker to write a test method and execute than it would be to spin up a new project in languages such as C# or Java to just try something out.

Next time you're stuck, try writing a test.

Tuesday, 1 April 2014

Dont Tie Yourself to a Framework

  • Programming is great
  • Software development is the crap bit. You'll spend more time configuring, integrating and faffing rather than writing logic most of the time.
  • Test Driven Development makes development easier as it forces you to decouple your code.
    • Your core logic should be pure, dependency free C#, Java, Python etc.
    • Your frameworks and libraries should be on the edge of the system.
  • Most people do this for some of their code, e.g. your data access.
  • What about the other parts of a system?
    • Web frontend
    • REST api's
    • Console applications
    • Desktop clients
  • Why should we couple our applications with these layers?

Hexagonal Architecture

  • Hexagonal Architecture is a solution to limit coupling
    • Easily switch out your delivery mechanism, e.g. test runner adapter for testing, HTML adapter for production.
    • Great example from Kevin Rutherford.
    • Excellent video by Uncle Bob though terminology differs.
    • Implementation details should be hidden behind adapters.
      • Tested manually in the majority of cases
      • Few integration tests for comfort
      • Third party code after all.
    • Inner hexagon should only communicate via ports (interfaces) - keeps domain pure.

Why?

  • Last few major projects involved with were due to the delivery mechanism becoming out of date.
    • Flash to Web
    • Web to Mobile
  • Easier to test
  • Easier to change

Why not?

  • CRUD apps - sometimes it's just CRUD.
  • Lightweight projects might not need hexagonal architecture
  • SOA or Microservices could mean hexagonal architecture actually introduces overhead or complexity - judge on context.

Flexible Selenium Tests via Page Objects

A fast, automated suite of unit and integration tests are not enough. At some point you'll need to test your presentation logic. Ideally your domain/business/game logic is stubbed so all you'll need to do at this point is check that the presentation is complete. For example, does view X load view Y? Does an error message appear when an error is raised?

With web sites and web applications the standard tool to use is the excellent Selenium. The problem with UI tests in Selenium is they are often slower to write. Not only this the maintenance cost of such tests can often be much more expensive that other styles of tests. If the cost of such tests is high, the likely hood of developers writing UI tests is low. In my experience there are three types of UI tests in use.

  • Low Level

    Here UI tests are wrote directly against Selenium. This low level approach means tests are scattered with assertions and UI details. For example element locators such as divs and ids will be used with methods on the Selenium driver in question. Despite this low level approach such tests are often quick and dirty to create. The downside to this style of test is that as the volume of tests increase, the cost of maintenance can become very costly. A simple UI change can cause a ripple that will cascade through many test cases.

    firefoxDriver.Type("id=username", "Shaun");
    firefoxDriver.Type("id=password", "Pa55word");
    firefoxDriver.Click("logIn");
    firefoxDriver.AssertText('You have logged in.');
    
  • Browser Abstraction

    The next level up from direct use of Selenium's driver is to create a facade around the browser or UI itself. For example rather than duplicating the steps to log in within each test you could create a method PerfromLogin(...) which each test could make use of. Another example would be abstracting messier details of UI automation such as clicking a button and waiting for an event. This style of test has the benefits of low level tests but gives some flexibility when it comes to maintenance. The downside with this facade approach is that UI changes can still cause havoc, as each test in question will be tied to the UI elements directly.

    // Method hides locator details and enables re-use.
    LogIn("Shaun", "Pa55word");
    firefoxDriver.AssertText("You have logged in.");
    
  • Page Objects

    Taking the browser abstraction to the next level, page objects are an abstraction over the UI itself. These high level tests are wrote in terms of the domain, rather than implementation details. There is of course one place where each page object is bound to a UI element, but as each test uses an object, rather than element locators you only have to change one place when your UI changes. Unlike the previous two styles of tests, page objects incur the most amount of code, though for more than a handful of tests this style of UI acceptance test will pay for itself in no time.

    var resultPage = new LogInPage()
                            .Username("Shaun")
                            .Password("Pa55word")
                            .LogIn();
    
    Assert.That(resultPage.SuccessMessage, Is.EqualTo("You have logged in"));
    

    With the above example the LogInPage object will be bound to UI locators. This will vary based on programming language, but using C# as an example each property would have a specific attribute to link up each element. The domain specific methods such as Username will fill in the correct UI element with the provided value. By writing the objects in a fluent interface style, you can achieve QA friendly tests which are easy to debug when they go wrong.

A more fleshed out example of the Page Object pattern can be found on Github.

Choose a style based on context. Given more than a handful of tests then page objects are worth the extra cost, the ability to evolve your UI while maintaining end to end tests is worth some additional complexity at first.

Design is Important

When I was a student I used to cheat. Not in exams or practical assignments, but I used to cheat when it came to my process to develop code. Early on I noticed a common pattern. After receiving an assignment I would perform some analysis, figure out a basic design and document my steps. The problem came when to code up the solution. I may have overlooked something, or made a mistake. Sometimes I would just come up with a better solution. This meant any time I spent documenting was lost. It turns out this wasn't cheating, after all there was nothing within the assignments enforcing a waterfall approach.

I wasn't alone with this experience. Most of my peers had the same issue, and the report aspects of an assignment were often disliked for this very reason. My solution was simple. Code up something, get it working then document the design aspect. Rinse and repeat. Back in the early 2004 I wasn't aware of agile methodologies, but this solution worked a treat. In turn my classmates started to adopt this similar approach, either from my encouragement or their own discovery.

Moving from university into a practical environment was a joy. It almost appeared as if little to no documentation was produced. The documentation that was produced, was often created by other teams. Developers simply wrote code. At the time I thought this was great, but after some reflection the errors of my ways have been highlighted.

Problems

In my experience a variety anti patterns are to blame.

  • No or limited design

    The worst thing that can be done when it comes to design or planning is the absence of any design or plan whatsoever.

  • Coding your way out of problems

    Given some limited or poor design, I've often experienced scenarios where 80% of the tasks will be complete, then you hit a roadblock. In order to progress the team will hack their way around it, introduce technical debt or put in some not so temporary fixes.

  • "Weeks of coding can save hours of planning"

    A colleague I used to work with used this once and I fell in love with the quote. Take an average web application, if the life cycle of this would be a meager two years, spending a few hours putting a design together is nothing. You could argue that spending a few days would be equally fitting, better yet a couple of weeks well thought out design is only a small percentage of the overall cost of delivery. When it's too late you can code your way around the problem. Though this debt will soon add up, meaning features are even slower to add going forwards.

  • Playing the "Agile" card

    A misconception of the agile manifesto is to favour "working software over comprehensive documentation". Most developers read this as never document anything. This is far from the truth. Documentation, design and planning should be built into the product in iteration. Just In Time (JIT), rather than all up front or never at all.

  • Greenfield projects

    Having been involved with a couple of "rewrites" I've seen this happen first hand. No design, limited design or bad planning in the first few iterations of a project can kill it. Only by iteration three, four or five will you notice something isn't right. At this point you've lost. Suggesting to restart, reboot or refactor is a hard sell, especially to management teams. Architectural changes are very difficult at this point, as you'll most likely have users, automated tests and other teams relying on what you have produced.

  • Refactoring can save the day!

    Give me a bad class or method and I can make it beautiful. Give me a bad application and we have a problem. Refactoring is a class or method based activity. I don't buy architectural refactoring - and I'm not alone. Emergent design is a very powerful tool, but without some upfront planning you'll be stuck in limbo.

Solutions

There are a few ways to overcome the previous problems.

  • Whiteboards

    As much as I love technology you cannot beat a whiteboard (or piece of paper) and a couple of engineers. Visual collaboration in this manner is very easy, plus physically having the presence of another individual helps. You can also snap a picture of these diagrams to reproduce them in a more friendly, shareable, digital form afterwards.

  • CRC's

    Class Responsibility and Collaboration cards are another low tech solution, but one I find incredibly valuable, yet for some reason don't appear to do enough of. Best performed in groups, though I've had some success on solo efforts.

  • JIT documentation

    Not pages of wiki articles or documents, just lean, self contained documents that serve a purpose. Develop these in iteration and you'll avoid a "documentation sprint" from hell.

  • Code itself

    Prototypes are worth their weight in gold. Spike solutions when used across a team are also incredibly effective. Rather than a single prototype being produced, each team has a crack at the problem in isolation. After regrouping you present back your solution and findings. The team then combine to form a "best of breed" approach.

    Iteration zero is often used for getting the build up and running. If you take this one step further, the ideal scenario is to produce a walking skeleton. This should consist of empty or basic class/method/function definitons that have not yet been implemented. With a basic API in place, fleshing out the details is rather enjoyable. You focus on the problem, not the design or architecture.

None of these are ground breaking ideas, but combined these approaches have served me well both personally and professionally.

Saturday, 1 February 2014

TDD is a Tool

I remember being introduced to Test Driven Development (TDD) very well. This is because it had such an overwhelming change on how I write code day to day. It was incredibly alien, difficult, yet rewarding. On this journey for the last five years I've changed my style, learned how not to do it and finally found my "sweet spot" when it comes to pragmatic TDD.

Deliver Value

Writing code is fun. Developing an application or system is fun. Using new technology is fun. Despite this the end goal should always be to deliver value. Delivering business value over religiously following a practice was a turning point in my journey. After all the user doesn't care about what is behind the scenes, as long as they can use your software, they're happy.

When to Write Tests?

One of the guidelines when starting TDD is

"Never write a line of code without a failing test" - Kent Beck

This rule is wrong on many levels. Firstly it cripples most developers when starting TDD. Secondly the guideline is broken all the time by seasoned evangelists. Writing some framework code? Writing data access code? Writing markup? Any of these scenarios would be wasted by writing a failing tests first. This rule should be reworded.

"Writing logic? Never write a line of code without a failing test" - me

It's OK to not use TDD

After adoption TDD practitioners tend to face two challenges. Other developers looking down on non TDD practices and feeling as if they are "cheating" when not using TDD. The later was an issue I struggled with. Newbies tend to find the same problem, and this goes back to the mantra above. One of the key lessons I've discovered over the past few years is that using TDD where appropriate is fine. Not all code needs TDD. Even Kent Beck discusses this when he refers to "Obvious Implementation".

Spike Solutions

Another game changer in my journey was the concept of "Spike and Stabilize". Using this technique you can deliver business value quickly. Gather feedback as soon as possible and either fail fast or wrap the code in tests and clean it up.

CRUD

Most of the code I (and others) write is very similar. I'd bet this is the same for different fields of software development. That being said, for each CRUD app we create there is a tiny aspect of this that is unique. Using TDD to write yet another CRUD app is tedious. I'd imagine this is why many ditch the practice of TDD after some time. However the benefit comes from using TDD for that 20% of domain logic. Here a combination of obvious implementation and spike and stabilize can assist in the creation of the other 80%.

It's about Design too

TDD by Example gives the impression that the practice is primarily a testing discipline. This is not true. TDD does limit the bugs I introduce and enforces basic correctness, however bugs will still slip through. After all the quality of the code is only as good as the quality of the tests. Growing Object Oriented Software: Guided by Tests and others introduce the concept that TDD is also a design process. Listening to the tests is a core concept. In other words, if something is hard to test, chances are the code in question can be improved.

Follow the Risks

The final lesson I've come to realise is that even if you happen to work with those who don't practice TDD, you can reap the benefits. Simply test where the risk lives. Ignore the framework, standard library and simply test what has risk. This might be a small, core part of your application. Aiming for 100% code coverage is not a goal, nor one worth aiming for.

It's a Tool

At the end of the day, TDD is a tool, not a goal. In this day and age many believe that TDD should be mandatory. While I agree, the use should be restricted to where and when it makes sense. As for when and where, this is up for the developer to decide. Using some of the findings above allow me to be pragmatic, yet still have confidence in the quality of my code.

The Correct Way to use var in C#

The .NET community is not widely controversial, though there is a strong topic that appears to come up time and time again when I pair with other developers - how to use var in C#.

The var keyword was introduced in .NET 3.5. Unlike other languages this is still a strongly typed declaration. For example if we declare a string using var then we cannot re-assign this variable to another type. This would be a compile time error.

There are two parties who have strong feelings about the use of var, both of which are wrong.

Never use var

Some developers suggest the use of var be denied. This leads to code such as the following. Overly verbose, and in some cases obscuring the intent of the code. This can commonly be seen when dealing with collections or generics.

Person person = new Person();
Dictionary<int, string> someCollection = new Dictionary<int, string>();
IList<Person> result = FilterInactive();

Always use var

Other developers claim you should "var all the things". This leads to code which has the opposite problem from above. The intent of the code can be obscured due to not knowing what type you are dealing with. This is especially important during code reviews or times when you are not relying on the IDE's intellisense to remind you what you are dealing with. After all code is read many more times than it is written.

var person = new Person();
var someCollection = new Dictionary<int, string>();
var result = FilterInactive();

Best of both worlds

The solution to this issue is simple. Where the type cannot be inferred just by looking at the source code (aka the type is on the right), use a strongly typed declaration. Where the type can be inferred, use implicit typing. Using the same examples as above, this would look like the following.

var person = new Person();
var someCollection = new Dictionary<int, string>();
IList<Person> result = FilterInactive();

As with most things when it comes to software development, there is never a black and white answer. Always gauge decisions and patterns based on context. Just because automated tooling such as the excellent Resharper suggests you use implicit typing doesn't always make it correct.

Bonus

Talking of Resharper, a quick Alt+Enter on a type/implicit declaration will allow you to switch between modes, meaning you can be lazy and have the IDE pull in the right type when required.

Top Down vs Bottom Up

Top down development has you starting at the highest point in the application that you can. From here you code down until there is nothing else left to develop. Once you reach this point you should be code complete. Along the way you may need to stub out areas that have not yet been created, or designed.

Bottom up development has you starting at the lowest point in the application. The idea being that this part of the application has the most complexity or will be the most important. You will build the system up from a series of smaller components.

Top down development and bottom up development was introduced to myself in my early days of university. At the time the distinction didn't really mean much - I was very much a developer who would work from the bottom up.

Over time I have completely switched my stance on this. I believe agile practices and TDD are the reason for this change. I feel so strongly about this that I would go as far as to claim that within an agile team - bottom up development is an anti pattern.

Consider the following tasks to be completed on a team of four developers.

  • Create controller - main entry point, request mapping.
  • Create service - service layer, simple business logic.
  • Database query - thin wrapper around complex DB query.

With a bottom up approach a pair of developers could work on the complex database query. After some time they would have this working. The other two developers could start with the controller or service.

The problem with this approach comes from the painful integration process. The developers working on the service might be coding against the interface the team discussed during a planning session, while the developers on the query may have had to change their approach.

// Interface the service is coded against.
query.Execute(id);

// Interface the query is implemented with.
query.Execute(id, typeId);

This example is trivial, but imagine a story with thirty tasks, more developers and more complexity and this bottom up approach is difficult. Over the past few years my top down approach has evolved.

My first step would be to stub out the workflow with the above implementation. There is no real logic here - only the objects collaboration is implemented. At this stage there are no tests, TDD would not be used. After all there is no logic here. The code is so simple it can be reasoned about with peer review, planning sessions and so on.

// Controller method.
public Result Index(IncomingRequest incomingRequest)
{
    var res = service.Invoke(incomingRequest.X, incomingRequest.Y);
    return new Result(res);
}

// Service method.
public QueryResult Invoke(int x, int y)
{
    return query.Execute(x, y);
} 

// Query method.
public QueryResult Execute(int id, int typeId)
{
    // Database logic would go here, for now we stub it.
    return new QueryResult();
}

At this stage all of the tasks are open for any developer to pick up. If a breaking change was required, there would be no way for one pair to commit these changes without the other pair knowing. Another benefit of this approach is that an end to end acceptance test could be wrapped around the functionality from the get go.

As part of these tasks each developer would use TDD. Remember no tests exist at this point. Building up the tests in stages would ensure the logic of how the objects collaborate is preserved, and ensures that the actual domain logic that is implemented is correct. Does this mean we aren't doing TDD? No, of course not. The tests will drive the implementation. If we need to introduce new objects that is fine - these simply become implementation details that the other devs need not worry about as long as the workflow is not broken.

This approach to top down development isn't new, though many don't appreciate its benefits. I plan on expanding on this style of pragmatic TDD in the coming months.