All tests should fail

So I've been staring at the screen trying to write something about SQL compatibility in migrations for a while, and it's just not flowing today. I had this topic in my backlog though, and it's something I feel strongly about...so here's a short and sweet one!

A lot of people are skeptical of TDD. I'm not one, but I get it. Writing tests first is hard and takes practice, and honestly we can't really prove yet that TDD as a process gets you better code. However, there is broad consensus that having automated tests is a good thing: it saves time manually testing and it documents features that might otherwise be forgotten. It seems to me, even, that it's become almost a bit embarrassing (at least on the open 'net) to not have a bit of a test suite.

So let's throw out TDD. How do we write valuable automated tests?

Tests should fail

If you write a test and it doesn't have an assertion, or perhaps a valid end state for a UI test, it's not a valuable test. If it asserts something that will always be true, it's not going to help you. What I'm getting at is that you need to see the test fail.

Write your test and then break your code. Seriously. If you're checking that a boolean flag changes the result of a calulation, go invert the boolean in code and see if it breaks. Change a div id in the UI and see what happens.

Tests should fail for the right reason 

If your test fails, but it fails from a null reference when you were expecting a wrong answer, that's not a good test. Well...ok - it's better than not failing at all! but still, it's a broken test. Check that you're getting the error or incorrect result you expect - at the very least, it'll help you fix the error faster.

Tests should fail, for the right reason, informatively

I never write a test without a failure message. I've also been to the moon.

Yep, we all neglect informative failure messages. Ye olde assertEquals(actual, expected, message); doesn't get a lot of use. Never fear - there are other less annoying ways to get informative tests! You can:

• Name your tests after what they expect! this is especially easy in more BDD-style frameworks like Jest, where it('throws an error if no username is provided') says a lot.
• Use fewer assertions in a given test. That makes it a bit easier to see what went wrong - not because the test won't tell you what broke, but because it's easier to comprehend what the test expects. It's a readability thing. Split up tests, if you have to.
• Look for ways to get the code to tell you what's wrong without you typing messages in every single test. For example, in Java you might implement toString on an object you make lots of assertions over, so that you can look at the test output for more context. Or, you might pull in a list assertion helper that prints something more useful than "lists are not identical". Write your own assertion method that generates a message for you, and use that in multiple places.

I'm not the first to write about this, and definitely not the best. Google around and you'll find many, many, many, many, examples of how to write tests. But, I like talking about tests, and maybe this will help jumpstart a few ideas of your own for writing tests that make your software more valuable.

Backwards Compatibility: Why should I care?

After I published my last post on HTTP API compatibility, a friend pointed out to me that the essay really didn't explain why backwards compatibility matters. Here's an attempt at making up for that.

Why do you need to worry about keeping your systems backwards compatible?

Let's say you've recently become a proud parent of a toddler. You go shopping every week, and before you had a toddler you could remember all the items you needed. But toddlers are very distracting, and between "Why is the sky blue?" and "Can I have a snack? pleeeease?" it's just impossible to remember little things like groceries. After the third time forgetting cheese sticks you build yourself a shopping list app. A friend sees you using it, and thinks it's really cool and wants to use it too! So you deploy it to the Cloud, call it Shopst, and start getting customers.

Shopst has a very simple architecture: just a single web application and a database. Since the application is a monolith, you update it by shutting down the app, uploading the new code, and starting it up again. And most of your users are in a single timezone, so they don't care if you take the app down for a while at night because they're asleep anyway.

Your customers are a friendly lot. They love shopping but hate forgetting things, so your app is a hit.You get a very interesting request - it seems that a major creative social site, PartyPinner, wants to embed your app in their site, so that people can add party supplies they find exploring everyone else's feed to their shopping lists. You think that's great, so you implement an API for them! You give them an endpoint that returns a users shopping lists, and another endpoint that lets them add to a given list.

GET /lists
[
  {
     "name": "Party Food",
     "items": ["Chips", "Dip", "Salsa"]
  }
]

POST /lists/add
{
  "listName": "Party Food"
  "itemToAdd": "Soda"
}

After a week of slamming your head against Oauth2, you can support clients. PartyPinner is very excited and you start getting more users right away.

Life is still good! You have more reach, but everyone still pretty much goes to sleep when you do so you can stay up a few minutes later than they do on Friday, update the application, and sleep in on Saturday.

Now, you get an email from a customer. "Dear You," they say, since, you haven't given them your real name, "I love using Shopst for shopping lists. I am going on a cruise in a month, and I want a packing list app to plan my trip. Is there any way you could add packing lists, too?"

"That's easy!", you think. "I'll just add a new list type." So, you update your website, and since you're a conscientious coder, you update your API too:

GET /lists
{
  "shoppingLists": [
    {
       "name": "Party Food",
       "items": ["Chips", "Dip", "Salsa"]
    }
  ],
  "packingLists": [
    {
      "name": "Cruisin'!",
      "items": ["Socks", "Swimsuit", "Snorkel"]
  ]
}

Another Friday deploy, and you look forward to sleeping in Saturday morning again.

At 7 AM your phone starts buzzing. It's PartyPinner emailing you frantically! They say, "All of our Partiers are trying to plan their Saturday barbecues at the last minute, and they get errors every time they try to find their 'Barbecue Supplies' lists! All they see is an error message that says 'Expected: List at index 0 but found: Object'! What gives?"

"Oh!", you say. "I changed that last night so that people could make packing lists too! You just need to change your code to expect an object from the GET, and populate your list...lists...from the 'shoppingList' property."

"Oh, ok, that seems like a great idea for someone...", they reply. "But our Partiers only care about Shopping! You broke all of them, and now we have to work on Saturday to update our code, and we don't care about trips at all."

So, you promise them that you'll never do that again. You've discovered A Thing here: Changes for new customers should not make existing customers break. You name this Thing API Compatibility.

Shopst continues to be a great success, and a year later people all over the world are using it! You start getting more complaints about your Friday deployments, because Friday night where you are is Saturday morning somewhere and people need to make their shopping lists! So, you decide to put up another server so you can upgrade one server at a time, without anyone having to stop making lists.

Now you have a high availability cluster. This is great, because you can tell the load balancer to direct all traffic to Node 2 while you deploy the code to Node 1, then reverse the traffic and deploy to Node 2. Sweet!

Sometime in the last year, you added a "Store" field to the shopping list object because you thought people might want to make different lists for different stores. Well, no one, and I mean absolutely no one, used it. They just put the store in the list name if they want it. "That was a good experiment", you think, "but I don't want to have a field no one uses. None of my API clients implemented it, so I'll just delete it. And since I have a High-Availability Cluster, I can deploy the change as soon as I finish it without having to stay up late on Friday!" 

Having just put your child down for their nap, you decide to make the code change quickly while they're sleeping. It's easy enough - just remove the column from a query and from some model objects and delete the column from the database once you deploy. You also add in a little script that will delete the column once the app deploys just so you don't forget about it. 

The child is blessedly still sleeping, so you take down Node 1 and deploy the app to it. You've just put traffic back to the node when, like they knew you were in the middle of something, your child awakens and is Ready. For. Snacks. Off you go to serve up some pretzels, and since this is really the first time you've deployed since you added the second node, you forget that you hadn't upgraded it - until your phone starts buzzing.

It's PartyPinner again - haven't heard from them since the Packing Lists update! "Something seems weird," they say. "About half the time Shopst seems fine...but the other half, we see an error that says 'Error: missing column "Store"'. Any idea what's up with that?"

You smack your forehead because you know exactly what's wrong. The second node with the older code you forgot to upgrade still expects the "Store" column! You quickly upgrade the second node and that fixes everything, and you plan not to do that again. And also to write a deployment script that updates both nodes so you don't forget that either. You name this concept Database Compatibility.

Systems often grow by adding pieces - components - to them. Those components need to be able to interact with each other, and to evolve as new capabilities get added. Understanding how to structure your interactions so that consumers don't break means that you can add features, fix bugs, and deploy all without breaking any clients or incurring any downtime. Which means, once you learn how, you can sleep in Saturdays. At least until your kid wakes up.

Backwards Compatibility: Techniques for HTTP APIs

Like most web developers, I do a lot of coordination between servers and clients. The most obvious interaction is between a customer's browser and my servers, but I also see a good amount of server-to-server communication.

Changes in distributed systems don't always get rolled out at the same time. Even inside a single "service", if you run multiple copies of that service for redundancy or high availability you have a distributed system and changes you make will not be applied simultaneously. If you offer an API, you don't have any control over when your clients upgrade their systems, and you probably don't want to break them.

I've found a few techniques for handling compatibility between systems during upgrades that I hope will give you some ideas for approaching the problem in your own work. Please don't consider them exhaustive - there are as many variations on this subject as there are systems. My work currently focuses on APIs, browsers, HTTP clients, and databases, so that's what I'm going to talk about.

Today's topic is HTTP APIs. Generally people want to consume them and get the any new features you make available: clients often have an interest in upgrading. So, you can often just worry about not breaking your clients without them knowing. With that in mind, lets explore how to make both non-breaking and breaking changes without causing your clients too much pain.

Non-breaking Changes 

Most of the time, you want to evolve your API without forcing your clients to implement your changes right away. These are non-breaking changes - your clients may upgrade to support them in the near future, but they won't stop working when you deploy the change.

Since we're talking about HTTP APIs, changes break down pretty easily to adding fields and adding or changing endpoints. This even holds true in non-RESTy environments, but frameworks like GraphQL probably give you an easier path to doing upgrades. I haven't used them, so I'm just commenting on REST-style environments. I'd love to hear how people handle this elsewhere!

Adding an endpoint

You've got one endpoint, say /users/{userId}. You want to add a /blogs path. That's completely safe - if no one's using it, you don't have to worry about breaking them! Clients can start using the new endpoint whenever it's convenient.

Adding a field 

Most JSON and XML parsers tolerate new fields well, or can be configured to do so. Unfortunately, you don't really get a say here - you have to define up front that clients need to tolerate unexpected fields and if you have one super-important client that can't, you'll just have to use other techniques. If they can,, though, you can add new fields as in the JSON example all you want and no one will break.

{
  "originalField": "value",
  "newField": "value",
  "addedAYearLater": "value"
}

However, you can't do this with lists. For example, you could define a Users endpoint as:

GET /users
[
  { "userId": 1, "username": "bob"},
  { "userId": 2, "username": "sally"}
]

...but then you'd only be able to add elements to the list or fields to the User objects themselves. It's often better for future work to return an object, and when you need to maybe add paging you don't need to break everyone to do it:

{
  "users": [
    { "userId": 1, "username": "bob"},
    { "userId": 2, "username": "sally"}
  ] ,
  "offset": 10
}

Breaking Changes

Sometimes you just have to change a system. Maybe you need a new naming scheme to better support new clients, or you want to change technologies. Either way, your clients will have to change. You don't have to force them to change in lockstep with you though. Here's a few ways to do it.

Changing a response type

Sometimes we get the structure of a response wrong. Returning a list and later needing an object as in "Adding a field" is a good example.

In my opinion, you should never change the structure of a given endpoint once published, like you you should never rebase a commit published to another Git repository. It really messes with your consumers and usually breaks them until they change their code to match your new output. Here's a few options you can do instead:

1. Implement a new endpoint that returns the desired structure, and deprecate the original until the clients have had a chance to update
2. (Variation on (1)) Version your endpoints, in the path (/v1/users, /v2/users) or perhaps using a header. I would recommend against using query parameters, because in the abstract you'd be mixing filters with response structure and that can be confusing. It may be an option if your clients are limited to paths and query parameters, though.
3. If the data is identical but the client needs a different structure, you might use the Accept header for a content-negiotiating strategy. This is perfect for letting clients choose between formats like XML and JSON, but it's also great for clients that need some metadata on the response that, for others, would be just noise and bandwidth.

Renaming a field

Naming is hard, 'nuff said. You should approach this differently depending on if you control the clients or not.

If you control the clients, you can do a three-step upgrade.

1. Add an identical field with your new, improved name. Do not remove the original yet.
2. Upgrade all your clients to use the new name.
3. Remove the original field once there's no one using it.

If you don't control the clients, I'd look really hard at why you need to change the name. You can deprecate the original and remove it after a reasonable amount of time, or...not, if people keep using it. It might be a better idea to leave the name alone and improve your documentation to compensate, and avoid the coordination.

You can also let a bunch of name changes stack up for a while, and deprecate and change them in a single batch. That would give your clients higher stability - for example, you might make changes on a 6-month schedule, adding the new names and deprecating the old in the first release and removing the old names in the next.

Wrapping up

Different frameworks and API styles have different ways of approaching compatibility - explore your tools and see which ways they can help you support changes without breaking your clients. For example, in the Java world Jersey abstracts out media types, which makes it really easy to support XML, JSON, or pretty much any structure you want. Maybe your framework lets you version endpoints without much effort, so you favor that strategy over relying on clients ignoring new fields.

Whatever you choose, think about what your clients need and evolve you API in a way that lets them evolve along with you.