Since 2001, the No Fluff Just Stuff Software Symposium Tour has delivered over 450 events with over 70,000 attendees. NFJS, as it's called, is known for its attendee centric approach, excellent speakers, and technically rich presentations which cover the latest trends within the Java/JVM ecosystem and Agility space.

I attended the NFJS Conference when the tour stopped in Reston, VA in November. What I really like about this conference is how applicable the information being disseminated is. I love taking away potential game changers that I can introduce to client or even personal projects to help enhance their success.

As developers, we are always pursuing more efficient engineering practices for building and releasing software. One of the standard ways we do this is by implementing continuous delivery. Continuous delivery is concerned with fitting together different components of software development in order to reduce errors, lower stress, and provide deployment flexibility.

This is a tried-and-true engineering practice, but a lot of projects inadvertently throw this practice away when they introduce feature branching. In order to manage in flight speculative features which may or may not make it in the next release, we create a separate branch outside the mainline codebase. This allows us to keep all of the changes for these particular features siloed, and then decide later when (or if) that feature should be included in the release. But, when you check everything into a completely different codebase than the mainline, you're technically no longer doing continuous integration and can run into several problems following this approach.

There is a whole engineering practice to avoid using feature branches around this idea of trunk-based development. Everything stays on trunk all the time, no feature branching at all. Instead, we use things like feature toggles. In its simplest form, a feature toggle is just a condition to add to your code, that says whether a feature should be on or not.

Benefits of trunk-based development using feature toggles over feature branching

Merging

  • Feature branches introduce the potential of scary merges. Although each developer can develop his feature isolated from changes going on elsewhere, eventually his work does have to be integrated. If there are multiple feature branches (or even just one) and the same class gets significantly modified, we introduce potential merge conflicts. And the longer the branch lives, the more horrible the merge is going to be. It is not unheard of for feature branches to last up to two years, in which case you are bound to run into a merge conflict in trunk that may be difficult if not impossible to reconcile.
  • With trunk-based development using feature toggles, you can turn the feature toggle off so no one else ever sees it or is ever impacted by it. And if the same class is touched, we're still doing continuous integration and still integrating on trunk, so we avoid a large nightmare merge conflict

Combining Features

  • In a feature branch, there's no easy way to see combined features. If I'm working on feature branch A and you're working on feature branch B, and boss wants to see these features working together, that's multiple merges and fixes we have to do every time that request is made. And that's a problem.
  • Combining features using feature toggles is trivial. Just turn them all on to see them working together because you pre merged the code. You don't have to go through a merge exercise because the code is merged all the time because it lives on trunk.

Refactoring

  • In working with feature branches, there exists the temptation to leave "broken windows" unrepaired. What does that mean? Well, on a proper agile project, you're supposed to fix what we call "broken windows" (bad designs, wrong decisions, and poor code). But let's say a class is misspelled, and in one feature branch you rename it. That touches forty or fifty files. And if that lives on a feature branch for a few days or weeks or months, the punishment when you commit that feature branch may be so grave that you may decide not to do it. We don't want road blocks or bear traps preventing us from doing what we're supposed to be encouraged to do.

  • With trunk-based development refactoring comes easy. If you're using feature toggles instead of feature branches, you can fix a broken window because you're already in trunk.

Implementation

Feature toggles are a powerful technique but can be very easy to implement. Below is a simple feature toggling implementation:


public class CapTech {

 public void process() {

 boolean useNewFeature = true;

 // useNewFeature = false; // UNCOMMENT IF WORKING WITH OLD FEATURE 

 if (useNewFeature) {

 return newFeatureMethod();

 } else {

 return oldFeatureMethod();

 }

 }

 public void oldFeatureMethod() {

 // current implementation lives here

 }

 public void newFeatureMethod() {

 // improved implementation lives here

 }

}

The above code snippet is a Java example but feature toggles work on all platforms and technology stacks. As long as the platform has an if-statement and can read external configurations, you can implement feature toggles. The one thing that doesn't really have those two characteristics is databases. Toggling around databases is unique and a little trickier and will be discussed later on in this blog.

Another thing to note about the code snippet is the toggle management approach. You can hard-code toggles directly into your application like in this example, however it is not recommended unless the scope of your toggle is very small. In most cases, you will want to use a parameterized version or config file which can be as simple as a properties file with name value pairs. You can also store name value pairs in an application database, or you can use a distributed asset management tool like Zookeeper. But you must ensure the application has access to the toggles when it needs it. It may be wise to split toggle information across technology stacks. If you have a JavaScript application with a Java backend, you may want to choose something they both can see. A toggle router is often used to dynamically control which code path is live.

Different types of feature toggles

Release toggles are used to perform canary releases where you gradually roll out a new feature. This approach is good if you have lots of users and infrastructure and you want to roll out new capability but not sure what the impact will be if you roll it out to everybody at one time, so you instead release it to a small subset of users to reduce risk. Facebook did this with chat. They turned it on in New Zealand first because it's the first country to get sunlight and it has a relatively small number of users. Next was Australia, and so on and so forth one time zone at a time. And while doing so, you can monitor the reaction as the feature is gradually rolled out and respond accordingly. Google does something similar when testing out a new search algorithm. They release to one percent of users the new search algorithm, and if users click higher on the page compared to the existing search algorithm, then they know that's a better search algorithm.

Experimental toggles are extremely dynamic and look at the context of each request and determine whether to turn on a particular feature or not. For example, if a user enters the application, the toggle router will be utilized to determine the behavior for this user.

Operational toggles are either very short lived or permanent kill switches for operational type things. For instance, Facebook uses them to move from one data center to another.

Permission toggles or product experience toggles allow you to give the base version of your product, but if the user pays or achieves something, a new slick feature (that has already been deployed) will be made available. With this approach, you can deploy the whole product which is a lot easier than deploying piecemeal.

Toggling Databases

The whole idea of feature toggles is toggling around a behavior and creating two different versions of the same behavior at the same time. We don't have the same mechanisms in databases as we do in programming languages. But what you do have is what is called the expand and contract pattern for database refactoring and this is also useful for feature toggles.

Let's say you have three applications all sharing the same database but one application needs a column moved from table X to table Y. The approach is to create the column on table Y and then create a trigger that synchronizes across both tables. This way, when somebody adds something to the original table it gets added to the other table as well, and vice versa. In effect, you are supporting both APIs.

This is how you tend to toggle things around databases, you don't have direct conditionals, so you build mechanisms like this to support multiple views. Once you are certain that all three applications need that column in the new table, you can then drop it from the original table and drop the trigger. This approach can also be used for rollbacks as well.

Other Benefits

There are also other benefits to explore. Companies sometimes struggle with setting up a real life production like environment for QA. As it turns out, there already is a real life production like environment, it's called production. You can roll out new features in production but only turn on those feature toggles for the users that include the QA group and there's no way any other user will see it. This may seem risky and may take a lot of convincing of clients and business partners that this approach is in fact safe, but consider this: the if-statement ALWAYS works! :)

Conclusion

Feature toggles are cheap to create but they do require some management as it's important to make sure they aren't overused or overly complex. They shouldn't be too plentiful, and shouldn't depend on other toggles because it'll make it hard to understand with inflight features what exists in your codebase. They also should be removed as soon as the feature decision is resolved. Feature toggles are essentially purposeful technical debt, but added to support continuous delivery. And with the goal of continuous delivery to mitigate risks, trunk-based development using feature toggles is about the least risk you can get. If you roll it out and it starts misbehaving, you turn the feature off and the feature disappears.

Resources

https://nofluffjuststuff.com/home/main

https://nofluffjuststuff.com/conference/reston/2016/11/session?id=37808

https://martinfowler.com/bliki/FeatureBranch.html

https://martinfowler.com/articles/feature-toggles.html