Incrementally Moving to Trunk Based Development and Continuous Integration - Part One

Backstory

I was eating lunch a few weeks ago and the leader of development at the company I was on contract for, sat down and asked me how things were going. I had just spent a lot of time merging from our Dev branch to our Main branch. We started talking and I was attempting to extoll the virtues of trunk based development and feature flags.

He challenged my theoretical ideals by pointing out that it’s hard to find a path to move from TFVC (TFS Version Control) branches to a trunk based development flow. He’s tried trainings and other things with his distributed team in the USA and the world (10-20 developers). Unit testing hasn’t stuck with this team yet. He also has insightful questions about how to have a feature flag that spans the full stack (DB, APIs, UI, services, etc). This conversation lead me to write this article.

This company has a common approach of using TFVC with a Dev branch, Main branch and RC branch. They have a nicely laid out development process that requires reviews, demos and testing before merging it to Main. They have dependencies on MS Dynamics CRM and Sharepoint that make isolating code this way necessary (testing needs to be done against the development environment).

There are already 1000s of articles on Continuous Integration DevOps, but I needed to take the time to write out what I’ve learned from reading, watching and observing. Hopefully, this brings value to the community and you the reader.

What are we trying to accomplish?

Reduce the friction of getting value (completed features/fixes) to our customers Back to the Future tracks

• Deliver value faster with increased quality
• Win new customers and improve current relationships => help our business succeed and excel in the marketplace
• Increase Net Promoter Score (NPS) as discussed in the Accelerate book
• attract and keep talent in house
• improve everyone’s working lives, reduce stress
  • “the CI/CD approach has been a definite stress reducer for me.” ~a friend who has lived through a transformation and reviewed this article for me.

This transformation will take time (years), but we are looking for an incremental approach and continuous improvement.

What is Trunk Based Development (TBD)?

Here’s how I see it:

• Developing off of the master branch with feature flags to hide in progress work.
• Avoid merges
• Continuous Integration and Deployment
• Promote an artifact through the pipeline, instead of by merging code

How does DevOps fit in this?

I’ll use Donovan Brown’s definition “DevOps is the union of people, process, and products to enable continuous delivery of value to our end users.” Trunk Based Development is essential for continuous delivery and DevOps.

I use these thoughts as guiding principles to improvement. The Accelerate Book highlights what they’ve learned from years of State of DevOps Reports research and TBD along with the DevOps concepts are indicators of higher performing teams.

Focusing in on Trunk Based Development (TBD)

I see TBD as a core aspect of delivering software with less friction. Thoughtworks has an excellent article about TBD . Take some time and read this article. I won’t re-hash TBD more here.

update on September 24th, 2021: The DORA State of DevOps 2021 Report confirms this is a good approach. “Our research has consistently shown that high performing organizations are more likely to have implemented trunk-based development, in which developers work in small batches and merge their work into a shared trunk frequently”

What are the capabilities/pre-requirements needed to do TBD?

I don't think you have to move from the center out in a linear fashion, but communication aka a part of DevOps is definitely at the center of moving towards TBD (and DevOps).

My analogy

• By Automation, I mean unit testing, integration testing, UI testing (smaller amounts at this level). Deployment automation

Is there an incremental path to TBD? / Where do I start?

On page 145 of the DevOps Handbook Mr. Gruver shares his experience moving to TBD.

“However, trunk-based development required them to build more effective automated testing. Gruver observed, ‘Without automated testing, CI is the fastest way to get a big pile of junk that never compiles or runs correctly.”

You should definitely not start by deleting all your branches and getting rid of your process. You need to do some preparation work first.

Thankfully, The DevOps Handbook has “part 2 - Where to start?” to give us guidance. Here are my thoughts, many from this book, other resources, stories and experiences over the years.

Start with building the team and an open/no blame culture

We are a team, let’s stop blaming each other (even if we don’t say this out loud) and work together. That means getting managers, developers, Ops, QA and more people together and talking. Buy some pizza, get people in a room together, have some fun and identify how to improve together.

The DevOps Handbook has a section and here’s a good post on blameless postmortems

Learn together

I highly recommend starting with The Phoenix Project as an introduction to the DevOps ideas. Then get The DevOps Handbook to use as a reference and read together. Accelerate is also a great reference based on research to convince people why changes are necessary. I’m thankful for Gene Kim and others for putting these books out in the last several years. Continuous Delivery by Jez Humble (at least the first 5 chapters) came out before the others and is a foundational book as well. There are many articles, talks and videos as well, that I’ve referenced at the end of this article.

Watching how Spotify organized their teams and delivers software was eye-opening for me when I watched it the first couple times years ago.

• Spotify Engineering Culture part 1
• Spotify Engineering Culture part 2

Maybe you should spin up a new prototype that is smaller to experiment and learn with less risk. A company that’s mentioned in the DevOps Handbook

Don’t get stuck in learning paralysis. You can do while you learn. Create a hypothesis, do small experiments, record, learn and adapt.

Identify

What are the company goals? Do we know why we are doing what we are doing?

As a developer, I can easily lose sight of the company goals while I’m focused on creating a feature, debugging problems that are reported, helping others, etc. Does you team know why you are building each piece of software and how it helps your company succeed in the marketplace (and/or serve others)?

What are the pain points?

Start by identifying the pain points you know you have. Ask your team. I’d bet you’ll have a number of things your team is aware of, but never have time to do it.

Chapter 6 of The DevOps Handbook says to “gain a sufficient understanding of how value is delivered to the customer”. “Conduct a workshop with all the major stakeholders and perform a value stream mapping exercise”. Figure out all the steps required to create value and identify where there are delays and wait times. Page 65 has an example diagram.

Can you put metrics on the delays? Examples: We spend 10% merging code. We spend 10 days of manual testing and 10 more days fixing issues before we can create a build to release. (see some good examples on pg 144 in the DevOps Handbook)

Put a number from 0 to 10 on each of the capabilities.

Where can we automate (environment creation, database seed data, database schema updates, unit testing, UI testing)?

Where can I start adding tests? I have a full talk about this that I named I know I should be Unit Testing, but I don’t know how or where to start

What are the major bottlenecks in the flow of your work? In the Phoenix Project everything seems to always come back to the same guy. Phoenix Project - 90 “Improvements anywhere besides the constraint is just an illusion” another great quote is “improving your daily work is more important than daily work”

What are you biggest areas of technical debt? Do you have large classes that are tightly coupled? When you can one area of the code, do other bugs pop up all over?

Make goals together

Jeremy Likeness has a great article about goals . Where do you want to be in each of these capabilities in 1 week, 1 month, 1 year?

Choose where to start to incrementally improve

First: Do you have your code in source control? If not do this ASAP!

Move your code to Azure DevOps?

Consider moving to git? (Note: TFVC is still supported and still a good option, I’ve just found Git to be much more enjoyable, pull requests much better then TFS code reviews and productive after getting over the learning curve. I recommend having someone available who has experience to help others out when they mess up a branch).

update note: since I first wrote this, I’ve since helped plan and execute a migration from TFVC to Git

Second Get a gated check in build running. Ensure the code builds, and run unit tests.

Third Move towards pipelines that create artifacts and deploy them automatically to your test environments. Automate everything you can.

Is there a new project or small team that you could experiment with these concepts? Then you’ll have stories to share about wins and what went well and able to spread this knowledge to other teams.

Combine a Dev branch with a Main branch? How do you isolate in progress code and your QA from things changing all the time?

• I think you’d have feature flags, and versioning in a build (installers can help with this, versioning dlls is a must as well for support) that QA can choose the version set they want deployed

Can your developers run the part of the system they are working on locally?

• Local Database
  • You need to be able to create the db with seed data quickly and reliably
• Do you have a complex environment?
  • Multiple services?
  • External dependencies (CRM, Sharepoint, Redis Cache, others)
  • Can you simplify?
• How long does it take to get a new developer up and running.
• Do you need to invest in more documentation of the system and setup?

Move DB into source control with dacpack ?

Can you create environments with scripts?

• Local development (get new developers going quickly, not taking days and having different setup then each other/environments)
• QA/Test/Demo/Prod

Breaking up monolithic systems into smaller pieces?

What makes the most sense for your team?

How do you achieve always being deployable?

There are many options, again avoid paralysis and start with some small experiments or where you’d most help the company’s goals.

The DevOps Handbook suggests creating a pilot team, outside of the normal process constraints, to move quickly on these ideas in an isolated part of th system or different project. Then share these learnings and achievements globally.

Ensure time to work on this

All this identifying will be for naught if you don’t get time to work on things. It’s necessary to have management’s help in securing time. You may have to make a case in terms of technical debt (how is the current situation slowing everything down?). Show your goals and plans to improve.

Evaluate often (Retrospective)

This is a grand experiment and adventure. You need to re-evaluate things with your team and constantly look for improvements that can be made. End of a sprint is a great time to do a retrospective .

Ask “What went well?”, “What didn’t go well?”, “What can we improve?”. Take action items and do them.

Celebrate and share these improvements with the full company.

A Grand Adventure

via GIPHY

As Bilbo said to Frodo “He often used to say there was only one Road; that it was like a great river: its springs were at every doorstep and every path was its tributary. “It’s a dangerous business, Frodo, going out of your door,” he used to say. “You step into the Road, and if you don’t keep your feet, there is no telling where you might be swept off to.” Frodo Baggins about Bilbo, The Fellowship of the Ring, Three is Company” ~ Found with Bing on

Part Two

This is getting really long. I hope you’ll join me for Part Two to read more.

---