October 1, 2021 Process DevOps Agile Improving Organizations Productivity
Improving Organizations - Moving to a Consistent Flow of work
I’m working with an organization that has many clients with on-premise installations. This has been difficult to deal with as the various clients have different upgrade thinking. There are many different versions out there. The current upgrade process takes a lot of time and has a lot of risk. That pushes the organization to have less frequent, big bang deployments and “drops”. This is far from the ideals of DevOps and Better Value, Sooner Safer Happier of smaller batch sizes, deploy more often, “if it hurts, do it more often”.
We are working on Sooner, Safer and Happier Upgrades, but that is going to take a big investment. In the meantime, I helped lead discussions about moving towards a consistent flow of work to be able to have a release created after every sprint and validated by the end of the next. We hope to reduce that over time. Our clients might not always take the latest release, but that’s a people problem and we need to build trust, automate more tests, simplify and making the upgrades Sooner, Safer and Happier.
We are also reducing long lived branches, moving towards Trunk Based Development and other continual improvements that you can read about in my past articles.
Watching how Spotify organized their teams and delivers software was eye-opening for me when I watched it the first couple times years ago and their release train was an inspiration.
I created some diagrams with Mermaid Markdown Editor and have asked for permission to post them.
Hopefully this gives you some ideas, but this will depend on your team, organization, culture, complexity of your system, etc. Keep in mind that this is an improvement step for us at the same time as setting a vision for where we want to go.
Never stop improving! Find the next weakest link (biggest pain) and improve that next and keep on going.
Goals
- have a QA validated release on a consistent schedule and reduce that time as we get better.
- Improve flow time (idea/bug to providing value and used by the client)
- Improve Quality and reduce regression issues
- Reduce the need for manual drops/hot fixes
- Reduce work and confusion about all the different versions and when the next release will come out
- We still need to learn more about feature flag and use them.
Viewing the diagrams
If you’d like to view a higher resolution version, uou can copy paste my Markdown code into the Mermaid JS online editor. I’ve also had success using an extension inside of VS code to preview it.
Development Flow
This is the normal flow of work.
Here’s the markdown for Mermaid JS
graph TD
subgraph Preparation
A[New Feature or Bug - State: New] --> B[Investigation] --> US1
%% Product Owner PO investigates idea for a feature, gathers client feedback
%% Determine feasability, how to apply to the product beyond a single client
%% Creates User Story or Stories with requirements
US1{Sizing: Multiple User Stories?} --> |Yes| US2[Create Epic]
--> US3[Assign a Feature Team and Channel] --> US4[Feature Flag or Feature Branch Needed?] -->C
%% bring work to the team, not people to the work
%% change our feature teams around to support this?
US1 --> |No| C
C[Priorization and HLOE - State: Approved] --> D
%% High Level Of Effort - HLOE, with developer input
%% will it fit in a sprint, are they small enough pieces of work?
%% what are the deadlines and expectations?
D[Sprint Planning -> Add to sprint] --> D2[Assign work - Dev and Testing]
D2 --> PreImp[Pre-Implementation Meeting]
%% 1 or multiple, PO, Dev(s), QA, Activations/Support
PreImp --> E{Enough Details?}
E --> |No| F[Investigation] --> PreImp
end
E --> |Yes| Dev1[Development Starts - State: Active] -->
Dev2{Feature Branch - FB?} --> |No| Dev3
subgraph Normal
Dev3[New local branch from master] -->
Dev4[Rebase From master often] --> Dev4
--> Dev5[Coding, Testing, Documentation] -->
DevPR[Create PR] --> DevPR1[Feedback, Complete PR]
DevPR1 --> DevPR3[Post Imp Meeting] --> DevPR2
%% all involved with the item should be in this meeting
DevPR1 --> DevPRDeploy[Deply to DEV01 & QA01] --> DevPRDeploy1[Run Integration Tests] -->
DevPR2{More to do?} --> |Yes| Dev3
DevPR2 --> |No| DevDone[State: Resolve, Test in QA01] -->
DevDone1{QA verify in QA01} --> |Good| Verified1
DevDone1 --> |Needs Work| DevDone3[State: Active, Record steps, Communicate with Dev] --> Dev3
Verified1[Close] --> Verified2
Verified1 --> Verified3
Verified2[Add UI Automation] -->
Verified3[Deploy to Int05] -->
Verified4[Run UI Automation Tests] -->
UITests{Passed?} --> |No| UITests1[Investigate]
UITests1 --> |Regression| UITests2[Record, Activate] --> Dev3
UITests1 --> |Not related| UITests3[Fix Test] --> Verified4
Done --> Done3[Follow Release Process and deliver the value]
Verified1 --> Done[Release Notes]
Verified1 --> Done1[Demo or Highlight reel?]
end
subgraph Feature Branch - FB
%% move to different diagram?
Dev2 --> |Yes| DevFB[Create FB branch from master] --> DevFB1
%% FB = feature branch this needs more refinement and instructions
%% don't rebase if multiple developers
DevFB1[New local branch from FB] -->
DevFB2[Merge from master to FB often] --> DevFB2 --> DevFB3
%% often = daily or more
DevFB3[Merge from FB to local branch often] --> DevFB3 -->
DevFB4[Development, automated tests] -->
DevFB5[Merge into FB from local branch for each working piece, feedback] -->
ReadyForPR{Done with working piece?} --> |Yes| ReadyForPR1
%% smaller PRs = easier reviews and quicker feedback
ReadyForPR1{Feature Flags in Place?} --> |Yes| PR
ReadyForPR1 --> |No| DevFB2
PR[Create PR] --> PR1[Feedback, Complete PR] --> PR2{More to do?} --> DevFB1
end
End of Sprint
graph TD
A[End Of Sprint] --> B
B[Create Release Branch] --> B1
B1{Any in progress work?} ---> |Not Finished| Hide
B1 --> |All Complete| C
Hide[Feature Flag, Complete or Revert from release branch] --> B1
%% for large features, Feature flags or features branches can be created at the start of the work
C[Build and Deploy to Test - QA approval] --> D[Run Sanity Automated Tests] --> G
C --> ReleaseNotes[Create Release Notes and details]
C --> E[Exploratory Testing] --> G
C --> F[Test changes since last release] --> G
G{Issues Found?} --> |Yes| Issues1a
G --> |No| Stable
Issues1a{Time to fix and test?} --> |Yes| Issues
Issues1a --> |No| H[Feature Flag, Complete or Revert - Next release]
Issues[Analysis, Fix with automated tests in master] --> Issues1
Issues1[Verify in QA01] --> Issues2
Issues2[Cherry-pick from master into release] --> C
%% can we fix and test before the end of the testing period?
%% if too close to needing a relase, then
subgraph Approved Release
Stable[Publish release] --> Stable2
Stable2[Notify Activations/Support] --> Stable3
Stable3[Upgrade all clients ASAP] --> Released[Delete feature flag code]
%% even if clients can't update this often, the flow still works
%% they may choose to release less often, but smaller releases => smaller batch size
Stable3 --> Released2[Reflect, Learn, Celebrate]
end
Timelines
journey
title Development Timeline
section Sprint 1
Develop, Test, Automate on the master branch : -100
Create release branch : -100
Start new sprint - Handle in progress items, help test : -100
section Sprint 2
Update INT Test Environment: -100
Test Changes And Data differences: -100
Stamp of approval & Publish installers: -100
Create Release Notes with change notices : -100
section Sprint 3
Goal - Upgrade all clients : -100
journey
title Development Timeline - Ideal 1
section Before Sprint
Planning, Grooming : -100
section Sprint 1
Pre-Imp, Develop, Feedback, Test : -100
Automate on the master branch : -100
Create release branch : -100
Finish in progress items, help test : -100
section Sprint 2 - one week or less
Update INT Test Environment: -100
Test Changes And Data differences: -100
Stamp of approval & Publish installers: -100
Create Release Notes with change notices : -100
section Every 4 weeks
Goal - Upgrade all clients : -100
Hot fixes and Drops
I leave this to the end, because I hope you don’t do this. Our team doesn’t trust the build/installers with past and present problems, it takes too much effort to upgrade, we have too many versions, the batch size is too large between builds, etc, that it’s become normal to create drops (build the code, copy paste dlls to servers or sql scripts). We really need to get out of this. It creates Frankenstein environments, is really hard to troubleshoot and a lot of coordination overhead.
We will improve and the consistent release + more automated tests and better quality are a few of many steps to reduce the need for drops. I hope they become a rare exception and that you never have to do one this way!
graph TD
A[Hot Fix Needed] --> B
B[Analysis] --> BB
B --> Drop
%% reproduction, meetings, talking with Support/Activations/Pete/QA/developers
Drop{Do we need a drop?} --> |No| I
Drop --> |Yes| BB
I[It's not a hotfix, move to backlog]
BB{Reproduce in QA01?} --> |Yes|C
BB --> |No|NoReproA
NoReproA[Are you sure? Is it already fixed? Config/Data?] --> NoReproB1
NoReproB1[Investigate. Why did this happen?] --> NoReproB2
NoReproB2{Start in master?} --> |Yes| C
NoReproB2 --> |No| NoReproB3
%% repo in environment with the client data
NoReproB3[Find correct branch] --> NoReproC
NoReproC[Development, automated tests in branch] -->NoReproD
NoReproD[PR with feedback]-->NoReproE
NoReproE[Cherry-pick to master] --> D
C[Development, automated tests in master] --> D
D[PR] --> E
E[Verify in QA01] --> F
F{What release branches need this?} --> G
G[Cherry-pick to release branches] --> J
J[Create drops] --> K
K[Create Instructions] --> L
L[Apply drop to internal test environment *no builds*] --> M
M{Verify} --> |Yes| Client1
M --> |No| Z(Reflect, Learn, start over)
Client1[Drop on client test] --> Client2[Verify] -->
Client3[CIPs, Apply to client prod] -->
O[Retrospective ]
Please consider using Brave and adding me to your BAT payment ledger. Then you won't have to see ads! (when I get to $100 in Google Ads for a payout, I pledge to turn off ads)
Also check out my Resources Page for referrals that would help me.
Use Swan Bitcoin to onramp with low fees and automatic daily cost averaging and get $10 in BTC when you sign up.