I'm interested in what your opinion is on Code Reviewing practices, in light with your strong preference for CI workflow. Some companies I've worked at, have a pre-merge code-review process like Github's PR system that needs a review before being merged into trunk. A pre-merge code-review process or mindset would then only work in an environment that considers a pair-programmed changeset as properly reviewed? Would you consider a pre-merge code-review policy to be an impediment towards incorporating CI? In companies that have a post-merge code-review process this problem doesn't seem to exist, but one would probably require an adequate level of (non)functional coverage in their automated tests to catch mistakes quickly, such that code-review is only needing to (ideally) focus on design. Are you therefore of the oppinion that a post-merge code-review process (or pairing as mention before) are two key processes to have in place for enabling CI?
This is a topic I see come up frequently, so I hope he does a video about it. If I remember the topic correctly from the Accelerate-book, as much as possible from the pull requests should be automated, leaving stuff like exploratory testing and such to the humans. And for the pull requests that remain, the team should make code reviews a high priority to ensure flow
Same question as I wanted to pose as from my experience the code review part is the one that causes your branch to stay non-merged longer than the design and implementation parts put together and even multiplied. For post-merge code reviews I don't see how they would work as they require far more experienced developers to go through (specially with continuously pushing unfinished work that makes the same mess as merge instead of rebase does in git... nightmare for reviewing so you just rubber stamp it and be done with it). With partially complete pushes you also require far more experienced developers as they must understand that something is still work in progress and somehow still envision the final design. And in my experience experienced and high quality developers are hard to come by so I still can't imagine getting higher quality code from CI compared to shorter tasks, rebasing and serious reviews of the design. Short term faster... Yes. Long term more maintainable... Can't imagine how.
@@defeqel6537 I've seen that video and it might be just me but I don't see pair programming as replacement for code reviews (at least not those that smell a bit of formal code reviews) - to be honest I can't even imagine somebody constantly messing up my train of thoughts in that manner...
I recently retired after 47 years in the field, slowly trending from coding to writing tools to version control to configuration management to as close as I could get to continuous delivery, which wasn't very close because I could not convince my teams of some key points. I am hugely enjoying your videos and your explanations, and thought I'd share a few observations from my time in the field. 1) I haven't heard you say this actual phrase, but some major luminary very early in the software world's development stated "Nothing of any significance has ever been accomplished by a team of more than 7". I've heard you express much the same idea, though, quite a few times. More people than that exponentially increases the chances of miscommunication about the design of the system. 2) I used to sort of half-heartedly suggest feature branches for extensive, disruptive, dangerous changes, but my teams always resisted it hard for the reasons that you clarify in this and the other videos about "don't feature branch"; essentially, merge hell will invariably result. 3) I worked in about 9 different shops that all claimed to be doing Agile Development. In one of them, they were actually doing Agile Development. In the others, they were indulging in rituals designed to fire off the buzz words managers wanted to hear, but these rituals usually were bumbled so badly they didn't do much good and in some cases were actually harmful, like when the boss was the scrum master without training and did most of the talking.
Once more insightful thinking of how sw can better made. Thanks Dave Farley. Since this is a so difficult practice to get the teams and/or Devs to move to, if possible I would love a future video of how to start with it. Something like what needs to be done from the beginning, and a possible adherence to other techniques, etc. Thanks in advance.
One of the issues I have around CI vs Feature branching is more related to maturity. As far as I can tell, it works great when you have a stream aligned team working with a code base where most of the work happens in parallel. When you have a monolith, the problem with CI (for me) is that integration events become unpredictable. One team may be testing a set of changes, when another team commits a change that is automatically deployed. All of the tests cases that had been performed at have been invalidated as a result of the change. That's primarily why we've used feature branches in the past, to isolate changes just long enough to test them.
@@michaelslattery3050 Does it not get somewhat tricky to understand the subjective concept of 'short-lived' here? How would you define short-lived? How would you enforce it? I have seen teams that aspire to that and somehow these short-lived branches suddenly become very long-lived.
If you don't think a single developer working alone can benefit from feature-branching, I don't think you and I are talking about the same thing when we say the word "branch", which would certainly explain the discrepancy. a "branch" collects units of work (commits) together. Each commit is the smallest change which works on its own. Each branch is the smallest change which justifies its own existence. Within a branch, prior to integration, commits are not a commitment - they are rewritten / re-ordered regularly, as the discovery process continues. Integrating them doesn't make sense, because they aren't final yet. Forcing integration prior to the change being done makes the history unreadable because each change becomes separated from its justification. And of course, the one and only reason we're even using version control (instead of just sending each-other diffs against the last timestamped snapshot) is to keep track of *why* changes happen. I think we both agree that integration needs to happen very regularly, but disagree strongly about the acceptable scope of both feature branches and commits. I don't want to be "more continuous" than the smallest justifiable unit. I also disagree that "a branch which can't be integrated" is the same as "lost work", and I would have thought (based on your other video) that you would be a strong advocate against that sort of position. Sometimes code needs to be rewritten based on new information, and when that happens, it's always faster than writing the original code. It really sounds like you're talking about traditional, long-lived branches, rather than feature branches. As with so many discussions encountered on the Internet, I'm pretty sure I agree with everything you say, except for the specific terminology.
@@OggerFN their individual justification, yes, but not their overall justification. Not the context of which commits relate to it, in order to define the goal. If your commit is fully justified without any related commits, go ahead and merge that. If whoever is reviewing could ever ask "why?", then it should probably be grouped into a branch. Meanwhile, if whoever is reviewing could never ask "why?" for any of your individual commits, then your commits are too large.
@@________w You should write the task the commit is supposed to accomplish AND the reason for that in a commit message. For example a habit I have been using is naming the ticket a commit is relating to in it's message in order to "group" commits.
- "When the information is modified in each place, then the long the time, the more those copies will diverge from each other". Well that is the description of entropy, it is so fascinating to understand that basic properties in universe affect everything even how we commit ...
You must still pass the integration pipeline, though.^^
3 года назад+19
Love the sarcasm! And also, I agree! It's perfectly acceptable to use many different strategies and CI and Feature branching aren't mutually exclusive. You can do feature branching in one step of your process and CI in a later stage. No worries. Honestly, this video is too dogmatic.
As someone pointed out...only if you get through the pipeline. And that should have human beings in it after the automated checks. :-) I mean..i heard about QA Somewhere......
@@timothyblazer1749 I don't think that this pipeline as described in the video includes human beings. (EDIT: You wouldn't be able to integrate that often if QA would look at every integration.)
I think big take away we need to remember is: “Continuos integration works by limiting the time period and thus limiting the fan out of the cone of changes, this reduces the risk of a clash”. That makes me wonder, if we use feature branches but could somehow guarantee the time period of unchecked code is the same or less as not using feature branches and forbid branching of feature branches to avoid the cone, are feature branches still bad?
That's precisely what you should do. In fact, this approach has many added benefits, such as you can increase your confidence, that no bad code will be pushed to trunk. With trunk based development, nobody can prevent a developer to accidentally push a commit that fails to build or breaks the project completely.
Depends on how would you guarantee that. If you have only 1 "big" feature branch and everything else goes to master, then you can actually keep the system integrated just fine (or systems to be precise - one on master, and one in the feature branch) just by merging master to that feature branch regularly. This has the downside of someone making a change in master that breaks you feature branch, but since that change is small, you can easily fix it in your feature branch after you pull that change from master. But what if you have 2 feature branches that take long to finish? If you merge them between each other (as well as pulling from master from both), then you have still all systems integrated AND keep the advantage of neighter branch "polluting" master until it is finished. But now you cannot release one branch without the other. One solution is to only merge those 2 branches "one way" (say, only merge A to B), but then B cannot be released without also releasing A. Also this strategy doesn't work for multiple unrelated branches. So you are left with 3 options: 1) don't work on the same code: when a big branch is ongoing, do not change the same codebase / features as this branch is changing. That way, you will avoid conflicts when merging it. 2) bite the bullet: just give up and prepare to do a massive merge once in a while 3) make smaller changes: by integrated smaller changes, you reduce improve in both above points - you are less likely to "hit" the same code as another change if those changes are small, AND when you do, the conflict is going to be small too. Of course, making smaller changes isn't as easy when you are not used to it and requires a complete paradigm shift in how you look at programming, especially if you want to go "i will work on this feature for 3 weeks and rebase my commit as I want and it only has to work at the end" to "I make a change and push it as working commit every 10 minutes", it will not happen over night.
Hmm, I'm still not 100% sold... Sometimes CI sounds like a really great idea, but the problem also is what do I do when I approach larger refactorings (like updating dependencies to a new major with breaking changes) that take time and will leave the project in an unworkable state until completely resolved or changes that we are not sure if we should carry them through... How do really revert a series of commits in a CI world, where they are mixed in with other people's stuff.
That's a really good point. I think CI as described is good for actual user-facing feature development, but you need a more nuanced approach when doing things like what you're describing. The platform upgrade is the obvious exception, but there are other types of changes that require a careful decision on a case-by-case basis. "Changes that we are not sure if we should carry them through" is an interesting one. Say for example you want to unify how your application does caching, and you need to audit every single network call it makes to update it to the new system. If you take the time to update each one before sending it out for review, and then (hopefully) merging, what's the likelihood of running into some catastrophic merge conflict? Seems pretty high. Maybe a smaller proof of concept would be more helpful then. But then say your team does dozens of such experiments, starting on them, but never actually following through to implementing them project-wide? Each experiment you run creates ambiguity in the codebase, where it's no longer clear just from looking at the code what the right thing to do is. In this case, maybe it really would be better to make a long-term branch, knowing full well that lots of code will have to be essentially rewritten multiple times over the course of the project. The new pattern isn't the deliverable in that case, but rather the consistency. I guess I'm not seeing easy answers either way.
If you wait for your code to rot into a state where a "large all-encompassing refactoring" is needed, you've already failed. Clean up your code before committing it and you won't have this problem.
@@froobly agree with Ross. even with legacy code, the first step is to write tests - both at a "unit" level and black box level (won't go into details). Make the constraints of the behavior of the code explicit, refactoring can definitely be done step by step in incremental and relatively risky way. if you are dealing with huge migration like breaking framework changes, then I would agree CI doesn't really apply, because it is a different problem. but you still cannot get away with automated tests.
3:20 I often feature branch when I'm alone. That gives me the benefit of switching features when I get stuck and I can come back later. You may argue that I should comment the changes out or implement a software switch when rewriting a function. That can indeed be a solution and I admit that I haven't tried it yet.
When working alone, I prefer to push every change immediately to Production, potentially toggled if it's still work in progress that users shouldn't see. The only way I can know whether my feature works as intended is to get real usage on it. Of course people's philosophies differ.
I think the context of what kind of team and what kind of product you are working with will affect whether this is effective. When you have many disparate repositories of code, each with few changes over time, it makes more sense to use pull requests, since things move so lethargically. But if you spend most of your time in one repository, CI seems like a much better option. For example, you won't have to re-synchronize your feature branches to ensure you're on latest if you are always working off trunk.
Interesting. We used feature branches at first - it is hard. Now we embraced, trunk based development and release branches. Release branches allow us to keep CI intact and enables patching. For new features, finished or not - we use feature toggles. Feature toggles allow several things: review partially implemented UX, understand complexity of database migration requirements. We also starting to embrace side-by-side approach vCurrent-by-vNext. You may say it is a lot of maintenance, on the other hand you are doing it anyways. Now we have much less feature branches, always up CI/CD, easier database CI/CD. Love the videos. Would like to see you touch code reviews topic. Often it is seen as one person offers ideas and knowledge sharing on optimizations or better code structure. But I think that it is more important to review how people handle exceptions, how people ensure that APIs are complete. Not all devs are experienced. I mean code reviews are not about beauty, it is about cleaning sh** left by others. This is more realistic and gives more value. Code reviews is about getting more experienced, not just about learning more patterns or algorithms.
How to use trunk-based development in a legacy system where there is huge technical debt and most of the code doesn't have good tests we can trust? How to ensure quality where each commit to the trunk can potentially break the system. Of course, we try to write more tests and do refactoring doing some new feature, but writing tests is not always possible when everything is coupled together.
Unfortunately you're in a difficult situation so you have to make moves to reduce the number of branches... Of course rewriting is one way of doing that, as nuclear as that may seem. There is no theoretical limit to how hard to maintain a system can become, but at some practical level it can become unmaintainable. All the best
If the conflict is only coming up in a meeting, you don't have good communications or process. Those should be being resolved individually by the actors involved, for the most part.
Just wait until a "certified master" that doesn't know anything about development starts hassling you about why "can't you just merge it in".... and to add insult to injury said person usually doesn't even posess enough of an understanding of the concepts involved to be able to understand a straightwordward explanation... whilst simultaneously they're looking down on you for being a "boffin".
@@dafyddrees2287 what you describe is some toxic work environment. In a healthy functional team this is not a concern at all. I wouldn't work in a team that would act the way you describe.
@@vyli1 "healthy functional team this is not a concern at all." That's a very subjective thing to say. All the TBD I've worked on has been awesome and all the branch-tastic stuff has been awful. It corroborates what Dave has been saying in his books and videos.
And that is why I enjoy the whole microservice hype in web development. It allows you to work in a small team. This way, you can have feature branches without worrying too much, given the tasks are also correctly planned in a way where the conflicts chance is very low. For such teams I find it very easy and enjoyable to have even bigger PRs, as they contain the full feature and you can observe all of the changes at once, giving you the opportunity to find more optimization routes.
I think feature branching within a single repository can work, as they are visible to the team, but you need to treat each branch sort of like it's a trunk and regularly merge branches into other branches, not just mainline trunk. The advantage of this is if you do get some nasty merge conflicts you can put them off, let those branches diverge a little while you consult with your team about how to resolve them and prevent them in the future, all while continuing to make forward progress. Some of that progress may end up being discarded, but some of it won't, so overall you aren't completely halted by having to deal with the divergence.
It would be great to have a clear explanation of the hypothesis. It's not clear what feature branching is. I believe that CI is not alien to use your own branch to work on your changes and then allow peers review. I understand that short time feature branches could be used. It's not clear to me if those are not even accepted in CI.
Added costs of feature toggles is not always worth it. It takes a lot of upfront work to design and plan work on features so that they wont clash with each other. If not managed carefully, it can end up in really messed up spaghetti code. Especially in early stages of projects, when no one knows what they are really doing.
Not only that, there are data model changes that simply cannot live together with the legacy model. What kind of toggle allows you to have a table column value unique in a place of the code and allows multiple values in other? Answer: none that realistically enforces the constraint. I’like to hear how his version of CI deals with that.
3 года назад+4
Day one: Checkout a new feature branch from develop. Day two: Merge said feature branch into develop. Rebase first if needed (if there are any conflicts). Day three: Merge develop into prod or staging or whatever deployment pipeline you have set up. Day four: Repeat. Feature branching and CI aren't mutually exclusive. You can do both. And I think you should. Keeping features small is just a sane thing to do and has nothing to do with CI or feature branching. If you can't keep the feature branch small, you should rebase it continuously (against the develop branch) until it's ready to merge. So, do feature branches against the develop-branch and CI/CD between develop and master with whatever steps in between you want. Ofc there's also other ways of doing it. Don't be dogmatic. Be flexible and adapt to needs and the preferences of your team.
I like this approach, but I would say you can also automate "merge dev into master": Day 1: Checkout a new feature branch from dev. Day 2.0: Merge said feature branch into dev. [you don't really need rebase, you can fix merging conflicts without it] Day 2.1: Automatically run tests using the lastest dev branch. (optional) Day 2.2: If all tests pass, automatically merge onto master (CI) (or merge manually) (after merging to master) Day 2.3: Run automated tests using master branch And of course, 2.2 could be done manually, allowing lets say a PR request or a code review to be made before it is merged.
As far as I understand, this is essentially the same thing as trunk based development. You could skip the "check out a new feature branch" step and get the same result.
Dave, I agree with what you say on so many levels. I am one of the people commenting on the other video. I would like to also refine my view. Please keep in mind, what I don't say explicitly is because I think we see things the same way. Let's recap what was the main issue in the other video's comments: the beginners of the team. Beginners are a healthy factor in a team for other reasons than the quality of the code. Technically, we tackle beginners by reviewing their code. Another reason to review the code is compliance. Either way, beginners will make mistakes that cannot be caught automatically, no matter how good your tests are. Let's add one more to the list of things to keep in mind: we fully agree that things should be integrated as often as possible. What we don't agree on is doing it at every commit. This is literally insane, for the reasons I hinted above. What we do instead is redefine the notion of a "feature". It's not an user story, but a subtask. Let's take an example. The junior on my team has been working for almost two months on a user story which ended up having about 30 subtasks. So on average, he finished (and integrated) one feature every 1-2 days. The reality is, some features (subtasks) were easy even for him, so some features were half a day, while a few took him as long as a week. I am fine with this, because we reduced the risk of breaking master, while also reducing the waste of effort (it was the beginner tinkering, so 1 week does not cost that much). Keep in mind, most of the time we were integrating every 1-2 days. We can also integrate all combinations of feature branches automatically and test them because since they're short-lived, there's less of them! We remove the branches after integration. So roughly we have only as many branches as the number of people in the team. Integrating at every commit would work only in a team of only seniors.
Not so. I have operated this way with junior devs, even raw beginners, many times, but you do it in combination with pair-programming. It is the fastest way to get juniors up to speed and productive, that I have seen.
@@ContinuousDelivery pair programming for 2 months continuously is not productive. They need to chew over problems themselves, sleep over them, then come back the next day and solve them. Yes, we paired up, and in addition to that he asks for help every 2-3 hours. I found this works best, in addition to pair programming. Also, the other big issue raised by me and others is code review, which you have not addressed so far. What is it then, reviewing each commit?
With open source software, you do three merge/pull requests. If and when they are successful, these qualify you to be a developer. As a member of the development team, continuous integration is the way to go. Note: The size of the project does not matter. Whether we are talking about that upcoming JavaScript framework or the Linux kernel, three contributions are enough, roughly.
Continuous Integration is a practice that requires a high trust environment-where you can trust the people on your team. In low-trust environments, like Open Source development, Pull Requests shine. GitHub was largely founded on Open Source, so it makes sense that Pull Requests were largely invented by GitHub. CI would only be appropriate for a small core team in Open Source.
Look at the Apache Maven project and how they manage all of their stuff, the same way I do, a single branch. Period. No need for a release branch either, to cut a release, tag it, build a release from the tag, and move on. Mainline development is what I recommend for all projects, as I’m sick of cleaning up left over branches from others. Git and Jira feature branches just makes things worse, as it tends to make the processes we follow fit the tools we use. The tools should suit the way we want to work, not the other way around. The number of times I’ve had to say “Son, just because you can do a thing, doesn’t meant you should do a thing”. Especially around Git branching models. The only time I use a release branch is when I have really long life releases (and I’m thinking 1-2 years here) and I have non backwards compatible breaking changes in happening. A good example of this would be a change of underlying middleware or java version for instance. Then each release branch effectively becomes its own master/trunk, and master fades into obscurity. I probably just triggered some with that last bit. But from first principles, it’s the easiest way to work - especially when the devs are inexperienced.
At the company I work for, we implement CI/CD, but instead of master, we work on the UAT branch. Then once a week, we do a pull request to merge UAT into master. That way our production environment only gets clean and tested code, and hasn't crashed due to broken code in over a year now.
I'm curious about when/if Product testing fits in CI. Say that a developer (alone or working as part of a pair) is working on a feature, which was specified by a Product owner of sorts. There's always a chance of some level of misunderstanding between what was asked and what's being delivered. So it's now about whether the code works or not, but more about whether it does the right thing. In that case, being able to deploy a version of the code to a test environment that can be exercised by a Product person is quite useful. However, if we're always committing to trunk and the code is doing the wrong thing, it might already be too late. I can imagine one solution for it being feature flags for every single customer facing feature. While I can see that working, it can clearly add a ton of incidental complexity to the act of adding features and changing a product. What's your recommendation for that kind of scenario? Thanks!
In general I'd agree, but in some cases it still seems useful to branch to me. Especially when doing some more experimental stuff, where you don't know, if you even want to add this change. But if you already know, you're going to merge this branch/feature sooner or later, it's always better to merge it now. It's much worse, when others already depend on a feature, which often changes, than when you are not able to merge this feature anymore. And you will always be able to merge your feature in most cases.
It seems to me that you need to watch the first video and use one of the techniques for deploying without releasing. What I hear when you say "feature branching is useful" is "I'm not good at deploy-without-release". I think you should get better at that and see if the tools, techniques, and paradigms you develop change your mind about feature branches over time. If you always fall back to feature branches, you'll never get good.
Nope - that's not continuously integrated is it? The risk is in the merges - and not understanding what other people are changing. If you wait until they've finished "their" feature (big problem already there with code ownership) then you're merging great big lumps of changes - which is asking for trouble.
@@joek4563 Great - now you have a broken build with tons of shit to track down. It will be much more difficult to sort out because you will be reconciling your finished work against the latest. Merges of huge amounts of changes suck absolute balls to sort out. It's much easier if you have to reconcile your tiny change against somebody else's tiny change.
@@joek4563 I've been doing this for almost twenty years now and the difference between trunk-based dev and this gitflow bullshit is massive. You just can't do big system-level refactorings at all if you're going to have lots of branches. You'd either break everybody else or never be able to work fast enough to deal with all the conflicts you'd get all the time. The only option is to work WITH other people - not to temporarily hide away from them.
I don't think your wrong, it's just that the software industry case been convinced that a feature is the smallest change worth integrating. This allows for all unit tests, functional tests, etc to be done over several commits that then mark this feature complete. We more use feature branches as short term backups then branches. I think where you would make this all one commit and the push it to the trunk, is really equivalent to a squash commit on the feature branch before merging. Also GitHub better supports the feature branch model with PRs, and other expectations of that system. Where would the review go in your CI model?
Thank you for the video. How would you go about prototyping different solutions whilst working on the same branch? It seems to me that rollback might be a really complicated in this case.
The more often you commit, and therefore the smaller your commits, the simpler the rollback! However, for experimental things treat them as something different. Do them on a branch, but then throw the branch away (or at least ignore the code in it) and re-write the code from scratch to production quality once you have learned the lesson. This approach is sometimes called "creating a spike". A spike is an exploration of some idea, the result is the learning, not the code. That gives you the freedom to write crap code if it helps you learn faster.
My expierienced rule of thumb: smaller skilled good teamworking may start & come quit far with a strong disciplined checkin strategy directly on *the main* trunk/master branch, but those conditions tends to be not realiable with much more growing/larger projects teams especially including unskilled/newbies. Fitting VCS checkin & branch strategies depend of multiple aspects: a) project/task change-sizes & risks/fail costs (e.g.: not every deep re-design/re-strucure checked in as lots of tiny checkins but project-wide atmoic commit/submit), b) developers team's size, skills and discipline/motivation, c) surrounding IT & Development Infrastructure including build tool chain support, dedicated DevOps, Q&A and a good project (not directory) scaled merge/integration tool support (bare git is not enough)!
What about changes that can't be completed in one commit and also can't really use tactics like Dark launching. In one of my pervious projects we had to make significant changes to multiple connected db-tables. I don't see how I can get around using a branch here. Yes the other developers will need to merke, but else we have broken code in the Main branch.
There may be strategies to work around that. For example, you could maintain more than one database table (or column) to represent the same data in different ways, and drop the older version when nobody uses it anymore.
@@cristianpallares7565 I guess that would be a good solution. I just fear the amount of copied code and the work to make sure everything gets deleted in the end. Well... those big changes should not happen to often anyway.
@@christophlutticke7150 I think this is only one way, you need to have that copy, cause I think yo need to have no downtime for you product during the deployment.
I am working in a big software company. 6 years ago we had big problems with pushing features from the different teams back to master because of the changes by other teams as you explained. Now nothing changed except two things: 1) We must push any change as soon as possible back to master. Even if the feature is not completed if a pice of code, e.g. on function, is completed so it can be pushed to master without coder breaking it should be pushed. 2) Daily build We do not have any integration problems anymore. Because the discipline of the developers increased because they do not want to break a daily build code quality and efficiency increased. And the "old" developer who knows the "old times" saying that work is now easier.
Not to be confused by not branching based on user stories. It is still helpfull to create isolated environment, a.k.a. branch, for user stories which takes couple of days only to implement and integrate. It is the smallest agile/scrum unit of work that validates a branch. It is more easy to discard a branch than to go back in history in case something goes wrong and developer has to start implement the functionality from scratch. Issue is, as pointed out in a video, a feature branch which takes several weeks until it gets tested and merged. Just my 2cents of course.
Look at the Apache Maven project and how they manage all of their stuff, the same way I do, a single branch. Period. No need for a release branch either, to cut a release, tag it, build a release from the tag, and move on. Mainline development is what I recommend for all projects, as I’m sick of cleaning up left over branches from others. Git and Jira feature branches just makes things worse, as it tends to make the processes we follow fit the tools we use. The tools should suit the way we want to work, not the other way around. The number of times I’ve had to say “Son, just because you can do a thing, doesn’t meant you should do a thing”. Especially around Git branching models. The only time I use a release branch is when I have really long life releases (and I’m thinking 1-2 years here) and I have non backwards compatible breaking changes in happening. A good example of this would be a change of underlying middleware or java version for instance. Then each release branch effectively becomes its own master/trunk, and master fades into obscurity. I probably just triggered some with that last bit. But from first principles, it’s the easiest way to work - especially when the devs are inexperienced.
My favorite story concerning this topic was having a dev lead who had no concept of how to merge using git. The dev lead was responsible for insuring the integration branch was kept in order. Additionally the dev lead had to coordinate this for multiple pods. The majority of our pod was able to commit and merge to the integration branch on our own. The dev lead insisted on doing the merge for the more junior devs. I noticed that one feature branch worked on by a junior dev was spanning multiple sprints. I made sure to let the dev lead know that having that branch that long was going to be trouble. I was ignored and you can all guess what happened...The dev lead overwrote a lot of changes from the others. To end the story, that dev lead no longer works at the company.
Working in isolation in a feature branch keeps your unfinished code from being merged into the main branch until its completely done. If you check in code constantly to main, then you will have unfinished code or one big push with all your local commits. Keeping commits local without pushing is a problem because they can get lost if your machine crashes. You should always keep your feature branch in sync with the other developers by locally merging the main branch into your feature branch, atleast once a day. Though these features branches should also be very small. I'd would say that they should only live for maximum a day before it's being merged with main. Otherwise your features are too big and you need to break it down further.
I'll only disagree that a feature that takes more than one day is too big. I know from experience that some things just don't break down that small and the effort to force everything into smaller pieces invariably takes longer than just implementing the bigger feature would have.
This is an argument against distributed version control, not just branching. With DVCS, even if no one branches, separately developed code isn’t integrated until it’s all pushed to a shared repository. Merely committing won’t combine anything.
So the team I'm working with now has three branches: master, validation, and integration. If I'm introducing a 'feature': I make a personal branch, complete the 'feature', and submit a PR to merge to integration branch. I'm putting 'feature' in quotes because I'm referring to any changes in the code base. We generally limit our changes to take no more than one week of work. Twice a day, integration is merged to a new validation branch (ie. validation-2021-05-27), which runs an enormous library of validation tests. If any of those tests fail, the culprit PR will be found and either fixed or reverted before merging to master. These fixes / reverts will then be merged back into integration. If your PR was reverted, you simply pull integration to a personal branch, revert the revert-commit, fix the issue and restart the process. Now my question is: is this CI or feature branching? There are separate branches for features, but each branch only consists of a single person doing a maximum of one week of work. Does feature branching turn into CI when the amount of people on a branch, and the lifespan of a branch, are limited? For instance what if we implemented the same strategy, but 'features' were split into daily deliverable chunks, as opposed to a maximum of one week, does that count as CI? Edit: Just found a video where you actually explain the difference explicitly, guess I'll watch that later.
I'm a fan of CI, but I'm having an issue with the idea that a feature branch necessarily means longer times between commits. If I were to run a development team, I would require the same interval between commits regardless ... basically when a small feature is ready. The gain (as I see it) from feature branching is the ability to merge from trunk and see any conflicts or failures before merging to common code.
That's why team members talk with each other. You can't say people don't know what the other is doing. If you work like that then you don't have team, but instead group of individuals or problems with project management. Pushing to master everyday means fixing merge conflicts everyday. Is it best way to deal with this problem? Even better, you can't decide feature one is ready to deploy, but feature two need some changes. How do you do this? Do you remove feature two from master and add it after release? Finally you chose the most corner case you could. Multiple big changes in the same code base at the same time. I don't know on what projects do you work but in my carier I didn't stumble upon that situation too many times. Because as I said at beginning, my team is there on plannings and rise concerns if someone tries to plan features like that. If it's needed you have more possibilities. You can create feature branch for 2 or more people and they work on the same code base but not on main branch. Still I needed this technic like few times in my carier.
I think we can have the best of both worlds: consider the following workflow: The is a "trunk" branch called "dev", and each person creates a feature branch from it and then merge it back to dev after the feature is complete, with a pull request and ab associated code review. After an arbitrary number of features are merged, a new release os born, and it gets merged into "master", the production branch. If we create a "dev-ci" branch, and have the team muse feature branch, but at the same time rebasing/pulling from dev-ci and merging their changes into dev-ci frequently, we then achieve the ci goal, and the feature branch with commits filtered by author is still good for a traditional code review
We basically do what you describe in the first paragraph. I don't think I understand how the process you describe in the second paragraph differs, though.
@@ShawnThuris sorry for the typos. Instead of commiting and pulling directly on the trunk branch, you merge and pull/rebase from your feature branch keeping it synced with the trunk branch. It may sound as extra work to accomplish the same result, but it would still allow us to link work items/issues on the management platform (GitHub, azure devops, etc.)
I have the same problem. I couldn't find any good solution to do code review after code is merged. Comparing selected commits seems to be very challenging (or even impossible). That is the only blocker which prevents my team to commit directly into master branch. At the moment we trying to creating branches which lives 1 or max 2 days. But I would prefer to merge directly into master.
@@rafacypcer1408 Why don't you use tags or a specified commit message? You can review the "sum" of these specific changes iteration by iteration. We used JIRA and Bitbucket, and we had commit message conventions where we put the feature id and the task id. This enabled us to review the task alone and also the feature ("FeatureId/TaskId message"). Each commit was visible under both the feature and task in JIRA. We used feature branches but with this you still can observe what is happening with your feature, and you can always create more tasks under your feature as it is only closed when the team is done. The only downside I see is that you can't reject feature merges, just the merges of your little tasks. But with trunk based development it's not really a factor as you want to apply changes rapidly instead of creating a big/coherent feature changeset.
Don't use code reviews as a gating mechanism for changes. Code Review should be something that gets done to look at how and why things were done the way they were and how it can be improved. If you use code reviews for enforcing methodologies and finding defects use automation for that. Yes it's 'after the fact' and the incorrect code is already in master, but using CI it can be quickly fixed. The flip side is if you and the other developers are constantly pushing changes and pulling changes code is being indirectly reviewed all the time.
@@Rekettyelovag Thanks. I also use similar pattern for commit message. I'm also able to see in my managment tool (in my case Azure Pull Requests) connected commits. But the problem is that I can see the commits, but not files with diff's (which is obvious, because it most probably git limitation in my case). I don't know any git feature which gathers selected commits and show diff's in files comparing the last commit. I san see changes by selecting two commits - not more. I understand that with CI/CD we need to change methodology. CI/CD is used mostly for fast development - not for code quality. Code quality should be resolved somehow different - like changing team culture, improving automated tests, applying linters/code rules etc. For approach with code review before commiting to master we had step where someone could verify someone code (which blocked development - because code 'frozen' during code review shouldn't be temporary used, by someone else). Merging directly to master forces every developer to review changes everytime when he/she needs to pull new changes. But it happens constantly. With feature branches reviewer was analysing the whole feature context. With branching directly to master this context is lost, because of continues delivery.
So. How do you get the branch through QA? Oh...that happens after you merge? What about merge conflicts? How long to resolve those? We know that feature branching is convenient for developers. But that's precisely why it is bad for development. CI is far better an approach overall.
@@timothyblazer1749 In most VCS merges are more complicated than they need to be. Git for example always needs a merge commit if you merge something which is not just a fast forward. In Pijul, which is still in development, you only need a merge commit if two changes change the same line of the same file. And merge conflicts should appear less often anyway.
Branching does allow for an individual developer to create ownership over the feature they are working on. I'm not sure if it's always the "I'm the hero" motivation described in the video. Having pride over the work you do does create motivation to complete it which is important. It's also important to note that individual contribution does have a tendency to create tangents of thought that could become unpredictably useful or not useful. That variation can come in handy. That said, I do find it hard to argue against the idea of more closely collaborating to create better awareness of the application state. Maybe the experimentation and offshooting should just be left to a branch that's not necessarily meant to integrate for experimentation purposes; left to individuals who want to propose changes in structure and such.
Excellent illustration of why the ingrained scatter-gather pattern of working is inherently flawed. We're experimenting with mob programming where all these concerns melt away, since no one is sent to their cubicle to work in parallel and in isolation.
Interesting, I certainly agree with the principal, but think that using feature branch and PRs is useful. IMO I like to isolate CI from source control as much as possible, so there is one mechanism to trigger a build. Tags. A modular code base helps avoid conflicts more naturally also. Frequent conflicts between independent features is a sign that concerns are not well encapsulated or decoupled.
This is pretty much close to how I always worked. I usually work on the master branch and push/commit regularly. Pretty much any change that brings a benefit while not breaking anything. I use feature branches for bigger changes that take longer, but I can't check in the intermediate work because it might break the existing state. It would not compile or break features. Usually a bigger refactoring. Where I want the intermediate states in the VCS but not break the master. As soon as possible I'd merge the feature branch into Master and continue working on it there. In some cases I combined it with the Dark Launch or Feature Toggles. Minimizing the time I work on a separate code base. Funnily enough, I rarely have merge conflicts as I usually merge Master back into my feature. My colleagues kept longer running branches and had to spend much more time on the merges. Mostly because they rarely pulled Master.
There may be an issue with semantics; TLDR; If we branch off of the trunk, but the branches are so small that several of them are PR'd in a day per person, does this still count as feature branching, or is it CI/CD as you describe? I run a small team of 3, sometimes 4 people, and we never work directly off of the trunk (which is actually our staging branch, but that is another story), as we want it protected, and instead, we make what we call 'feature branches', which is often small break downs of larger features into tasks (so maybe we should call it task branches), small enough to complete at least one, but usually several of in a day. We want the trunk protected as we want to review all the code via PRs before it goes into the trunk. We do this so frequently, that merge conflicts are quite rare, and our CD pipeline is often backlogged near the end of the day. In addition to this, we have small 10 to 30 minute (depending on what needs to be discussed) meetings throughout the day to discuss what we are working on and the direction we are taking, which helps avoid having to re-do a lot of work at PR. This strategy involves, more frequent, but much smaller PR that requires very little comment. Seems to be working great for us, except as the lead I need to work on getting my code reviewed by the other members more often and not just self approve. So basically, does this workflow still count as CI/CD as you describe, because it's still frequent, or is it disqualified because we are not working directly off of trunk. And if it's the latter, is it possible to protect the trunk in any way in this workflow?
This is exactly my situation with my team. We create small task/feature branches that get made into a PR within the day. On a good day I can make as many as 4-5 small branches/PRs. The hard thing though is our code review culture. Because reviews require context-switching and there’s so many PRs that get generated, there’s too much pressure put on the two reviewers that each PR requires (and we don’t practice pair programming). What ends up happening is people only review 1 or 2 PRs early in the day before tuning out of reviews to do their own work until the end of the day and by that time there’s a backlog of more PRs. Curious about the answer to your question and the code review process too.
@@DanielSanchez_dsanc89 The short meetings I mentioned later in the day is where we do our code reviews, which helps contains the context switching to specific, narrow parts of the day, and since we do it verbally and with screen sharing, it goes faster as well, and we do them in batches. And the other thing that I mentioned that helps reduce PRs is showing the code to each other (and hence your reviewer) in a checkup meeting earlier in the day, before any PRs, to pre-empt mistakes before they are made and need to be corrected in a PR. So our meetings are an early morning standup (7:30), where we plan out what we are going to do (but we sometimes skip it if we discussed it at the end of the previous day), then we have an 11 AM code checkup meeting where there are no PRs expected (but sometimes quick tasks get PRs by then), then a 2:30 PM code review session where we are supposed to do all the PRs (but in practice half the time it's another code checkup), then a 4:30 PM staging review session, where we are supposed to see the finished work running in a staging environment (or production with feature flagging if we would not have a staging environment) (but in practice half the time it's code review and seeing the code running on our local dev machines). The goal is to get your code running as close to production as possible on the same day that you are making it. Our client wants a staging environment, so the closest we can do is a staging server, but we do have feature flagging so we can deploy but disable incomplete features to master, so it's seldom far behind.
@@DanielSanchez_dsanc89 Sounds like our situation: the review tasks piled up. We switched from a gated review process to pair programming. We are not pairing for every tasks, but for important ones (with a more or less accurate guess)
@@DanielSanchez_dsanc89 I've found that talking about your goals and ideas before-hand can help reduce the burden when reviewing PRs. The reviews go from "what is being attempted here" to actually evaluating whether the code accomplishes the intended changes (without side effects).
I was struggling to articulate similar thoughts. Short-lived branches for PRs of small, well-focused changes; and being disciplined to always keep master in not just buildable but in a state where it is deployable to production seems like it achieves the desired goal.
What if one day your product owner decided not to release and undo that one complex feature (i.e. shared library change, db schema update) that everyone else had already integrated deep into the trunk?
This is the reason you need loose coupling in software. Ideally, you'd have some level of abstraction between your code and dependencies, such that you can swap out one dependency for another, one DBMS for another, with the core code blissfully unaware of the change. It's why the Repository layer exists in Android applications.
I have a shell script, git-sync, that replaces git pull --all + git rebase + git commit + git push, which mitigates some of these issues. Dave criticizes this approach at 7:21, but my script goes farther. It also checks for git conflicts across all other active features branches (we delete branches after merge), and list other branches. Finally, we have very small tickets, and we break them down even smaller to tasks which are safely atomically merged to trunk. We aim for "tasks" to last no more than 24 hours. In stand-up, we view branches older than 24 hours. Of course, none of that's full integration, but it gets us very close. Close enough to be better than having tons of feature flags, which is often required for trunk-based.
I think the concept of CI has a lot of merit. We had to change our concept of a 'feature' to help work towards this. On the basis that 'no one should commit code to master that breaks the build', we couldn't get an entire 'feature' developed quick enough to commit to trunk/master, so we used feature branches. But then we just found that if we broke down what we meant by 'feature' into smaller bits, we could do commit to trunk/master without 'breaking' anything. This did lead to some issues about testing these 'mini-features', but nothing insurmountable. And breaking a classical idea of a 'feature' down into 'mini-features' gave us a lot of insight into just what and why the 'feature' was in the first place. This worked well with a general 'Agile' thinking in that there were a few times our initial 'mini-feature' break down showed us where we had gone astray.
I'm thinking here that even in a team that works using "feature branches" and has periodic long delays in "pull requests" getting reviewed it would still make perfect sense to split the work into a sequence of small focused changes for the reviewer to review simply because it's gonna be easier to go deal with and to give feedback on compared to dealing with a big ball of mud of changes that inevitably turns into merge hell. To me this seems like it would be a clear step forward by practicing the parts of CI that one can in a broader environment that doesn't do CI... maybe this could be used to smuggle more CI thinking in there in the long term anyway if folks don't believe in its benefits until they themselves have seem them working for someone else in a part of what they do. This is seems like something that is perfectly doable with just about any regular toolkit. This whole thing reminding me of how in an environment that isn't agile the easiest way to make things easier for oneself is to have one's own surroundings self-organize (even if it's just you) to the extent one has the ability to do so and to just work like an agile organization within a wider organization because it's still better for one team to be agile inside bigger organization as that at least still makes working more efficient and pleasant for you as a developer.
I do get the overall idea but of course there exist several cases where feature branching is actually more productive and perhaps even mandatory. A few examples that come to mind: 1. perform a refactoring in order to replace an external library 2. Any form of "experiments" that are not supposed to be deployed in prod 3. large changes that are difficult to release "darkly" (or that can be released darkly, but with a high extra cost in coding the "darkly" (which is actually a feature itself). Again: I get the general advice but in my opinion "continuous" must be taken with a grain of salt, as with most practices.
I'd say it the other way around. CI works for nearly all cases. Occasionally branching, not necessarily feature branching, is useful in unusual situations. So "take branching with a grain of salt". The other take on this is that CI applies a pressure on you to think a little more about the design of your system. So, for example, I think it extremely unusual to need to use branching for your first example. Sure, I have done it, but these days I don't need to. Because I'd probably have a facade or adapter between my code and the third party library to make my code testable. So switching out the library would be easy. That is only not true when your code is tightly-coupled to the library, which is in some, very unusual, circumstances I may allow in my code, but very rarely.
thanks for your comment@@ContinuousDelivery. As stated in my first comment, I agree that the worst evil is multiple long lived branching, to the point of every single developer in the team creating a personal branch which quickly becomes impossible to merge (I personally witnessed a few of these edge cases). Yet sometimes branching is good, but you (that is: not the single developer but the team maintaining the codebase) must know why and what you are doing. Other aspect, this being more a technical detail: what do you think about the "short lived branching" technique, compared to the "commit to trunk"? In my opinion the short lived branching (branching but merging the branch very often) preserves the goal of continuous integration but is more solid in terms of protecting the main branch from breaking commits and it's easier to implement with modern CI tools.
But I do have a question. If I have to create features, that can take more than one day, I must create a new branch, and create a Merge Request, right? Or even if my feature takes more than 2 days, should I merge it into master without review from other developers? I agree with CI, but I have some doubts about large features and hotfixes.
The client I'm working for they have a controlled feature release, so constant merging seems impossible for this case. We only merge if the feature is business approve else we don't. Is CI can be implemented with this case?
I totally agree with fact that the longer you wait, the more clashes and conflicts may occur. The problem I faced with my team is that some time the work on a feature is incomplete before a deadline and we wanted to deliver some other complete features. Without feature branches we had to do all sorts of things, like turn off flags, hiding controls and even rolling back changes.
Teams that embrace CD change how they organise their work. CD is working so that your SW is always releasable, so after every successful commit, it could go into production. You can do that by waiting till each feature is complete, but that slows feedback and results in lower quality, or you can work so that it’s ok to release, even if a feature isn’t complete. This sounds weird, but is the higher-quality approach, according to the State of DevOps Report.
I tend to rebase on top of whatever the main branch is regularly so as to act as if I started my work after the current code base state. This isn't technically that different, but I find it easier to reason about. Essentially I agree with your video that a continuous straight line commit graph is preferable to me. At the same time, small merges are pretty inconsequential and easy to reason about. People do get quite heated about this so I leave it to them to deal with the worst merge nightmares while I commit on top of the latest when I'm done.
CD, i'm new to programming and I just wonder, you often say you expect these CI branches to last rought around 15-20 minutes, or a day a most. So my question is: What if the feature I'm refactoring or working on is incomplete? like say we're making a video game, and I decide that the jump mechanic needs work - just doesn't feel good to jump in this game. Well 20 minutes into doing it over, i realize I need to spend some hours researching the subject and how other games did it. Do I commit, or possibly spend a day or two being thorough in my current approach to the jump mechanic? meanwhile, other devs are building level design choices based on the jump mechanic that depends on it being what they thought it was.
Generally agree with this precept to integrate small bits of code frequently. The only exception I can think of has to do with cybersecurity or life-safety system requirements that disallow dead code (i.e., can’t get there except in an unplanned error state) under all circumstances due to the risk of accidentally entering those code segments. In less critical codebases, I can easily agree that feature toggles (and similar control measures) are adequate and should be used instead of feature branches. If you follow a stratified development approach (e.g., Strata from Vitech for systems engineering), then you can, for example, stub the feature and integrate the stubs with the trunk, then add functionality as the feature is developed.
Literally a day ago was talking with a friend of mine who is a QA lead at a big IT. He complained that it's a nightmare to deal with 10 teams pushing changes into the develop branch. He said that it is unmanageable because it's almost impossible to blame what feature, what change, and who introduced a bug. They spent hours just to identify this info. They do CI, but everyone understands and complains for that kind of workload branching model would most likely be much more manageable for QA.
What about things like ROS where you have different distributions that behave differently? Typically, some packages will have different versions on different branches depending on your distro.
I've only rarely run into major conflicts when merging. Even when we had several people working on major additions to the same product. Most of the time branches are used to make changes or experiment without breaking the main branch builds. And since there are many things that we are unable to build and run locally code has to be committed to put it through the build, test and deploy pipeline. We could certainly investigate making it possible for each person to do that locally, but since branching is actually working it would be hard to justify the time and expense. Personally I'm not a fan of pair programming. We certainly use it on occasion, but I'd burn out in months if I had to work with someone looking over my shoulder all day. I've turned down lucrative job offers because the job would have involved working like that all day every day.
One thing confuses me with the “no branch” concept: in order to have a code review prior to merging, we *do need* a branch other than master (or whatever trunk) to be able to create a pull request, right? Or how else a code review would work if everyone is committing to the same branch?
While you could do "trunk based development" (commit and push directly), CI does not necessarily mean no branches. You still might want code review and other QA gates in place.
I have also seen "short-lived" branches basically for code-review and such. So you do all your TBD in this "short lived" branch and merge after code-review/checks
Well, that's the thing. Often, to do trunk-based development as effectively as possible, you don't do a pull request based code review workflow - instead favouring continuous review via pairing or mobbing. That's the ideal. However: there's a difference between a pull request branch and a feature branch, and it's mostly a question of scale. If said branch only lasts for hours, then sure, that's continuous integration. Directly pushing to trunk is just the taking that approach to the max.
I don't do branches. Everything on main. Git tells me when I haven't updated when I try to push. I don't see the point in PRs either. Whenever I see a PR I lack the context around the decisions, I let the pipeline feedback in the form of testing and static scans etc to ensure quality is brought in up front.
Yeah, I don't see how it makes sense at the end of the day without some branching. My understanding about what he's pushing for is just more frequent pull requests
I think there is more to it than thinking ourselves as heroic loaners 15:01, one of the ways we breakdown complex problems and also with the code we write is the idea of breaking down a complex problem into specific concerns where we can think solely about each part in as much isolation as possible be it steps in a process or the responsibilities of an object. Isolate a single part of the greater problem at a time. Therefore, merge conflicts with someone else's code means we need to context switch to understand a potential different problem context in order to resolve the merge issue. Now in my experience its the guy who commits later that is responsible for resolving the issue (is this the unresolved special source that's nagging at the back of my head), and hence has to do this extra work of understanding an additional problem domain/context. Resolving who has to context switch stop the breakdown of their own feature and understand how their own problem context interacts with someone else's current work in the same problem domain ... should this be left to who commits second is seen as responsible for the merge ... the associated work is delegated on nothing more than who can smash that commit button before the other? It also means this is the point at which people's different coding styles interact, has the code been refactored or is still in a state of just putting together something to pass unit tests. I'm not saying leaving people in isolation till their own feature is complete before attempting merge with another complete feature, meaning there is a potential for a large divergence of code. I just think there is a social/phycological dimension to ... THE MERGE!!! I've found myself merging code ... before commit, "oh wait ... it's changed again" ... ok spend the time understanding the merge ... "ok right think that's my changes merged ... better do a pull form the remote" ... "oh jeez again some more small changes made before I was able to commit". This is doing my head in if only I had just been able to commit before I had to pull my hair out trying to understand this stuff written by someone else. Maybe it creates a feeling of winners and losers, and it's the speed at which you commit that is being rewarded not the quality of the code. Maybe I'm just crap and slow at merging peoples code, maybe needs ideas on "Pair Merging?", but does anyone else have experience of he who commits last is the loser??
Never really had that problem myself. But then I have always been vocal when I see conflicts and hero-culture and try to break out of it as soon as I see the signs. There is a saying in the DevOps world - "a bug, is a bug, is a bug". Meaning that bad code, bad process, bad security, bad compliance, low trust, bad logging, metrics and audit are all bugs. But we tend to pretend they are someone else's problem because "I'm just trying to get my task done". As long as everyone has the mindset that it's a contest, that developers shall sit alone in their corners with their headphones on, typing code all day. Instead of working as a team (across roles), have good practices, a good modular design (so that there is less chance of stepping on each other), working in pairs and sometimes entire groups (see mob programming) we tend to default to defensive behaviour and then there will be friction, frustration and no joy. And then we try to smooth over the cultural issues with processes, Gitflow and feature branches and silos and handoffs with 27 new roles. A team should be measured based on result and outcomes - not individual contributions. My point? Any mindset or culture that focuses on individual "performance" (stack ranking, ego, loudest-guy-opinion, "winners-and-losers") and cop-outs like "but I'm an introvert" is A BUG! The issue with seeing ourselves as heroic loners is indeed the biggest issue in SWE. Since in your scenario both of you apparently were working in related parts of the codebase - I think you should have taken a step back, sat down with each other, talk, plan, pair-program and collaborate on both of the changes instead if sitting there frustrated that "the other guy screwed up my changes again and now I'm getting behind and losing the scrum story-point race". Maybe I'm taking this over the top, but only because I've seen it happen to other teams so many times. Learned helplessness is the worst place to be, because it absolutely drains all energy and joy of going to work and life in general.
@@ddanielsandberg I find it a bit scary when normal human phycology gets referred to as “A BUG”, I think everyone has different personality traits, strengths and weaknesses, and in different situations weaknesses become strengths, in some situations being an introvert that isolates themselves to push through a single focused idea is what’s needed, but then there’s other situations where it’s collaboration that’s going to be the enabler. That doesn’t mean I haven’t been pissed off and got people kicked off a team … but I find it most stimulating and sometimes most frustrating to work in a diverse team, argh and though it pains me to say so, good management of people counts.
@@derekcarter4157 I didn't mean to imply that people are bugs. :( I meant it more like the system is buggy if it enables and incentivises certain negative behaviours and discourages other positive behaviours. The software industry has spent the past 40 years hiring and attracting certain personality types and I think it has had a detrimental effect. I know how it feels some days, I'm thinking of leaving the software industry and become a carpenter or painter or something because this has given me CPPS. “A bad system will beat a good person every time.”
@@ddanielsandberg aye this covid lockdown from last year seems to have been terrible for my eyes just being in the house never looking more than 7 feet away (or maybe I’m just getting old). But yes it’s a software dev is a creative endeavour, even if all our best ideas are stolen! Processes and techniques are to help people create and communicate (and I’m the case of SAFe and Scrum make a quick buck) teeheehee.
Hello, I get the idea and I don't disagree. I think we'd need some detailed examples though, about how to do it. The rationale seems as correct as it is high level. The practical level is, I believe, what keeps many doubtful.
I would really like to hear about how folks do code review in trunk based development. Years ago I worked for a place that did trunk based dev without code review. Integration worked well and the test suite was solid but the design of the code sucked because there wasn't a system for reviewing code design and architecture. I am wondering if it is possible to change workflow to accommodate this without having to have every change reviewed? I wonder if docs around feature architecture committed to source control coupled with well planned conversations would make sense? Anyone tried something like that?
A) We are pretty much stuck with feature branches on an organizational level - we are trying to keep our branches small and frequent though (still not perfect, but we are improving), in the end, communication is key here whether on the individual feature-branch level or when talking about the "complete" feature and the plans for it B) I fear CI is too straightforward (for a lack of a better word) - you either need really good pairs programming together to avoid silly and/ or hard to find bugs creeping in, especially as less capable developers are just as capable of breaking tests as any other part of the code - you miss that part of separation that a pull request and review gives you; this is personal, but I can better focus on reviewing changes and trying to figure out what they affect when I go into code review for a (small[ish]) PR. This is also true of my own PRs. This, I think, is especially true when your code deals with multiple concepts that you need to mentally context switch from. I'm fully willing to accept that this might be my prejudice talking, since I haven't really been involved in a real CI project thus far. edit: I do adore your videos though, including this one
Branching with small FF trunk based development just works. Especially if you want to do different types of testing that is not similar to your main app (i.e. different frontend with a ephemeral env). This is something that doesn't have to even be defended. It's not either or, we can take parts of what your saying and branch as well. Branching is not guess work though and having everything in 1 branch stops dev testing and exploration. Testing one small change to code is testing more than just that code itself. You have to also test how it interacts with the env you have set up and I don't see how that would work with only 1 branch.
We have builds and test (functional and unit) runs on every commit of our branch. We do rebasing instead of merging of main develop branch into ours. We strive to make multiple smaller MR/PRs but they always have to be a finished working bit. Since codebase requires quite a lot of refactoring so that C++ api guarantees code correctness far more than we are able to achieve in Java version of the product even though Java version is younger... Which is really important for multi threading purposes (when using shared data). But the most important part are code reviews (no tests are not replacement for code reviews - and I'm talking about to some extent almost formal code reviews - they are additional help and documentation). Based on this and the fact that branch is developed for a week and then gets stuck in code review for about 14 days (including changes to the code) I would really like to see how CI and constant pushing of code would produce better code. Faster written for short term gains? Yes. Better... Please explain how as I can't see it work for anything more than service development where partial uptime is more important than stability of the whole.
Hi, I love your videos! Can you please share of what data you are talking about that shows that you build software faster and better using CI/CD? Are there any scientific studies about that?
Just understanding the issue should make it obvious why it's better to integrate code early than having developers hide code from each other and do giant merges after the code branches have diverged. If it's not obvious, I suggest watching the video again, with thought.
@@rosstempo3765 sorry, but that statement seems to miss the point. Of course it is better not to have to deal with this specific problem. But to say that it makes for better software and faster at that is a big stretch without any data, is it not? At least for someone with a different opinion, and those are the ones I have to convince 🙃
@@Feyon You can't convince other people. I've learned this the hard way. You can only provide them with information and they choose whether to take it in or not. If they are not willing to watch a RUclips video explaining the concept and make the conclusions themselves, any kind of research paper won't convince them either. A person who isn't willing to learn, will not learn no matter how much "convincing" you try to do.
What about the argument of atomic commits? Should every push to trunk be guarded by tests? If so, wouldn’t the scope of tests dictate the frequency of integration? Is it therefore better to base integration on a set time frame, or on set scopes? How would you determine scope?
Yes, every commit should be covered by tests. Yes that means that a very effective approach is to allow tests to dictate the commit frequency. If you practice TDD, as I do, that means that my commit frequency is usually roughly once every 10 or 15 minutes. That is pretty close to "continuous" in a practical sense.
Great video Dave. I heard you first on the Changelog podcast, then looked you up here. You've spoken to my years of managing feature branched projects and suffering the inevitable conflicts - even in the very small teams I worked with. It's spot on. I'm running the rebuild of a Drupal project, which will initially be built without any automated tests and all in feature branches. In the very long run I'd like to move towards CI and TDD. One of many questions I have though: Our development so far has gone like this: A developer works in branch, I review the PR and test functionality and pass it over to another non-dev stakeholder who also reviews it against the spec and tests it on a staging set up. In CI, do non-dev stakeholders have to do their testing and reviewing directly on production? In our experience this non-dev stakeholder often catches mistakes where the spec was misunderstood, and we feel strongly that these mistakes should never make it into production. Sorry if that's been addressed already! Thanks for the videos.
My preference is to try an in-line this kind of overview, so that it isn't a "gate" in the development/release process, but is part of the development. PRs and reviews tend to slow the process, so the idea is to do these things continually alongside the development. Instead of performing a code review at the time of a PR, pair-programming gives you a continuous review as well as lots of other benefits. Instead of non-dev stakeholders looking at the code catching mistakes after the code is thought to be finished, build automated testing into the development process so that catches most bugs. Where a review is meant to help sanity check that the change looks as expected, rather than acting as a regression test, then do this early in the dev process as soon as you can. Overall the aim is to build quality into the code from the outset, rather than try to "inspect quality in" to the code later. Thanks for the question, I think I may do a video on QA and it's role in CD teams that may explain some of this stuff.
@@ContinuousDelivery Thanks for the reply! Yes, I could review the code over the dev's shoulder prior to commit I guess, but the functionality that was added has to be tested/reviewed within some non-production environment. So yeah, that is a gate in our process. It seems to me that in CI you have no need of a staging environment. You appear to just have local development environments, and production. So if that's true, then our non-dev stakeholder can only review what is built when it's on production, because they can't use the dev's local environment. For us, the key thing that this person brings to quality is ensuring that what is built matches what was intended by the spec. Automated tests written by a dev that has misunderstood the spec won't catch that sort of mistake, it can only be caught by the person who knows the requirements. Love to hear your thoughts on that!
@@dansearle5994 Not really, the deployment pipeline is a controlled environment, outside of local development environment of an individual dev. It is a shared space where changes are integrated and evaluated. Usually in production-like test environments that mimic prod config and scenarios. These are close to a traditional staging env, but not quite the same. Ideally the person that "knows the requirements" is part of the team too, so close enough to the dev to see the soliution evolve and so catch these kinds of misunderstandings before they progress to the (too late) point of be deemed "finished" by the dev team.
@@ContinuousDelivery ok I see. We're used to reviewing the produced work in our own time, on our own machines (accessing the staging environment), so it's a big shift to go from that to review system where we're with the dev more often, looking at what they're working on on their machine (either in person or via screenshare I imagine) in little bits. It's food for thought, I'll see what I can do, over time!
Not mentioning if you are talking about short or long living branches does raise a lot of questions for me. The big four are: 1: If I work in a situation where I can't communicate with the people who work on the same code as I do, do I trust them enough to allow unreviewed code into my main branch? 2: Is there a chance that your commits are as big as my feature branches? (one unit of work for a feature done, between a day or two while daily re-intergrating the main branch.) 3: the whole thing seems to only work if you don't use the de facto standard tools like PR's since they are by design only applicable to branches, therefore we need to get an extra tool to do code reviews? 4: Do I really want to trigger all my pipelines when I push a typo fix just so everybody can integrate early? btw trunkbaseddevelopment dot com acknowledges Short-Lived Feature Branches, what's your opinion on that? Keep up the good content :)
So this topic makes a lot of sense in a lot of ways but one issue I see is with how my studio works. We have new programmers and interns from a local college working on projects often. This means that any one of them could break things on a moments notice with CI just due to discipline and skill being not there yet. On the flip side it would force them to pay attention and work more effectively though at a cost. This isn't a finished thought really and context matters but I felt like voicing it can't hurt, all while I continue to think about it.
I think that is a fair point. My prefered way to introduce people to a team is through pair-programming, that instils the discipline and gives the newby a guide to the codebase, working practices and so on.
Thanks for the vid! It's very refreshing to see motivations for "why we do things like we do" other than: * that's how google does it * that's how good developers do it I appreciate you adding a scientific-like point of view. That being said, the context where CI really matters is almost clear, but not quite, so I'd like to understand: * Does CI matter more for companies where really big teams (or lots of teams) are working on the same code-base? * Does a microservice architecture make CI more relevant or slightly less important?
While all I can say is "I have an experience-based opinion", here's my take on the two questions: 1) It matters for different reasons. In smaller teams, you will have fewer versions of truth. At the same time, you should also have fewer reasons to diverge. In bigger teams, the it's the exponential growth of failure points over time that makes CI pretty much a necessity. 2) There's something interesting about microservices, and that's the decoupling of implementation from API. As long as you're not changing the API, there's no need to withhold any tested change. Indeed, if you can push out the change the minute it's made, you're maximizing the value of your work by minimizing lead time. And one of the core advantages of microservices is that they let you do this, so if you don't - you miss out on a large portion of their value. BUT - when you're continuously changing the API, you need a strategy. Many companies don't think about that, and hence get into a terrible mess when CI'ing microservices.
Interesting idea. I would like to try it in my work, but I cant imagine how the code review process looks like. Also, when someone pushes a broken commit to master, all work stops until it is fixed, what to do with that? I suppose that this approach requires TDD.
I think this is where forks come in handy. There's one main/master repository from which developers fork their own copy of the repo. They can now work on their own master branch without touching the master on the main repo. The developer pushes changes to his/her master and makes a pull/merge request from their master to the original, and this is where code reviews and automated tests can be used; developers changes won't be merged to the original repo until the code review is done and the tests and the build passes.
make sense to me, but I have this question, I have worked in real big companies with hundres of developers around the world, 24/7 coding and adding stuff, using the branching approach, some devs (no me :P) sometimes pushes their broken branch to master, blocking the rest of the team, with CI that will happen too, and even more often is my suposition, Thanks for the video.
in providers like Github and Bitbucket there exists settings and controls to ensure that tests pass after merging. there's also a practice for the team to fix a broken build immediately if it fails for example see GitHub.com -> Administering a repository -> Defining the mergeability of pull requests -> About protected branches
That is biased, in my view. What you call fundamental is not working in 100% real life scenarios. There are cases when it's useful to guarantee changes isolation for the sake of risks mitigation. Think of it as full transaction isolation in DB, like when evential consistency needs to be avoided.
In small projects your approach is doable and make sense but in bigger ones involving many people I don't beleave that it would be more efficient than feature branching.
@@ContinuousDelivery I'm not sure I understood you correctly - do you mean working in a team containing hundreds of people? For sure it was not a SCRUM team then, right? I know that Google they are working mosty in micro-teams (like 2-4 developers in a team) and then as I said I agree that the strategy makes sense.
@@zenonkurcz7215 well it wasn't Scrum, not my favourite approach, but you are right, agile projects are delivered by many small teams, so our teams of hundreds were divided up into smaller, functional teams focused on one part of the problems, but working in a shared codebase, with a single deployment pipeline. Since we were talking about CI and branching, that was the context of my comment. Google hold nearly all of their code, billions of lines, worked on by thousands of people, in a single repo!
I think I can see the point you're making, but I don't think not branching is the way to go if you'd like to be able to always deploy the head of your main branch (master of development). In mature projects there can be huge CI pipelines, which are not desirable to run locally. If this pipeline determines I introduced a performance problem in the code for example, I would not want to have this in my main branch. Instead I'd want to fix it in my own little sandbox (my feature branch) before merging. In companies where I worked, the burden of proof was always with the person trying to merge new code. Thus, you merge the main branch into your feature branch regularly and you fix any problems that are introduced (in your feature branch, not in the main branch) before merging your branch. Besides that, I'd recon this is why we have all those scrum rituals to inspect and adapt.
Recently I have had to branch to test a new version of a third party library that needed a large refactor to integrate. This took around a week. Once complete I merged it in after it was tested and verified. If I just worked in the main branch for that I would have created a window where the software is not releasable. Is that a valid time to do branching? I typically do CI but in that case I did not.
Since you seemed curious about why people have an emotional response to this subject, I thought I'd chime in. What emotionally turns me off about ci is sharing my code before it's finished. Imposter syndrome is rampant in our profession and merging code into a shared branch before the feature is "done" make me anxious. Rationally I agree with you. Shorter cycles lead to better code and seeing everyone's else's "incomplete" might even reduce feeling of inferiority and serve as a learning aid for junior developers. Nevertheless the anxiety remains. Love the videos
4:35 Totally understand this moment. Despite your advice not to fork, if we as a team decide to violate the suggested rule, is there some tool (for example as GitHub bot or VSCode extension) to watch all feature branches and warn two developers that they are going into the "danger zone" and probably need to communicate what is whom scope?
So from what I understand you integrate more frequently? But what if it's going to take some time before your code is in a compilable state? Sometimes we need to make big changes.
Why would it take "some time to get the code into a compilable state"? Just throwing it out there; what would need to change for things to always be in a working state?
I need more information on this. Can anyone point me to something? Let's say you keep your branches short, so you merge your branch as soon as you've abstracted away a certain part of the code you want to write a new implementation for. Now you create a new branch after merging the previous one that was just for refactoring. What do you call both branches? Do you create separate stories so you have unique ids to name the branches with?
You can read more at www.behaviourdriven.org No need for named branches at all. Work locally on master, when you commit also push to origin/master and run CI there. It’s pretty simple really.
I've really enjoyed your channel for a while now. I have picked up far more insightful ideas than I ever thought I would when I began watching. So my first serious comment on your channel is this... I absolutely love the shirt!
If you are dealing with a live system, with live customers who may report bugs that need to be fixed ASAP, maintaining an active "golden master" branch is the easiest, safest and fastest way to deal with it - and that means NOT having a bunch of barely tested or reviewed code go into the mainline. And even for an individual developer, I often use feature branches on my own projects, where I'm the only developer, if I want to experiment with different approaches that I may end up discarding later. I really don't get the hate for feature branches. I work with some trainee developers that need more review than others. I don't WANT their code to be continuously integrated - that would actually cause problems. And as for long-lived feature branches creating a growing tech debt/merge risk - just keep rebasing your side branch on whatever's been accepted into master, and you'll never be that far from a merge. Seems pretty obvious to me. To avoid branches and to just have everyone work in the master branch seems like chaos to me, and putting any customers of a live system at risk of not being able to provide them with timely bugfixes.
People hate feature branches because they don't know how to merge and are not competent devs imho and want to pretend they are some code ninja when they are just code hackers. 100% as you say keep rebasing from master on a regular basis. It is not rocket science to pull to keep up to date.
I agree. It's sad many people after gaining experience will look back on this video and say they got ill advice about not branching. What made Git popular is that rebase and merge are easy because branching is a must.
The problems mentioned in this comment seem to be the product of not enough test automation and not utilizing pair programming. Having junior developers work on features by themself in the dark is the reason why you don't want to integrate their code. Have the seniors pair with them instead of waiting for a PR to review.
Great presentation! Thank you. I've come across some arguments/trade-offs which I've had a hard time answering. Perhaps someone can help? 1. Exploratory Work - someone told me they like to take a piece of code, break it up, and put it back together in different ways. They do this to make sense of the code and to explore different ideas. The ideas they're exploring don't always work out. Hence, they don't want them to be in master. Is this type of work something we still want to continuously integrate? 2. Squashing Commits - some teams want the commit history to follow a specific pattern. CI/TBD changes the pattern in some case - i.e. commits are not neccessirly grouped by feature, fix, etc. Clean up commits are no longer grouped by a branch and therefore squashing those clean-up commits becomes more difficult. I believe the argument here is about making the commit history a bit harder to understand and/or adversely affecting troubleshooting techniques such as git bisect.
1) If you are doing exploratory work and want the freedom to move fast and break things, then that is not "production quality" code. So you should throw it away after you have learned the lesson. No problem with that being done on a branch, but don't EVER merge the branch back to something releasable. Instead, learn whatever you want to learn, then re-write the code with producton-quality, TDD etc. These experiments are called "Spikes" in Extreme Programming, and should always be done to learn something, not with the aim of creating releasable code. 2) I think that this is a mistake, there are other ways to "tell the story" of a change, without compromising quality in order to make the commit history read nicely. I wrote about this here: www.davefarley.net/?p=263
I'm struggling to see what you're getting at at the 4:20 mark and in general I'm struggling to see why you would consider CI and Feature Branching as mutually exclusive concepts. A feature branch only exists for the duration of the development of that feature, once you're finished it gets merged in (integrated if you will...). To make sure a feature branch isn't straying too far away from its parent branch I just rebase or merge. Have I missed something?
Where I work we use feature branches.. one thing that happens from time to time is to remove a particular feature branch from a release candidate... good luck doing that if you constantly committing to master.
Well it is easy enough if the design is nice and modular. Version control doesn't fix this problem, because if you need to merge your changes with other peoples, you still have to disentangle them to revert. This is much more about good design than it is about VCS strategy.
@@ContinuousDelivery Appreciate the reply! Find your videos challenging but I am trying to keep an open mind. I'd guess what I am doing according to TDD is an antipattern. I rebase all of my commits in my feature branch into a single or fewer commits instead of filling git history with 100 commits for a single feature. I feel like I'm dong part of TDD as far as starting with the test cases first.. I guess I'm trying to wrap my mind around how to implement TDD fully and thinking of the use cases that I do go thru during the release process. For example merge hell for feature branches that are really old. .that is one major drawback. Thanks for the videos!
For example with feature branches.. if my boss decides for whatever reason to remove a feature from the release.. we can create a RC branch without merging that feature branch.. if everything is always committed to master.. that is going to be a lot of work. If things go really bad.. I can just destroy the RC branch and try again (leaving out any feature branch) from that RC branch. I guess that is my point.. thanks again!
I've asked in one of your previous videos about the process of reviewing changes to a code base with continuous integration. I'm a bit confused about what the workflow would look like. I work with a small team, so cashes with merges are far and few between, but the most important part of our workflow is the review process we get when we ask for merge request. What does it looks like? How do you get that feedback from your teammates? Specially since that feedback happens when you got hands around the changes and it's all fresh in your mind, not 3 weeks from now when I moved on and a teammate happens to look at that part of the code.
Yeah, I am in a similar situation. I am working on a small team with currently 2 other developers. We do use feature branching and when we are done, we issue a pr requesting a review from the others. Even, or perhaps especially, in a bigger team I think reviewing code is important. It also helps you seeing what impact changes from others may have to your own branch and knowing what you will need to adapt. With CI, I feel like you might run into situations where you see a conflict and don't know what the other's changes are for and how to adapt them, whereas a more complete view of there changes may help with that. That being said, it is generally obviously easier to merge tiny commits. I should also mention, while we usually have one, and exactly one, bigger feature in development at all times, the other branches are usually pretty small and never last more than a couple days or maybe a week in some cases, so there are never big discrepancies and when working on the bigger feature, it is feasible to pull from master. We work in feature branches on user stories, not epics, and I feel like this video might talk about the idea of developing epics in a feature branch, which I would agree with is not a good idea and would make the review process quite cumbersome as well.
I also have this question. If we want to be continuously ready to deploy, we need to have some limitation on developers just pushing to master. Even if we do pair programming (which is another big change to the development workflow) just those two developers may not be enough to confirm a change is correct. At my work we are trying shorter lived branches. So still basically feature branches, but putting it up for review/feedback/merge before the whole feature is done. Aiming to have branches live a few days at more. It does take more time for reviews, but I think it is worth it for the more frequent feedback and avoiding conflicts.
@@edwardweir5784 Well, the techniques all work together. If you're doing TDD, you should have some confidence that the code is correct. If you're doing pair programming, the code gets reviewed before it goes in. And if you're doing CI, the code goes into the main branch frequently and you don't get surprised by sudden big changes.
You get feedback by communicating with each other. A feature branch in itself won't review your code for you, and CI won't either. If you want to set up a code review process, you can literally do it however you want with CI. Use the method you want and one that works for your team. It's that simple.
@@markjreed even if you do TDD, that still can mean, that a developer doesn't write a test together with the new feature, so the new code could be incorrect, or the test itself could be incorrect.
I am not sure if in my work we have CI. We do have a dev branch that we commit to it almost everyday and max 3 days, and every couple of weeks we merge to master. We are very small team and merge conflicts rarely happen, and if they do happen, they are very small. I am not sure what to call our setup. I really like the concept of CI and when I talked about it with the team, they told me that they tried it before I joined and didn't work " because we are small team", could it be the case that CI might not the best option for small teams
Since you all work on the dev branch, it is CI. That you only merge with master every now and then, shouldn't be an issue. That might just be your company's way of declaring something "stable" or releasing it. So I'd say it is CI but not necessarily CD. And that's okay.
Thanks, you correctly answered my arguments of pulling every day. I agree CI is the best way and committing to the trunk every day is the best idea. That graphics was useful. Only point is we must have test suite there that covers well.
you cant just pick the worst examples of your opposing viewpoint and the best examples of your own and then say yours is better... sure if you are messy you can lose 12 months of work using feature branches. Another thing: Your method introduces loads of dead code into the production branch. If I look into the production branch, I want to only see code that is active, or about to become active. Not half written code for a feature that will be released in five months. For me personally, a clean codebase is way more important than occasionally having a merge conflict or two.
Not merging for weeks is not called feature branching that's just working on a different project.
not really.
Some things just take very long and even as a CI enthusiast I have to admit that long living branches can have a purpose
What about merging other branches into yours once on a while to test the diversion and revert the merge after that?
@@OggerFN Then your features are too big and needs to be broken down into smaller tasks. A feature branch should only live for a day.
@@PelFox
Some things can't be broken down so you either implement it unfinished and feature toggle it or branch until it's finished.
@@PelFox a day is a bit on the short side but it definitely shouldn't last longer than a sprint
branching is like time travel, it's sometimes helpful but if you stay away for a long time the consequences are unpredictable
🤣🤣🤣👌👌👌👌
I'm interested in what your opinion is on Code Reviewing practices, in light with your strong preference for CI workflow. Some companies I've worked at, have a pre-merge code-review process like Github's PR system that needs a review before being merged into trunk. A pre-merge code-review process or mindset would then only work in an environment that considers a pair-programmed changeset as properly reviewed? Would you consider a pre-merge code-review policy to be an impediment towards incorporating CI?
In companies that have a post-merge code-review process this problem doesn't seem to exist, but one would probably require an adequate level of (non)functional coverage in their automated tests to catch mistakes quickly, such that code-review is only needing to (ideally) focus on design. Are you therefore of the oppinion that a post-merge code-review process (or pairing as mention before) are two key processes to have in place for enabling CI?
This is a topic I see come up frequently, so I hope he does a video about it. If I remember the topic correctly from the Accelerate-book, as much as possible from the pull requests should be automated, leaving stuff like exploratory testing and such to the humans. And for the pull requests that remain, the team should make code reviews a high priority to ensure flow
Same question as I wanted to pose as from my experience the code review part is the one that causes your branch to stay non-merged longer than the design and implementation parts put together and even multiplied.
For post-merge code reviews I don't see how they would work as they require far more experienced developers to go through (specially with continuously pushing unfinished work that makes the same mess as merge instead of rebase does in git... nightmare for reviewing so you just rubber stamp it and be done with it).
With partially complete pushes you also require far more experienced developers as they must understand that something is still work in progress and somehow still envision the final design.
And in my experience experienced and high quality developers are hard to come by so I still can't imagine getting higher quality code from CI compared to shorter tasks, rebasing and serious reviews of the design. Short term faster... Yes. Long term more maintainable... Can't imagine how.
He has made a video in the past about preferring pair programming, and seeing it as valid continuous code review.
@@defeqel6537 I've seen that video and it might be just me but I don't see pair programming as replacement for code reviews (at least not those that smell a bit of formal code reviews) - to be honest I can't even imagine somebody constantly messing up my train of thoughts in that manner...
@@DerDoMeN Yeah, I'm not sure I agree with him on that, but lacking the experience, I couldn't really say.
And Dave Farley said not to branch and there was no branching. And Dave Farley saw no branches, and it was good.
I recently retired after 47 years in the field, slowly trending from coding to writing tools to version control to configuration management to as close as I could get to continuous delivery, which wasn't very close because I could not convince my teams of some key points. I am hugely enjoying your videos and your explanations, and thought I'd share a few observations from my time in the field.
1) I haven't heard you say this actual phrase, but some major luminary very early in the software world's development stated "Nothing of any significance has ever been accomplished by a team of more than 7". I've heard you express much the same idea, though, quite a few times. More people than that exponentially increases the chances of miscommunication about the design of the system.
2) I used to sort of half-heartedly suggest feature branches for extensive, disruptive, dangerous changes, but my teams always resisted it hard for the reasons that you clarify in this and the other videos about "don't feature branch"; essentially, merge hell will invariably result.
3) I worked in about 9 different shops that all claimed to be doing Agile Development. In one of them, they were actually doing Agile Development. In the others, they were indulging in rituals designed to fire off the buzz words managers wanted to hear, but these rituals usually were bumbled so badly they didn't do much good and in some cases were actually harmful, like when the boss was the scrum master without training and did most of the talking.
Once more insightful thinking of how sw can better made.
Thanks Dave Farley.
Since this is a so difficult practice to get the teams and/or Devs to move to, if possible I would love a future video of how to start with it.
Something like what needs to be done from the beginning, and a possible adherence to other techniques, etc.
Thanks in advance.
One of the issues I have around CI vs Feature branching is more related to maturity. As far as I can tell, it works great when you have a stream aligned team working with a code base where most of the work happens in parallel. When you have a monolith, the problem with CI (for me) is that integration events become unpredictable. One team may be testing a set of changes, when another team commits a change that is automatically deployed. All of the tests cases that had been performed at have been invalidated as a result of the change. That's primarily why we've used feature branches in the past, to isolate changes just long enough to test them.
Feature flags partially solve most of those issues. That said, I still prefer short-lived feature branches as I don't like tons of feature flags.
@@michaelslattery3050 Does it not get somewhat tricky to understand the subjective concept of 'short-lived' here? How would you define short-lived? How would you enforce it? I have seen teams that aspire to that and somehow these short-lived branches suddenly become very long-lived.
This 🙌
If you don't think a single developer working alone can benefit from feature-branching, I don't think you and I are talking about the same thing when we say the word "branch", which would certainly explain the discrepancy.
a "branch" collects units of work (commits) together. Each commit is the smallest change which works on its own. Each branch is the smallest change which justifies its own existence.
Within a branch, prior to integration, commits are not a commitment - they are rewritten / re-ordered regularly, as the discovery process continues. Integrating them doesn't make sense, because they aren't final yet. Forcing integration prior to the change being done makes the history unreadable because each change becomes separated from its justification. And of course, the one and only reason we're even using version control (instead of just sending each-other diffs against the last timestamped snapshot) is to keep track of *why* changes happen.
I think we both agree that integration needs to happen very regularly, but disagree strongly about the acceptable scope of both feature branches and commits. I don't want to be "more continuous" than the smallest justifiable unit.
I also disagree that "a branch which can't be integrated" is the same as "lost work", and I would have thought (based on your other video) that you would be a strong advocate against that sort of position. Sometimes code needs to be rewritten based on new information, and when that happens, it's always faster than writing the original code.
It really sounds like you're talking about traditional, long-lived branches, rather than feature branches.
As with so many discussions encountered on the Internet, I'm pretty sure I agree with everything you say, except for the specific terminology.
the changes aren't separated from their justification if you know how to write commit messages.
@@OggerFN their individual justification, yes, but not their overall justification. Not the context of which commits relate to it, in order to define the goal. If your commit is fully justified without any related commits, go ahead and merge that. If whoever is reviewing could ever ask "why?", then it should probably be grouped into a branch.
Meanwhile, if whoever is reviewing could never ask "why?" for any of your individual commits, then your commits are too large.
@@________w
You should write the task the commit is supposed to accomplish AND the reason for that in a commit message.
For example a habit I have been using is naming the ticket a commit is relating to in it's message in order to "group" commits.
@@OggerFN You can also just squash everything into a single commit once you have some sort of logical unit related to your feature.
@@mAcCoLo666
Yes if you care enough to do that
- "When the information is modified in each place, then the long the time, the more those copies will diverge from each other".
Well that is the description of entropy, it is so fascinating to understand that basic properties in universe affect everything even how we commit ...
Yay! Finally I got an excuse for committing directly into master: it's called CI
You must still pass the integration pipeline, though.^^
Love the sarcasm! And also, I agree!
It's perfectly acceptable to use many different strategies and CI and Feature branching aren't mutually exclusive. You can do feature branching in one step of your process and CI in a later stage. No worries.
Honestly, this video is too dogmatic.
As someone pointed out...only if you get through the pipeline. And that should have human beings in it after the automated checks. :-) I mean..i heard about QA Somewhere......
@@timothyblazer1749 I don't think that this pipeline as described in the video includes human beings. (EDIT: You wouldn't be able to integrate that often if QA would look at every integration.)
@@timothyblazer1749 so the branch is still needed for a code review and QA? Wait, that's a feature branch!
I think big take away we need to remember is: “Continuos integration works by limiting the time period and thus limiting the fan out of the cone of changes, this reduces the risk of a clash”. That makes me wonder, if we use feature branches but could somehow guarantee the time period of unchecked code is the same or less as not using feature branches and forbid branching of feature branches to avoid the cone, are feature branches still bad?
That's precisely what you should do. In fact, this approach has many added benefits, such as you can increase your confidence, that no bad code will be pushed to trunk. With trunk based development, nobody can prevent a developer to accidentally push a commit that fails to build or breaks the project completely.
Depends on how would you guarantee that. If you have only 1 "big" feature branch and everything else goes to master, then you can actually keep the system integrated just fine (or systems to be precise - one on master, and one in the feature branch) just by merging master to that feature branch regularly.
This has the downside of someone making a change in master that breaks you feature branch, but since that change is small, you can easily fix it in your feature branch after you pull that change from master.
But what if you have 2 feature branches that take long to finish? If you merge them between each other (as well as pulling from master from both), then you have still all systems integrated AND keep the advantage of neighter branch "polluting" master until it is finished. But now you cannot release one branch without the other.
One solution is to only merge those 2 branches "one way" (say, only merge A to B), but then B cannot be released without also releasing A. Also this strategy doesn't work for multiple unrelated branches.
So you are left with 3 options:
1) don't work on the same code: when a big branch is ongoing, do not change the same codebase / features as this branch is changing. That way, you will avoid conflicts when merging it.
2) bite the bullet: just give up and prepare to do a massive merge once in a while
3) make smaller changes: by integrated smaller changes, you reduce improve in both above points - you are less likely to "hit" the same code as another change if those changes are small, AND when you do, the conflict is going to be small too.
Of course, making smaller changes isn't as easy when you are not used to it and requires a complete paradigm shift in how you look at programming, especially if you want to go "i will work on this feature for 3 weeks and rebase my commit as I want and it only has to work at the end" to "I make a change and push it as working commit every 10 minutes", it will not happen over night.
Hmm, I'm still not 100% sold... Sometimes CI sounds like a really great idea, but the problem also is what do I do when I approach larger refactorings (like updating dependencies to a new major with breaking changes) that take time and will leave the project in an unworkable state until completely resolved or changes that we are not sure if we should carry them through... How do really revert a series of commits in a CI world, where they are mixed in with other people's stuff.
That's a really good point. I think CI as described is good for actual user-facing feature development, but you need a more nuanced approach when doing things like what you're describing. The platform upgrade is the obvious exception, but there are other types of changes that require a careful decision on a case-by-case basis. "Changes that we are not sure if we should carry them through" is an interesting one.
Say for example you want to unify how your application does caching, and you need to audit every single network call it makes to update it to the new system. If you take the time to update each one before sending it out for review, and then (hopefully) merging, what's the likelihood of running into some catastrophic merge conflict? Seems pretty high. Maybe a smaller proof of concept would be more helpful then.
But then say your team does dozens of such experiments, starting on them, but never actually following through to implementing them project-wide? Each experiment you run creates ambiguity in the codebase, where it's no longer clear just from looking at the code what the right thing to do is. In this case, maybe it really would be better to make a long-term branch, knowing full well that lots of code will have to be essentially rewritten multiple times over the course of the project. The new pattern isn't the deliverable in that case, but rather the consistency.
I guess I'm not seeing easy answers either way.
I actually think continuous integration solves this far better than feature branches.
If you wait for your code to rot into a state where a "large all-encompassing refactoring" is needed, you've already failed. Clean up your code before committing it and you won't have this problem.
@@rosstempo3765 great if you have that choice. If you're brought in on a legacy code base, you need other tools at your disposal besides time travel.
@@froobly agree with Ross. even with legacy code, the first step is to write tests - both at a "unit" level and black box level (won't go into details). Make the constraints of the behavior of the code explicit, refactoring can definitely be done step by step in incremental and relatively risky way. if you are dealing with huge migration like breaking framework changes, then I would agree CI doesn't really apply, because it is a different problem. but you still cannot get away with automated tests.
3:20 I often feature branch when I'm alone. That gives me the benefit of switching features when I get stuck and I can come back later. You may argue that I should comment the changes out or implement a software switch when rewriting a function. That can indeed be a solution and I admit that I haven't tried it yet.
When working alone, I prefer to push every change immediately to Production, potentially toggled if it's still work in progress that users shouldn't see. The only way I can know whether my feature works as intended is to get real usage on it.
Of course people's philosophies differ.
I think the context of what kind of team and what kind of product you are working with will affect whether this is effective.
When you have many disparate repositories of code, each with few changes over time, it makes more sense to use pull requests, since things move so lethargically.
But if you spend most of your time in one repository, CI seems like a much better option. For example, you won't have to re-synchronize your feature branches to ensure you're on latest if you are always working off trunk.
Interesting. We used feature branches at first - it is hard. Now we embraced, trunk based development and release branches. Release branches allow us to keep CI intact and enables patching. For new features, finished or not - we use feature toggles. Feature toggles allow several things: review partially implemented UX, understand complexity of database migration requirements. We also starting to embrace side-by-side approach vCurrent-by-vNext. You may say it is a lot of maintenance, on the other hand you are doing it anyways. Now we have much less feature branches, always up CI/CD, easier database CI/CD.
Love the videos.
Would like to see you touch code reviews topic. Often it is seen as one person offers ideas and knowledge sharing on optimizations or better code structure. But I think that it is more important to review how people handle exceptions, how people ensure that APIs are complete. Not all devs are experienced. I mean code reviews are not about beauty, it is about cleaning sh** left by others. This is more realistic and gives more value. Code reviews is about getting more experienced, not just about learning more patterns or algorithms.
Did you completely get rid of branches and started to push directly to trunk?
How to use trunk-based development in a legacy system where there is huge technical debt and most of the code doesn't have good tests we can trust? How to ensure quality where each commit to the trunk can potentially break the system.
Of course, we try to write more tests and do refactoring doing some new feature, but writing tests is not always possible when everything is coupled together.
Unfortunately you're in a difficult situation so you have to make moves to reduce the number of branches... Of course rewriting is one way of doing that, as nuclear as that may seem. There is no theoretical limit to how hard to maintain a system can become, but at some practical level it can become unmaintainable. All the best
Hey who doesn't like the smell of a merge conflict in the morning standup meeting? 😜🤔
Clearly not all the *--ours* fans out there. Excellent coworkers, the lot of them.
If the conflict is only coming up in a meeting, you don't have good communications or process. Those should be being resolved individually by the actors involved, for the most part.
Just wait until a "certified master" that doesn't know anything about development starts hassling you about why "can't you just merge it in".... and to add insult to injury said person usually doesn't even posess enough of an understanding of the concepts involved to be able to understand a straightwordward explanation... whilst simultaneously they're looking down on you for being a "boffin".
@@dafyddrees2287 what you describe is some toxic work environment. In a healthy functional team this is not a concern at all. I wouldn't work in a team that would act the way you describe.
@@vyli1 "healthy functional team this is not a concern at all." That's a very subjective thing to say. All the TBD I've worked on has been awesome and all the branch-tastic stuff has been awful. It corroborates what Dave has been saying in his books and videos.
And that is why I enjoy the whole microservice hype in web development. It allows you to work in a small team. This way, you can have feature branches without worrying too much, given the tasks are also correctly planned in a way where the conflicts chance is very low. For such teams I find it very easy and enjoyable to have even bigger PRs, as they contain the full feature and you can observe all of the changes at once, giving you the opportunity to find more optimization routes.
I think feature branching within a single repository can work, as they are visible to the team, but you need to treat each branch sort of like it's a trunk and regularly merge branches into other branches, not just mainline trunk. The advantage of this is if you do get some nasty merge conflicts you can put them off, let those branches diverge a little while you consult with your team about how to resolve them and prevent them in the future, all while continuing to make forward progress. Some of that progress may end up being discarded, but some of it won't, so overall you aren't completely halted by having to deal with the divergence.
It would be great to have a clear explanation of the hypothesis. It's not clear what feature branching is. I believe that CI is not alien to use your own branch to work on your changes and then allow peers review. I understand that short time feature branches could be used. It's not clear to me if those are not even accepted in CI.
Added costs of feature toggles is not always worth it. It takes a lot of upfront work to design and plan work on features so that they wont clash with each other. If not managed carefully, it can end up in really messed up spaghetti code. Especially in early stages of projects, when no one knows what they are really doing.
Not only that, there are data model changes that simply cannot live together with the legacy model. What kind of toggle allows you to have a table column value unique in a place of the code and allows multiple values in other? Answer: none that realistically enforces the constraint. I’like to hear how his version of CI deals with that.
Day one: Checkout a new feature branch from develop.
Day two: Merge said feature branch into develop. Rebase first if needed (if there are any conflicts).
Day three: Merge develop into prod or staging or whatever deployment pipeline you have set up.
Day four: Repeat.
Feature branching and CI aren't mutually exclusive. You can do both. And I think you should.
Keeping features small is just a sane thing to do and has nothing to do with CI or feature branching.
If you can't keep the feature branch small, you should rebase it continuously (against the develop branch) until it's ready to merge.
So, do feature branches against the develop-branch and CI/CD between develop and master with whatever steps in between you want.
Ofc there's also other ways of doing it. Don't be dogmatic. Be flexible and adapt to needs and the preferences of your team.
I like this approach, but I would say you can also automate "merge dev into master":
Day 1: Checkout a new feature branch from dev.
Day 2.0: Merge said feature branch into dev. [you don't really need rebase, you can fix merging conflicts without it]
Day 2.1: Automatically run tests using the lastest dev branch.
(optional) Day 2.2: If all tests pass, automatically merge onto master (CI) (or merge manually)
(after merging to master) Day 2.3: Run automated tests using master branch
And of course, 2.2 could be done manually, allowing lets say a PR request or a code review to be made before it is merged.
As far as I understand, this is essentially the same thing as trunk based development. You could skip the "check out a new feature branch" step and get the same result.
Dave, I agree with what you say on so many levels. I am one of the people commenting on the other video. I would like to also refine my view. Please keep in mind, what I don't say explicitly is because I think we see things the same way.
Let's recap what was the main issue in the other video's comments: the beginners of the team. Beginners are a healthy factor in a team for other reasons than the quality of the code.
Technically, we tackle beginners by reviewing their code. Another reason to review the code is compliance. Either way, beginners will make mistakes that cannot be caught automatically, no matter how good your tests are.
Let's add one more to the list of things to keep in mind: we fully agree that things should be integrated as often as possible.
What we don't agree on is doing it at every commit.
This is literally insane, for the reasons I hinted above.
What we do instead is redefine the notion of a "feature". It's not an user story, but a subtask.
Let's take an example. The junior on my team has been working for almost two months on a user story which ended up having about 30 subtasks. So on average, he finished (and integrated) one feature every 1-2 days. The reality is, some features (subtasks) were easy even for him, so some features were half a day, while a few took him as long as a week.
I am fine with this, because we reduced the risk of breaking master, while also reducing the waste of effort (it was the beginner tinkering, so 1 week does not cost that much).
Keep in mind, most of the time we were integrating every 1-2 days.
We can also integrate all combinations of feature branches automatically and test them because since they're short-lived, there's less of them! We remove the branches after integration.
So roughly we have only as many branches as the number of people in the team.
Integrating at every commit would work only in a team of only seniors.
Not so. I have operated this way with junior devs, even raw beginners, many times, but you do it in combination with pair-programming. It is the fastest way to get juniors up to speed and productive, that I have seen.
@@ContinuousDelivery pair programming for 2 months continuously is not productive. They need to chew over problems themselves, sleep over them, then come back the next day and solve them.
Yes, we paired up, and in addition to that he asks for help every 2-3 hours. I found this works best, in addition to pair programming.
Also, the other big issue raised by me and others is code review, which you have not addressed so far.
What is it then, reviewing each commit?
Can you please further discuss the application of this in the context of open source software. Thanks
Yes, like with pull requests (aka "feature branches") and code review in general
With open source software, you do three merge/pull requests. If and when they are successful, these qualify you to be a developer. As a member of the development team, continuous integration is the way to go.
Note: The size of the project does not matter. Whether we are talking about that upcoming JavaScript framework or the Linux kernel, three contributions are enough, roughly.
Continuous Integration is a practice that requires a high trust environment-where you can trust the people on your team. In low-trust environments, like Open Source development, Pull Requests shine. GitHub was largely founded on Open Source, so it makes sense that Pull Requests were largely invented by GitHub. CI would only be appropriate for a small core team in Open Source.
@@djason_music CI can, and should be used in both cases.
Look at the Apache Maven project and how they manage all of their stuff, the same way I do, a single branch. Period.
No need for a release branch either, to cut a release, tag it, build a release from the tag, and move on.
Mainline development is what I recommend for all projects, as I’m sick of cleaning up left over branches from others.
Git and Jira feature branches just makes things worse, as it tends to make the processes we follow fit the tools we use. The tools should suit the way we want to work, not the other way around.
The number of times I’ve had to say “Son, just because you can do a thing, doesn’t meant you should do a thing”. Especially around Git branching models.
The only time I use a release branch is when I have really long life releases (and I’m thinking 1-2 years here) and I have non backwards compatible breaking changes in happening. A good example of this would be a change of underlying middleware or java version for instance. Then each release branch effectively becomes its own master/trunk, and master fades into obscurity. I probably just triggered some with that last bit. But from first principles, it’s the easiest way to work - especially when the devs are inexperienced.
At the company I work for, we implement CI/CD, but instead of master, we work on the UAT branch. Then once a week, we do a pull request to merge UAT into master. That way our production environment only gets clean and tested code, and hasn't crashed due to broken code in over a year now.
I'm curious about when/if Product testing fits in CI. Say that a developer (alone or working as part of a pair) is working on a feature, which was specified by a Product owner of sorts. There's always a chance of some level of misunderstanding between what was asked and what's being delivered. So it's now about whether the code works or not, but more about whether it does the right thing. In that case, being able to deploy a version of the code to a test environment that can be exercised by a Product person is quite useful. However, if we're always committing to trunk and the code is doing the wrong thing, it might already be too late.
I can imagine one solution for it being feature flags for every single customer facing feature. While I can see that working, it can clearly add a ton of incidental complexity to the act of adding features and changing a product.
What's your recommendation for that kind of scenario?
Thanks!
In general I'd agree, but in some cases it still seems useful to branch to me. Especially when doing some more experimental stuff, where you don't know, if you even want to add this change.
But if you already know, you're going to merge this branch/feature sooner or later, it's always better to merge it now.
It's much worse, when others already depend on a feature, which often changes, than when you are not able to merge this feature anymore.
And you will always be able to merge your feature in most cases.
It seems to me that you need to watch the first video and use one of the techniques for deploying without releasing. What I hear when you say "feature branching is useful" is "I'm not good at deploy-without-release". I think you should get better at that and see if the tools, techniques, and paradigms you develop change your mind about feature branches over time. If you always fall back to feature branches, you'll never get good.
What about code reviews before closing an issue/feature?
Nope - that's not continuously integrated is it? The risk is in the merges - and not understanding what other people are changing. If you wait until they've finished "their" feature (big problem already there with code ownership) then you're merging great big lumps of changes - which is asking for trouble.
@@dafyddrees2287 that’s why you write automated tests
@@joek4563 Great - now you have a broken build with tons of shit to track down. It will be much more difficult to sort out because you will be reconciling your finished work against the latest. Merges of huge amounts of changes suck absolute balls to sort out. It's much easier if you have to reconcile your tiny change against somebody else's tiny change.
@@joek4563 I've been doing this for almost twenty years now and the difference between trunk-based dev and this gitflow bullshit is massive. You just can't do big system-level refactorings at all if you're going to have lots of branches. You'd either break everybody else or never be able to work fast enough to deal with all the conflicts you'd get all the time. The only option is to work WITH other people - not to temporarily hide away from them.
@@dafyddrees2287 So code reviews are incompatible with CI?
I don't think your wrong, it's just that the software industry case been convinced that a feature is the smallest change worth integrating. This allows for all unit tests, functional tests, etc to be done over several commits that then mark this feature complete. We more use feature branches as short term backups then branches. I think where you would make this all one commit and the push it to the trunk, is really equivalent to a squash commit on the feature branch before merging. Also GitHub better supports the feature branch model with PRs, and other expectations of that system. Where would the review go in your CI model?
Thank you for the video. How would you go about prototyping different solutions whilst working on the same branch? It seems to me that rollback might be a really complicated in this case.
The more often you commit, and therefore the smaller your commits, the simpler the rollback! However, for experimental things treat them as something different. Do them on a branch, but then throw the branch away (or at least ignore the code in it) and re-write the code from scratch to production quality once you have learned the lesson. This approach is sometimes called "creating a spike". A spike is an exploration of some idea, the result is the learning, not the code. That gives you the freedom to write crap code if it helps you learn faster.
My expierienced rule of thumb: smaller skilled good teamworking may start & come quit far with a strong disciplined checkin strategy directly on *the main* trunk/master branch, but those conditions tends to be not realiable with much more growing/larger projects teams especially including unskilled/newbies.
Fitting VCS checkin & branch strategies depend of multiple aspects:
a) project/task change-sizes & risks/fail costs (e.g.: not every deep re-design/re-strucure checked in as lots of tiny checkins but project-wide atmoic commit/submit),
b) developers team's size, skills and discipline/motivation,
c) surrounding IT & Development Infrastructure including build tool chain support, dedicated DevOps, Q&A and a good project (not directory) scaled merge/integration tool support (bare git is not enough)!
What about changes that can't be completed in one commit and also can't really use tactics like Dark launching.
In one of my pervious projects we had to make significant changes to multiple connected db-tables. I don't see how I can get around using a branch here. Yes the other developers will need to merke, but else we have broken code in the Main branch.
There may be strategies to work around that. For example, you could maintain more than one database table (or column) to represent the same data in different ways, and drop the older version when nobody uses it anymore.
@@cristianpallares7565 I guess that would be a good solution. I just fear the amount of copied code and the work to make sure everything gets deleted in the end. Well... those big changes should not happen to often anyway.
@@christophlutticke7150 I think this is only one way, you need to have that copy, cause I think yo need to have no downtime for you product during the deployment.
I’m trying to promote the term “reluctant integration” (RI) for those techniques that involve maintaining lots of branches...
I like to call it
'severe heap integration technique'
as a mnemonic acronym. :)
I've been using "continuous disintegration"
wishful thinking integration procrastination - trying to reduce risk by delaying integration rather than meeting them head on
I am working in a big software company. 6 years ago we had big problems with pushing features from the different teams back to master because of the changes by other teams as you explained.
Now nothing changed except two things:
1) We must push any change as soon as possible back to master. Even if the feature is not completed if a pice of code, e.g. on function, is completed so it can be pushed to master without coder breaking it should be pushed.
2) Daily build
We do not have any integration problems anymore. Because the discipline of the developers increased because they do not want to break a daily build code quality and efficiency increased. And the "old" developer who knows the "old times" saying that work is now easier.
Not to be confused by not branching based on user stories. It is still helpfull to create isolated environment, a.k.a. branch, for user stories which takes couple of days only to implement and integrate. It is the smallest agile/scrum unit of work that validates a branch. It is more easy to discard a branch than to go back in history in case something goes wrong and developer has to start implement the functionality from scratch.
Issue is, as pointed out in a video, a feature branch which takes several weeks until it gets tested and merged. Just my 2cents of course.
Look at the Apache Maven project and how they manage all of their stuff, the same way I do, a single branch. Period.
No need for a release branch either, to cut a release, tag it, build a release from the tag, and move on.
Mainline development is what I recommend for all projects, as I’m sick of cleaning up left over branches from others.
Git and Jira feature branches just makes things worse, as it tends to make the processes we follow fit the tools we use. The tools should suit the way we want to work, not the other way around.
The number of times I’ve had to say “Son, just because you can do a thing, doesn’t meant you should do a thing”. Especially around Git branching models.
The only time I use a release branch is when I have really long life releases (and I’m thinking 1-2 years here) and I have non backwards compatible breaking changes in happening. A good example of this would be a change of underlying middleware or java version for instance. Then each release branch effectively becomes its own master/trunk, and master fades into obscurity. I probably just triggered some with that last bit. But from first principles, it’s the easiest way to work - especially when the devs are inexperienced.
My favorite story concerning this topic was having a dev lead who had no concept of how to merge using git. The dev lead was responsible for insuring the integration branch was kept in order. Additionally the dev lead had to coordinate this for multiple pods. The majority of our pod was able to commit and merge to the integration branch on our own. The dev lead insisted on doing the merge for the more junior devs. I noticed that one feature branch worked on by a junior dev was spanning multiple sprints. I made sure to let the dev lead know that having that branch that long was going to be trouble. I was ignored and you can all guess what happened...The dev lead overwrote a lot of changes from the others. To end the story, that dev lead no longer works at the company.
Working in isolation in a feature branch keeps your unfinished code from being merged into the main branch until its completely done.
If you check in code constantly to main, then you will have unfinished code or one big push with all your local commits. Keeping commits local without pushing is a problem because they can get lost if your machine crashes.
You should always keep your feature branch in sync with the other developers by locally merging the main branch into your feature branch, atleast once a day.
Though these features branches should also be very small. I'd would say that they should only live for maximum a day before it's being merged with main. Otherwise your features are too big and you need to break it down further.
I'll only disagree that a feature that takes more than one day is too big. I know from experience that some things just don't break down that small and the effort to force everything into smaller pieces invariably takes longer than just implementing the bigger feature would have.
This is an argument against distributed version control, not just branching. With DVCS, even if no one branches, separately developed code isn’t integrated until it’s all pushed to a shared repository. Merely committing won’t combine anything.
So the team I'm working with now has three branches: master, validation, and integration. If I'm introducing a 'feature': I make a personal branch, complete the 'feature', and submit a PR to merge to integration branch. I'm putting 'feature' in quotes because I'm referring to any changes in the code base. We generally limit our changes to take no more than one week of work.
Twice a day, integration is merged to a new validation branch (ie. validation-2021-05-27), which runs an enormous library of validation tests. If any of those tests fail, the culprit PR will be found and either fixed or reverted before merging to master. These fixes / reverts will then be merged back into integration. If your PR was reverted, you simply pull integration to a personal branch, revert the revert-commit, fix the issue and restart the process.
Now my question is: is this CI or feature branching? There are separate branches for features, but each branch only consists of a single person doing a maximum of one week of work. Does feature branching turn into CI when the amount of people on a branch, and the lifespan of a branch, are limited? For instance what if we implemented the same strategy, but 'features' were split into daily deliverable chunks, as opposed to a maximum of one week, does that count as CI?
Edit: Just found a video where you actually explain the difference explicitly, guess I'll watch that later.
I'm a fan of CI, but I'm having an issue with the idea that a feature branch necessarily means longer times between commits. If I were to run a development team, I would require the same interval between commits regardless ... basically when a small feature is ready. The gain (as I see it) from feature branching is the ability to merge from trunk and see any conflicts or failures before merging to common code.
Thank you for making the CD concepts presented clear and concise, that aspect pulled me in from busy, busy work. And much to my benefit :)
Thanks, glad to be of help 😎
That's why team members talk with each other. You can't say people don't know what the other is doing. If you work like that then you don't have team, but instead group of individuals or problems with project management.
Pushing to master everyday means fixing merge conflicts everyday. Is it best way to deal with this problem? Even better, you can't decide feature one is ready to deploy, but feature two need some changes. How do you do this? Do you remove feature two from master and add it after release?
Finally you chose the most corner case you could. Multiple big changes in the same code base at the same time. I don't know on what projects do you work but in my carier I didn't stumble upon that situation too many times. Because as I said at beginning, my team is there on plannings and rise concerns if someone tries to plan features like that. If it's needed you have more possibilities.
You can create feature branch for 2 or more people and they work on the same code base but not on main branch. Still I needed this technic like few times in my carier.
I think we can have the best of both worlds: consider the following workflow:
The is a "trunk" branch called "dev", and each person creates a feature branch from it and then merge it back to dev after the feature is complete, with a pull request and ab associated code review. After an arbitrary number of features are merged, a new release os born, and it gets merged into "master", the production branch.
If we create a "dev-ci" branch, and have the team muse feature branch, but at the same time rebasing/pulling from dev-ci and merging their changes into dev-ci frequently, we then achieve the ci goal, and the feature branch with commits filtered by author is still good for a traditional code review
We basically do what you describe in the first paragraph. I don't think I understand how the process you describe in the second paragraph differs, though.
@@ShawnThuris sorry for the typos. Instead of commiting and pulling directly on the trunk branch, you merge and pull/rebase from your feature branch keeping it synced with the trunk branch.
It may sound as extra work to accomplish the same result, but it would still allow us to link work items/issues on the management platform (GitHub, azure devops, etc.)
You are basically describing what he did in the video though.
@@defeqel6537 yes, but using both feature branch and trunk ci
What to do with code review? I can't imagine that there is no place for it in Continuous Integration ?
I have the same problem. I couldn't find any good solution to do code review after code is merged. Comparing selected commits seems to be very challenging (or even impossible). That is the only blocker which prevents my team to commit directly into master branch. At the moment we trying to creating branches which lives 1 or max 2 days. But I would prefer to merge directly into master.
@@rafacypcer1408 Why don't you use tags or a specified commit message? You can review the "sum" of these specific changes iteration by iteration.
We used JIRA and Bitbucket, and we had commit message conventions where we put the feature id and the task id. This enabled us to review the task alone and also the feature ("FeatureId/TaskId message"). Each commit was visible under both the feature and task in JIRA.
We used feature branches but with this you still can observe what is happening with your feature, and you can always create more tasks under your feature as it is only closed when the team is done. The only downside I see is that you can't reject feature merges, just the merges of your little tasks. But with trunk based development it's not really a factor as you want to apply changes rapidly instead of creating a big/coherent feature changeset.
In Daves philosophy you ensure code quality through TDD and a significant amount of pair programming. No code review
Don't use code reviews as a gating mechanism for changes. Code Review should be something that gets done to look at how and why things were done the way they were and how it can be improved. If you use code reviews for enforcing methodologies and finding defects use automation for that. Yes it's 'after the fact' and the incorrect code is already in master, but using CI it can be quickly fixed.
The flip side is if you and the other developers are constantly pushing changes and pulling changes code is being indirectly reviewed all the time.
@@Rekettyelovag Thanks. I also use similar pattern for commit message. I'm also able to see in my managment tool (in my case Azure Pull Requests) connected commits. But the problem is that I can see the commits, but not files with diff's (which is obvious, because it most probably git limitation in my case). I don't know any git feature which gathers selected commits and show diff's in files comparing the last commit. I san see changes by selecting two commits - not more.
I understand that with CI/CD we need to change methodology. CI/CD is used mostly for fast development - not for code quality. Code quality should be resolved somehow different - like changing team culture, improving automated tests, applying linters/code rules etc.
For approach with code review before commiting to master we had step where someone could verify someone code (which blocked development - because code 'frozen' during code review shouldn't be temporary used, by someone else). Merging directly to master forces every developer to review changes everytime when he/she needs to pull new changes. But it happens constantly. With feature branches reviewer was analysing the whole feature context. With branching directly to master this context is lost, because of continues delivery.
Also when working alone, branching is sometimes useful. I've already done it a few times.
Yes, I actually do it regularly. I often have between two and 4 branches in my personal projects.
So. How do you get the branch through QA?
Oh...that happens after you merge? What about merge conflicts? How long to resolve those?
We know that feature branching is convenient for developers. But that's precisely why it is bad for development. CI is far better an approach overall.
When working alone, branching is only a way to mark where you're going to come back to if your experiment is a flop.
@@timothyblazer1749 In most VCS merges are more complicated than they need to be.
Git for example always needs a merge commit if you merge something which is not just a fast forward.
In Pijul, which is still in development, you only need a merge commit if two changes change the same line of the same file.
And merge conflicts should appear less often anyway.
But that's more of a problem when doing CI. When everyone is pushing on the same branch, up to every second commit is a merge.
Branching does allow for an individual developer to create ownership over the feature they are working on. I'm not sure if it's always the "I'm the hero" motivation described in the video. Having pride over the work you do does create motivation to complete it which is important. It's also important to note that individual contribution does have a tendency to create tangents of thought that could become unpredictably useful or not useful. That variation can come in handy. That said, I do find it hard to argue against the idea of more closely collaborating to create better awareness of the application state. Maybe the experimentation and offshooting should just be left to a branch that's not necessarily meant to integrate for experimentation purposes; left to individuals who want to propose changes in structure and such.
Excellent illustration of why the ingrained scatter-gather pattern of working is inherently flawed. We're experimenting with mob programming where all these concerns melt away, since no one is sent to their cubicle to work in parallel and in isolation.
Interesting, I certainly agree with the principal, but think that using feature branch and PRs is useful. IMO I like to isolate CI from source control as much as possible, so there is one mechanism to trigger a build. Tags. A modular code base helps avoid conflicts more naturally also. Frequent conflicts between independent features is a sign that concerns are not well encapsulated or decoupled.
This is pretty much close to how I always worked. I usually work on the master branch and push/commit regularly. Pretty much any change that brings a benefit while not breaking anything.
I use feature branches for bigger changes that take longer, but I can't check in the intermediate work because it might break the existing state. It would not compile or break features. Usually a bigger refactoring. Where I want the intermediate states in the VCS but not break the master. As soon as possible I'd merge the feature branch into Master and continue working on it there.
In some cases I combined it with the Dark Launch or Feature Toggles. Minimizing the time I work on a separate code base.
Funnily enough, I rarely have merge conflicts as I usually merge Master back into my feature. My colleagues kept longer running branches and had to spend much more time on the merges. Mostly because they rarely pulled Master.
There may be an issue with semantics;
TLDR; If we branch off of the trunk, but the branches are so small that several of them are PR'd in a day per person, does this still count as feature branching, or is it CI/CD as you describe?
I run a small team of 3, sometimes 4 people, and we never work directly off of the trunk (which is actually our staging branch, but that is another story), as we want it protected, and instead, we make what we call 'feature branches', which is often small break downs of larger features into tasks (so maybe we should call it task branches), small enough to complete at least one, but usually several of in a day.
We want the trunk protected as we want to review all the code via PRs before it goes into the trunk.
We do this so frequently, that merge conflicts are quite rare, and our CD pipeline is often backlogged near the end of the day.
In addition to this, we have small 10 to 30 minute (depending on what needs to be discussed) meetings throughout the day to discuss what we are working on and the direction we are taking, which helps avoid having to re-do a lot of work at PR.
This strategy involves, more frequent, but much smaller PR that requires very little comment.
Seems to be working great for us, except as the lead I need to work on getting my code reviewed by the other members more often and not just self approve.
So basically, does this workflow still count as CI/CD as you describe, because it's still frequent, or is it disqualified because we are not working directly off of trunk. And if it's the latter, is it possible to protect the trunk in any way in this workflow?
This is exactly my situation with my team. We create small task/feature branches that get made into a PR within the day. On a good day I can make as many as 4-5 small branches/PRs.
The hard thing though is our code review culture. Because reviews require context-switching and there’s so many PRs that get generated, there’s too much pressure put on the two reviewers that each PR requires (and we don’t practice pair programming). What ends up happening is people only review 1 or 2 PRs early in the day before tuning out of reviews to do their own work until the end of the day and by that time there’s a backlog of more PRs.
Curious about the answer to your question and the code review process too.
@@DanielSanchez_dsanc89 The short meetings I mentioned later in the day is where we do our code reviews, which helps contains the context switching to specific, narrow parts of the day, and since we do it verbally and with screen sharing, it goes faster as well, and we do them in batches.
And the other thing that I mentioned that helps reduce PRs is showing the code to each other (and hence your reviewer) in a checkup meeting earlier in the day, before any PRs, to pre-empt mistakes before they are made and need to be corrected in a PR.
So our meetings are an early morning standup (7:30), where we plan out what we are going to do (but we sometimes skip it if we discussed it at the end of the previous day), then we have an 11 AM code checkup meeting where there are no PRs expected (but sometimes quick tasks get PRs by then), then a 2:30 PM code review session where we are supposed to do all the PRs (but in practice half the time it's another code checkup), then a 4:30 PM staging review session, where we are supposed to see the finished work running in a staging environment (or production with feature flagging if we would not have a staging environment) (but in practice half the time it's code review and seeing the code running on our local dev machines).
The goal is to get your code running as close to production as possible on the same day that you are making it. Our client wants a staging environment, so the closest we can do is a staging server, but we do have feature flagging so we can deploy but disable incomplete features to master, so it's seldom far behind.
@@DanielSanchez_dsanc89 Sounds like our situation: the review tasks piled up. We switched from a gated review process to pair programming. We are not pairing for every tasks, but for important ones (with a more or less accurate guess)
@@DanielSanchez_dsanc89 I've found that talking about your goals and ideas before-hand can help reduce the burden when reviewing PRs. The reviews go from "what is being attempted here" to actually evaluating whether the code accomplishes the intended changes (without side effects).
I was struggling to articulate similar thoughts. Short-lived branches for PRs of small, well-focused changes; and being disciplined to always keep master in not just buildable but in a state where it is deployable to production seems like it achieves the desired goal.
What if one day your product owner decided not to release and undo that one complex feature (i.e. shared library change, db schema update) that everyone else had already integrated deep into the trunk?
This is the reason you need loose coupling in software. Ideally, you'd have some level of abstraction between your code and dependencies, such that you can swap out one dependency for another, one DBMS for another, with the core code blissfully unaware of the change. It's why the Repository layer exists in Android applications.
I have a shell script, git-sync, that replaces git pull --all + git rebase + git commit + git push, which mitigates some of these issues. Dave criticizes this approach at 7:21, but my script goes farther. It also checks for git conflicts across all other active features branches (we delete branches after merge), and list other branches. Finally, we have very small tickets, and we break them down even smaller to tasks which are safely atomically merged to trunk. We aim for "tasks" to last no more than 24 hours. In stand-up, we view branches older than 24 hours. Of course, none of that's full integration, but it gets us very close. Close enough to be better than having tons of feature flags, which is often required for trunk-based.
I think the concept of CI has a lot of merit. We had to change our concept of a 'feature' to help work towards this. On the basis that 'no one should commit code to master that breaks the build', we couldn't get an entire 'feature' developed quick enough to commit to trunk/master, so we used feature branches. But then we just found that if we broke down what we meant by 'feature' into smaller bits, we could do commit to trunk/master without 'breaking' anything. This did lead to some issues about testing these 'mini-features', but nothing insurmountable. And breaking a classical idea of a 'feature' down into 'mini-features' gave us a lot of insight into just what and why the 'feature' was in the first place. This worked well with a general 'Agile' thinking in that there were a few times our initial 'mini-feature' break down showed us where we had gone astray.
I'm thinking here that even in a team that works using "feature branches" and has periodic long delays in "pull requests" getting reviewed it would still make perfect sense to split the work into a sequence of small focused changes for the reviewer to review simply because it's gonna be easier to go deal with and to give feedback on compared to dealing with a big ball of mud of changes that inevitably turns into merge hell.
To me this seems like it would be a clear step forward by practicing the parts of CI that one can in a broader environment that doesn't do CI... maybe this could be used to smuggle more CI thinking in there in the long term anyway if folks don't believe in its benefits until they themselves have seem them working for someone else in a part of what they do. This is seems like something that is perfectly doable with just about any regular toolkit.
This whole thing reminding me of how in an environment that isn't agile the easiest way to make things easier for oneself is to have one's own surroundings self-organize (even if it's just you) to the extent one has the ability to do so and to just work like an agile organization within a wider organization because it's still better for one team to be agile inside bigger organization as that at least still makes working more efficient and pleasant for you as a developer.
I do get the overall idea but of course there exist several cases where feature branching is actually more productive and perhaps even mandatory. A few examples that come to mind: 1. perform a refactoring in order to replace an external library 2. Any form of "experiments" that are not supposed to be deployed in prod 3. large changes that are difficult to release "darkly" (or that can be released darkly, but with a high extra cost in coding the "darkly" (which is actually a feature itself). Again: I get the general advice but in my opinion "continuous" must be taken with a grain of salt, as with most practices.
I'd say it the other way around. CI works for nearly all cases. Occasionally branching, not necessarily feature branching, is useful in unusual situations. So "take branching with a grain of salt".
The other take on this is that CI applies a pressure on you to think a little more about the design of your system. So, for example, I think it extremely unusual to need to use branching for your first example. Sure, I have done it, but these days I don't need to. Because I'd probably have a facade or adapter between my code and the third party library to make my code testable. So switching out the library would be easy. That is only not true when your code is tightly-coupled to the library, which is in some, very unusual, circumstances I may allow in my code, but very rarely.
thanks for your comment@@ContinuousDelivery. As stated in my first comment, I agree that the worst evil is multiple long lived branching, to the point of every single developer in the team creating a personal branch which quickly becomes impossible to merge (I personally witnessed a few of these edge cases). Yet sometimes branching is good, but you (that is: not the single developer but the team maintaining the codebase) must know why and what you are doing. Other aspect, this being more a technical detail: what do you think about the "short lived branching" technique, compared to the "commit to trunk"? In my opinion the short lived branching (branching but merging the branch very often) preserves the goal of continuous integration but is more solid in terms of protecting the main branch from breaking commits and it's easier to implement with modern CI tools.
But I do have a question. If I have to create features, that can take more than one day, I must create a new branch, and create a Merge Request, right?
Or even if my feature takes more than 2 days, should I merge it into master without review from other developers?
I agree with CI, but I have some doubts about large features and hotfixes.
The client I'm working for they have a controlled feature release, so constant merging seems impossible for this case. We only merge if the feature is business approve else we don't. Is CI can be implemented with this case?
I totally agree with fact that the longer you wait, the more clashes and conflicts may occur. The problem I faced with my team is that some time the work on a feature is incomplete before a deadline and we wanted to deliver some other complete features. Without feature branches we had to do all sorts of things, like turn off flags, hiding controls and even rolling back changes.
Teams that embrace CD change how they organise their work. CD is working so that your SW is always releasable, so after every successful commit, it could go into production. You can do that by waiting till each feature is complete, but that slows feedback and results in lower quality, or you can work so that it’s ok to release, even if a feature isn’t complete. This sounds weird, but is the higher-quality approach, according to the State of DevOps Report.
@@ContinuousDelivery Thank you so much for clarification.
I tend to rebase on top of whatever the main branch is regularly so as to act as if I started my work after the current code base state. This isn't technically that different, but I find it easier to reason about. Essentially I agree with your video that a continuous straight line commit graph is preferable to me. At the same time, small merges are pretty inconsequential and easy to reason about.
People do get quite heated about this so I leave it to them to deal with the worst merge nightmares while I commit on top of the latest when I'm done.
I get a lot of pleasure from watching these videos!
The reactive system one has blown my mind and really got me thinking.
CD, i'm new to programming and I just wonder, you often say you expect these CI branches to last rought around 15-20 minutes, or a day a most.
So my question is: What if the feature I'm refactoring or working on is incomplete? like say we're making a video game, and I decide that the jump mechanic needs work - just doesn't feel good to jump in this game. Well 20 minutes into doing it over, i realize I need to spend some hours researching the subject and how other games did it. Do I commit, or possibly spend a day or two being thorough in my current approach to the jump mechanic? meanwhile, other devs are building level design choices based on the jump mechanic that depends on it being what they thought it was.
Generally agree with this precept to integrate small bits of code frequently. The only exception I can think of has to do with cybersecurity or life-safety system requirements that disallow dead code (i.e., can’t get there except in an unplanned error state) under all circumstances due to the risk of accidentally entering those code segments. In less critical codebases, I can easily agree that feature toggles (and similar control measures) are adequate and should be used instead of feature branches. If you follow a stratified development approach (e.g., Strata from Vitech for systems engineering), then you can, for example, stub the feature and integrate the stubs with the trunk, then add functionality as the feature is developed.
Literally a day ago was talking with a friend of mine who is a QA lead at a big IT. He complained that it's a nightmare to deal with 10 teams pushing changes into the develop branch. He said that it is unmanageable because it's almost impossible to blame what feature, what change, and who introduced a bug. They spent hours just to identify this info. They do CI, but everyone understands and complains for that kind of workload branching model would most likely be much more manageable for QA.
I hate to break it to you, but if you have a develop branch and each team is branching using something like git-flow, you're not doing CI.
What about things like ROS where you have different distributions that behave differently? Typically, some packages will have different versions on different branches depending on your distro.
I've only rarely run into major conflicts when merging. Even when we had several people working on major additions to the same product. Most of the time branches are used to make changes or experiment without breaking the main branch builds. And since there are many things that we are unable to build and run locally code has to be committed to put it through the build, test and deploy pipeline. We could certainly investigate making it possible for each person to do that locally, but since branching is actually working it would be hard to justify the time and expense.
Personally I'm not a fan of pair programming. We certainly use it on occasion, but I'd burn out in months if I had to work with someone looking over my shoulder all day. I've turned down lucrative job offers because the job would have involved working like that all day every day.
One thing confuses me with the “no branch” concept: in order to have a code review prior to merging, we *do need* a branch other than master (or whatever trunk) to be able to create a pull request, right? Or how else a code review would work if everyone is committing to the same branch?
While you could do "trunk based development" (commit and push directly), CI does not necessarily mean no branches. You still might want code review and other QA gates in place.
I have also seen "short-lived" branches basically for code-review and such. So you do all your TBD in this "short lived" branch and merge after code-review/checks
Well, that's the thing. Often, to do trunk-based development as effectively as possible, you don't do a pull request based code review workflow - instead favouring continuous review via pairing or mobbing. That's the ideal.
However: there's a difference between a pull request branch and a feature branch, and it's mostly a question of scale. If said branch only lasts for hours, then sure, that's continuous integration. Directly pushing to trunk is just the taking that approach to the max.
I don't do branches. Everything on main. Git tells me when I haven't updated when I try to push.
I don't see the point in PRs either.
Whenever I see a PR I lack the context around the decisions, I let the pipeline feedback in the form of testing and static scans etc to ensure quality is brought in up front.
Yeah, I don't see how it makes sense at the end of the day without some branching. My understanding about what he's pushing for is just more frequent pull requests
I think there is more to it than thinking ourselves as heroic loaners 15:01, one of the ways we breakdown complex problems and also with the code we write is the idea of breaking down a complex problem into specific concerns where we can think solely about each part in as much isolation as possible be it steps in a process or the responsibilities of an object. Isolate a single part of the greater problem at a time. Therefore, merge conflicts with someone else's code means we need to context switch to understand a potential different problem context in order to resolve the merge issue. Now in my experience its the guy who commits later that is responsible for resolving the issue (is this the unresolved special source that's nagging at the back of my head), and hence has to do this extra work of understanding an additional problem domain/context. Resolving who has to context switch stop the breakdown of their own feature and understand how their own problem context interacts with someone else's current work in the same problem domain ... should this be left to who commits second is seen as responsible for the merge ... the associated work is delegated on nothing more than who can smash that commit button before the other? It also means this is the point at which people's different coding styles interact, has the code been refactored or is still in a state of just putting together something to pass unit tests. I'm not saying leaving people in isolation till their own feature is complete before attempting merge with another complete feature, meaning there is a potential for a large divergence of code. I just think there is a social/phycological dimension to ... THE MERGE!!!
I've found myself merging code ... before commit, "oh wait ... it's changed again" ... ok spend the time understanding the merge ... "ok right think that's my changes merged ... better do a pull form the remote" ... "oh jeez again some more small changes made before I was able to commit". This is doing my head in if only I had just been able to commit before I had to pull my hair out trying to understand this stuff written by someone else.
Maybe it creates a feeling of winners and losers, and it's the speed at which you commit that is being rewarded not the quality of the code. Maybe I'm just crap and slow at merging peoples code, maybe needs ideas on "Pair Merging?", but does anyone else have experience of he who commits last is the loser??
Never really had that problem myself. But then I have always been vocal when I see conflicts and hero-culture and try to break out of it as soon as I see the signs.
There is a saying in the DevOps world - "a bug, is a bug, is a bug". Meaning that bad code, bad process, bad security, bad compliance, low trust, bad logging, metrics and audit are all bugs. But we tend to pretend they are someone else's problem because "I'm just trying to get my task done".
As long as everyone has the mindset that it's a contest, that developers shall sit alone in their corners with their headphones on, typing code all day. Instead of working as a team (across roles), have good practices, a good modular design (so that there is less chance of stepping on each other), working in pairs and sometimes entire groups (see mob programming) we tend to default to defensive behaviour and then there will be friction, frustration and no joy. And then we try to smooth over the cultural issues with processes, Gitflow and feature branches and silos and handoffs with 27 new roles. A team should be measured based on result and outcomes - not individual contributions.
My point? Any mindset or culture that focuses on individual "performance" (stack ranking, ego, loudest-guy-opinion, "winners-and-losers") and cop-outs like "but I'm an introvert" is A BUG! The issue with seeing ourselves as heroic loners is indeed the biggest issue in SWE.
Since in your scenario both of you apparently were working in related parts of the codebase - I think you should have taken a step back, sat down with each other, talk, plan, pair-program and collaborate on both of the changes instead if sitting there frustrated that "the other guy screwed up my changes again and now I'm getting behind and losing the scrum story-point race". Maybe I'm taking this over the top, but only because I've seen it happen to other teams so many times. Learned helplessness is the worst place to be, because it absolutely drains all energy and joy of going to work and life in general.
@@ddanielsandberg I find it a bit scary when normal human phycology gets referred to as “A BUG”, I think everyone has different personality traits, strengths and weaknesses, and in different situations weaknesses become strengths, in some situations being an introvert that isolates themselves to push through a single focused idea is what’s needed, but then there’s other situations where it’s collaboration that’s going to be the enabler. That doesn’t mean I haven’t been pissed off and got people kicked off a team … but I find it most stimulating and sometimes most frustrating to work in a diverse team, argh and though it pains me to say so, good management of people counts.
@@derekcarter4157 I didn't mean to imply that people are bugs. :(
I meant it more like the system is buggy if it enables and incentivises certain negative behaviours and discourages other positive behaviours.
The software industry has spent the past 40 years hiring and attracting certain personality types and I think it has had a detrimental effect.
I know how it feels some days, I'm thinking of leaving the software industry and become a carpenter or painter or something because this has given me CPPS.
“A bad system will beat a good person every time.”
@@ddanielsandberg aye this covid lockdown from last year seems to have been terrible for my eyes just being in the house never looking more than 7 feet away (or maybe I’m just getting old). But yes it’s a software dev is a creative endeavour, even if all our best ideas are stolen! Processes and techniques are to help people create and communicate (and I’m the case of SAFe and Scrum make a quick buck) teeheehee.
Hello, I get the idea and I don't disagree. I think we'd need some detailed examples though, about how to do it. The rationale seems as correct as it is high level. The practical level is, I believe, what keeps many doubtful.
I would really like to hear about how folks do code review in trunk based development. Years ago I worked for a place that did trunk based dev without code review. Integration worked well and the test suite was solid but the design of the code sucked because there wasn't a system for reviewing code design and architecture. I am wondering if it is possible to change workflow to accommodate this without having to have every change reviewed? I wonder if docs around feature architecture committed to source control coupled with well planned conversations would make sense? Anyone tried something like that?
Just realised I am one of many asking the same thing here
A) We are pretty much stuck with feature branches on an organizational level
- we are trying to keep our branches small and frequent though (still not perfect, but we are improving), in the end, communication is key here whether on the individual feature-branch level or when talking about the "complete" feature and the plans for it
B) I fear CI is too straightforward (for a lack of a better word)
- you either need really good pairs programming together to avoid silly and/ or hard to find bugs creeping in, especially as less capable developers are just as capable of breaking tests as any other part of the code
- you miss that part of separation that a pull request and review gives you; this is personal, but I can better focus on reviewing changes and trying to figure out what they affect when I go into code review for a (small[ish]) PR. This is also true of my own PRs. This, I think, is especially true when your code deals with multiple concepts that you need to mentally context switch from. I'm fully willing to accept that this might be my prejudice talking, since I haven't really been involved in a real CI project thus far.
edit: I do adore your videos though, including this one
Branching with small FF trunk based development just works. Especially if you want to do different types of testing that is not similar to your main app (i.e. different frontend with a ephemeral env). This is something that doesn't have to even be defended. It's not either or, we can take parts of what your saying and branch as well. Branching is not guess work though and having everything in 1 branch stops dev testing and exploration. Testing one small change to code is testing more than just that code itself. You have to also test how it interacts with the env you have set up and I don't see how that would work with only 1 branch.
We have builds and test (functional and unit) runs on every commit of our branch. We do rebasing instead of merging of main develop branch into ours. We strive to make multiple smaller MR/PRs but they always have to be a finished working bit. Since codebase requires quite a lot of refactoring so that C++ api guarantees code correctness far more than we are able to achieve in Java version of the product even though Java version is younger... Which is really important for multi threading purposes (when using shared data). But the most important part are code reviews (no tests are not replacement for code reviews - and I'm talking about to some extent almost formal code reviews - they are additional help and documentation). Based on this and the fact that branch is developed for a week and then gets stuck in code review for about 14 days (including changes to the code) I would really like to see how CI and constant pushing of code would produce better code. Faster written for short term gains? Yes. Better... Please explain how as I can't see it work for anything more than service development where partial uptime is more important than stability of the whole.
Hi, I love your videos! Can you please share of what data you are talking about that shows that you build software faster and better using CI/CD? Are there any scientific studies about that?
The state of devops report is commonly cited although I haven’t dug into it.
Just understanding the issue should make it obvious why it's better to integrate code early than having developers hide code from each other and do giant merges after the code branches have diverged. If it's not obvious, I suggest watching the video again, with thought.
@@rosstempo3765 sorry, but that statement seems to miss the point. Of course it is better not to have to deal with this specific problem. But to say that it makes for better software and faster at that is a big stretch without any data, is it not? At least for someone with a different opinion, and those are the ones I have to convince 🙃
@@Feyon You can't convince other people. I've learned this the hard way. You can only provide them with information and they choose whether to take it in or not. If they are not willing to watch a RUclips video explaining the concept and make the conclusions themselves, any kind of research paper won't convince them either. A person who isn't willing to learn, will not learn no matter how much "convincing" you try to do.
It's DORA organisation, state of DevOps report. A light weight overview is found in the book Accelerate, which I recommend!
What about the argument of atomic commits? Should every push to trunk be guarded by tests? If so, wouldn’t the scope of tests dictate the frequency of integration? Is it therefore better to base integration on a set time frame, or on set scopes? How would you determine scope?
Yes, every commit should be covered by tests. Yes that means that a very effective approach is to allow tests to dictate the commit frequency. If you practice TDD, as I do, that means that my commit frequency is usually roughly once every 10 or 15 minutes. That is pretty close to "continuous" in a practical sense.
Great video Dave. I heard you first on the Changelog podcast, then looked you up here. You've spoken to my years of managing feature branched projects and suffering the inevitable conflicts - even in the very small teams I worked with. It's spot on. I'm running the rebuild of a Drupal project, which will initially be built without any automated tests and all in feature branches. In the very long run I'd like to move towards CI and TDD.
One of many questions I have though: Our development so far has gone like this: A developer works in branch, I review the PR and test functionality and pass it over to another non-dev stakeholder who also reviews it against the spec and tests it on a staging set up. In CI, do non-dev stakeholders have to do their testing and reviewing directly on production? In our experience this non-dev stakeholder often catches mistakes where the spec was misunderstood, and we feel strongly that these mistakes should never make it into production. Sorry if that's been addressed already! Thanks for the videos.
My preference is to try an in-line this kind of overview, so that it isn't a "gate" in the development/release process, but is part of the development. PRs and reviews tend to slow the process, so the idea is to do these things continually alongside the development.
Instead of performing a code review at the time of a PR, pair-programming gives you a continuous review as well as lots of other benefits.
Instead of non-dev stakeholders looking at the code catching mistakes after the code is thought to be finished, build automated testing into the development process so that catches most bugs. Where a review is meant to help sanity check that the change looks as expected, rather than acting as a regression test, then do this early in the dev process as soon as you can.
Overall the aim is to build quality into the code from the outset, rather than try to "inspect quality in" to the code later.
Thanks for the question, I think I may do a video on QA and it's role in CD teams that may explain some of this stuff.
@@ContinuousDelivery Thanks for the reply! Yes, I could review the code over the dev's shoulder prior to commit I guess, but the functionality that was added has to be tested/reviewed within some non-production environment. So yeah, that is a gate in our process.
It seems to me that in CI you have no need of a staging environment. You appear to just have local development environments, and production. So if that's true, then our non-dev stakeholder can only review what is built when it's on production, because they can't use the dev's local environment. For us, the key thing that this person brings to quality is ensuring that what is built matches what was intended by the spec. Automated tests written by a dev that has misunderstood the spec won't catch that sort of mistake, it can only be caught by the person who knows the requirements. Love to hear your thoughts on that!
@@dansearle5994 Not really, the deployment pipeline is a controlled environment, outside of local development environment of an individual dev. It is a shared space where changes are integrated and evaluated. Usually in production-like test environments that mimic prod config and scenarios. These are close to a traditional staging env, but not quite the same. Ideally the person that "knows the requirements" is part of the team too, so close enough to the dev to see the soliution evolve and so catch these kinds of misunderstandings before they progress to the (too late) point of be deemed "finished" by the dev team.
@@ContinuousDelivery ok I see. We're used to reviewing the produced work in our own time, on our own machines (accessing the staging environment), so it's a big shift to go from that to review system where we're with the dev more often, looking at what they're working on on their machine (either in person or via screenshare I imagine) in little bits. It's food for thought, I'll see what I can do, over time!
Not mentioning if you are talking about short or long living branches does raise a lot of questions for me. The big four are:
1: If I work in a situation where I can't communicate with the people who work on the same code as I do, do I trust them enough to allow unreviewed code into my main branch?
2: Is there a chance that your commits are as big as my feature branches? (one unit of work for a feature done, between a day or two while daily re-intergrating the main branch.)
3: the whole thing seems to only work if you don't use the de facto standard tools like PR's since they are by design only applicable to branches, therefore we need to get an extra tool to do code reviews?
4: Do I really want to trigger all my pipelines when I push a typo fix just so everybody can integrate early?
btw trunkbaseddevelopment dot com acknowledges Short-Lived Feature Branches, what's your opinion on that?
Keep up the good content :)
So this topic makes a lot of sense in a lot of ways but one issue I see is with how my studio works. We have new programmers and interns from a local college working on projects often. This means that any one of them could break things on a moments notice with CI just due to discipline and skill being not there yet. On the flip side it would force them to pay attention and work more effectively though at a cost.
This isn't a finished thought really and context matters but I felt like voicing it can't hurt, all while I continue to think about it.
I think that is a fair point. My prefered way to introduce people to a team is through pair-programming, that instils the discipline and gives the newby a guide to the codebase, working practices and so on.
Thanks for the vid! It's very refreshing to see motivations for "why we do things like we do" other than:
* that's how google does it
* that's how good developers do it
I appreciate you adding a scientific-like point of view.
That being said, the context where CI really matters is almost clear, but not quite, so I'd like to understand:
* Does CI matter more for companies where really big teams (or lots of teams) are working on the same code-base?
* Does a microservice architecture make CI more relevant or slightly less important?
While all I can say is "I have an experience-based opinion", here's my take on the two questions:
1) It matters for different reasons.
In smaller teams, you will have fewer versions of truth. At the same time, you should also have fewer reasons to diverge.
In bigger teams, the it's the exponential growth of failure points over time that makes CI pretty much a necessity.
2) There's something interesting about microservices, and that's the decoupling of implementation from API.
As long as you're not changing the API, there's no need to withhold any tested change. Indeed, if you can push out the change the minute it's made, you're maximizing the value of your work by minimizing lead time. And one of the core advantages of microservices is that they let you do this, so if you don't - you miss out on a large portion of their value.
BUT - when you're continuously changing the API, you need a strategy. Many companies don't think about that, and hence get into a terrible mess when CI'ing microservices.
I think you should start with a definition of feature branching because everyone has a slightly different one
Interesting idea. I would like to try it in my work, but I cant imagine how the code review process looks like.
Also, when someone pushes a broken commit to master, all work stops until it is fixed, what to do with that? I suppose that this approach requires TDD.
I think this is where forks come in handy. There's one main/master repository from which developers fork their own copy of the repo. They can now work on their own master branch without touching the master on the main repo. The developer pushes changes to his/her master and makes a pull/merge request from their master to the original, and this is where code reviews and automated tests can be used; developers changes won't be merged to the original repo until the code review is done and the tests and the build passes.
@@Codefortyseven nope, u now have late integration. this is not ci
make sense to me, but I have this question, I have worked in real big companies with hundres of developers around the world, 24/7 coding and adding stuff, using the branching approach, some devs (no me :P) sometimes pushes their broken branch to master, blocking the rest of the team, with CI that will happen too, and even more often is my suposition, Thanks for the video.
Rigorous automated testing is imperative when it comes to CI. Devs need to pass all the automated tests before they push to master.
in providers like Github and Bitbucket there exists settings and controls to ensure that tests pass after merging. there's also a practice for the team to fix a broken build immediately if it fails
for example see
GitHub.com -> Administering a repository -> Defining the mergeability of pull requests -> About protected branches
Wow, 368 comments in a few days. You channel sure has grown, that's great to see.
That is biased, in my view. What you call fundamental is not working in 100% real life scenarios. There are cases when it's useful to guarantee changes isolation for the sake of risks mitigation. Think of it as full transaction isolation in DB, like when evential consistency needs to be avoided.
In small projects your approach is doable and make sense
but in bigger ones involving many people I don't beleave that it would be more efficient than feature branching.
Well, it is how Google work, and I have done it successfully with teams in the hundreds of people. ruclips.net/video/W71BTkUbdqE/видео.html
@@ContinuousDelivery I'm not sure I understood you correctly - do you mean working in a team containing hundreds of people?
For sure it was not a SCRUM team then, right?
I know that Google they are working mosty in micro-teams (like 2-4 developers in a team) and then as I said I agree that the strategy makes sense.
@@zenonkurcz7215 well it wasn't Scrum, not my favourite approach, but you are right, agile projects are delivered by many small teams, so our teams of hundreds were divided up into smaller, functional teams focused on one part of the problems, but working in a shared codebase, with a single deployment pipeline. Since we were talking about CI and branching, that was the context of my comment. Google hold nearly all of their code, billions of lines, worked on by thousands of people, in a single repo!
I think I can see the point you're making, but I don't think not branching is the way to go if you'd like to be able to always deploy the head of your main branch (master of development). In mature projects there can be huge CI pipelines, which are not desirable to run locally. If this pipeline determines I introduced a performance problem in the code for example, I would not want to have this in my main branch. Instead I'd want to fix it in my own little sandbox (my feature branch) before merging.
In companies where I worked, the burden of proof was always with the person trying to merge new code. Thus, you merge the main branch into your feature branch regularly and you fix any problems that are introduced (in your feature branch, not in the main branch) before merging your branch.
Besides that, I'd recon this is why we have all those scrum rituals to inspect and adapt.
Recently I have had to branch to test a new version of a third party library that needed a large refactor to integrate. This took around a week. Once complete I merged it in after it was tested and verified. If I just worked in the main branch for that I would have created a window where the software is not releasable. Is that a valid time to do branching? I typically do CI but in that case I did not.
Since you seemed curious about why people have an emotional response to this subject, I thought I'd chime in.
What emotionally turns me off about ci is sharing my code before it's finished. Imposter syndrome is rampant in our profession and merging code into a shared branch before the feature is "done" make me anxious.
Rationally I agree with you. Shorter cycles lead to better code and seeing everyone's else's "incomplete" might even reduce feeling of inferiority and serve as a learning aid for junior developers. Nevertheless the anxiety remains.
Love the videos
4:35 Totally understand this moment. Despite your advice not to fork, if we as a team decide to violate the suggested rule, is there some tool (for example as GitHub bot or VSCode extension) to watch all feature branches and warn two developers that they are going into the "danger zone" and probably need to communicate what is whom scope?
There is such a tool, but it is Continuous Integration - sorry!
So from what I understand you integrate more frequently? But what if it's going to take some time before your code is in a compilable state? Sometimes we need to make big changes.
Why would it take "some time to get the code into a compilable state"? Just throwing it out there; what would need to change for things to always be in a working state?
I need more information on this. Can anyone point me to something? Let's say you keep your branches short, so you merge your branch as soon as you've abstracted away a certain part of the code you want to write a new implementation for. Now you create a new branch after merging the previous one that was just for refactoring. What do you call both branches? Do you create separate stories so you have unique ids to name the branches with?
You can read more at www.behaviourdriven.org
No need for named branches at all. Work locally on master, when you commit also push to origin/master and run CI there. It’s pretty simple really.
I've really enjoyed your channel for a while now. I have picked up far more insightful ideas than I ever thought I would when I began watching.
So my first serious comment on your channel is this...
I absolutely love the shirt!
If you are dealing with a live system, with live customers who may report bugs that need to be fixed ASAP, maintaining an active "golden master" branch is the easiest, safest and fastest way to deal with it - and that means NOT having a bunch of barely tested or reviewed code go into the mainline. And even for an individual developer, I often use feature branches on my own projects, where I'm the only developer, if I want to experiment with different approaches that I may end up discarding later.
I really don't get the hate for feature branches. I work with some trainee developers that need more review than others. I don't WANT their code to be continuously integrated - that would actually cause problems. And as for long-lived feature branches creating a growing tech debt/merge risk - just keep rebasing your side branch on whatever's been accepted into master, and you'll never be that far from a merge. Seems pretty obvious to me.
To avoid branches and to just have everyone work in the master branch seems like chaos to me, and putting any customers of a live system at risk of not being able to provide them with timely bugfixes.
People hate feature branches because they don't know how to merge and are not competent devs imho and want to pretend they are some code ninja when they are just code hackers. 100% as you say keep rebasing from master on a regular basis. It is not rocket science to pull to keep up to date.
I agree. It's sad many people after gaining experience will look back on this video and say they got ill advice about not branching. What made Git popular is that rebase and merge are easy because branching is a must.
The problems mentioned in this comment seem to be the product of not enough test automation and not utilizing pair programming. Having junior developers work on features by themself in the dark is the reason why you don't want to integrate their code. Have the seniors pair with them instead of waiting for a PR to review.
Great presentation! Thank you.
I've come across some arguments/trade-offs which I've had a hard time answering. Perhaps someone can help?
1. Exploratory Work - someone told me they like to take a piece of code, break it up, and put it back together in different ways. They do this to make sense of the code and to explore different ideas. The ideas they're exploring don't always work out. Hence, they don't want them to be in master. Is this type of work something we still want to continuously integrate?
2. Squashing Commits - some teams want the commit history to follow a specific pattern. CI/TBD changes the pattern in some case - i.e. commits are not neccessirly grouped by feature, fix, etc. Clean up commits are no longer grouped by a branch and therefore squashing those clean-up commits becomes more difficult. I believe the argument here is about making the commit history a bit harder to understand and/or adversely affecting troubleshooting techniques such as git bisect.
1) If you are doing exploratory work and want the freedom to move fast and break things, then that is not "production quality" code. So you should throw it away after you have learned the lesson. No problem with that being done on a branch, but don't EVER merge the branch back to something releasable. Instead, learn whatever you want to learn, then re-write the code with producton-quality, TDD etc. These experiments are called "Spikes" in Extreme Programming, and should always be done to learn something, not with the aim of creating releasable code.
2) I think that this is a mistake, there are other ways to "tell the story" of a change, without compromising quality in order to make the commit history read nicely. I wrote about this here: www.davefarley.net/?p=263
@@ContinuousDelivery Makes sense - thank you!
I'm struggling to see what you're getting at at the 4:20 mark and in general I'm struggling to see why you would consider CI and Feature Branching as mutually exclusive concepts. A feature branch only exists for the duration of the development of that feature, once you're finished it gets merged in (integrated if you will...). To make sure a feature branch isn't straying too far away from its parent branch I just rebase or merge. Have I missed something?
Where I work we use feature branches.. one thing that happens from time to time is to remove a particular feature branch from a release candidate... good luck doing that if you constantly committing to master.
Well it is easy enough if the design is nice and modular. Version control doesn't fix this problem, because if you need to merge your changes with other peoples, you still have to disentangle them to revert. This is much more about good design than it is about VCS strategy.
@@ContinuousDelivery Appreciate the reply! Find your videos challenging but I am trying to keep an open mind. I'd guess what I am doing according to TDD is an antipattern. I rebase all of my commits in my feature branch into a single or fewer commits instead of filling git history with 100 commits for a single feature. I feel like I'm dong part of TDD as far as starting with the test cases first.. I guess I'm trying to wrap my mind around how to implement TDD fully and thinking of the use cases that I do go thru during the release process. For example merge hell for feature branches that are really old. .that is one major drawback. Thanks for the videos!
For example with feature branches.. if my boss decides for whatever reason to remove a feature from the release.. we can create a RC branch without merging that feature branch.. if everything is always committed to master.. that is going to be a lot of work. If things go really bad.. I can just destroy the RC branch and try again (leaving out any feature branch) from that RC branch. I guess that is my point.. thanks again!
I've asked in one of your previous videos about the process of reviewing changes to a code base with continuous integration. I'm a bit confused about what the workflow would look like. I work with a small team, so cashes with merges are far and few between, but the most important part of our workflow is the review process we get when we ask for merge request. What does it looks like? How do you get that feedback from your teammates? Specially since that feedback happens when you got hands around the changes and it's all fresh in your mind, not 3 weeks from now when I moved on and a teammate happens to look at that part of the code.
Yeah, I am in a similar situation. I am working on a small team with currently 2 other developers. We do use feature branching and when we are done, we issue a pr requesting a review from the others. Even, or perhaps especially, in a bigger team I think reviewing code is important.
It also helps you seeing what impact changes from others may have to your own branch and knowing what you will need to adapt. With CI, I feel like you might run into situations where you see a conflict and don't know what the other's changes are for and how to adapt them, whereas a more complete view of there changes may help with that. That being said, it is generally obviously easier to merge tiny commits.
I should also mention, while we usually have one, and exactly one, bigger feature in development at all times, the other branches are usually pretty small and never last more than a couple days or maybe a week in some cases, so there are never big discrepancies and when working on the bigger feature, it is feasible to pull from master.
We work in feature branches on user stories, not epics, and I feel like this video might talk about the idea of developing epics in a feature branch, which I would agree with is not a good idea and would make the review process quite cumbersome as well.
I also have this question. If we want to be continuously ready to deploy, we need to have some limitation on developers just pushing to master. Even if we do pair programming (which is another big change to the development workflow) just those two developers may not be enough to confirm a change is correct.
At my work we are trying shorter lived branches. So still basically feature branches, but putting it up for review/feedback/merge before the whole feature is done. Aiming to have branches live a few days at more. It does take more time for reviews, but I think it is worth it for the more frequent feedback and avoiding conflicts.
@@edwardweir5784 Well, the techniques all work together. If you're doing TDD, you should have some confidence that the code is correct. If you're doing pair programming, the code gets reviewed before it goes in. And if you're doing CI, the code goes into the main branch frequently and you don't get surprised by sudden big changes.
You get feedback by communicating with each other. A feature branch in itself won't review your code for you, and CI won't either.
If you want to set up a code review process, you can literally do it however you want with CI. Use the method you want and one that works for your team. It's that simple.
@@markjreed even if you do TDD, that still can mean, that a developer doesn't write a test together with the new feature, so the new code could be incorrect, or the test itself could be incorrect.
I am not sure if in my work we have CI. We do have a dev branch that we commit to it almost everyday and max 3 days, and every couple of weeks we merge to master.
We are very small team and merge conflicts rarely happen, and if they do happen, they are very small.
I am not sure what to call our setup.
I really like the concept of CI and when I talked about it with the team, they told me that they tried it before I joined and didn't work " because we are small team", could it be the case that CI might not the best option for small teams
it sounds like CI to me.
Since you all work on the dev branch, it is CI. That you only merge with master every now and then, shouldn't be an issue. That might just be your company's way of declaring something "stable" or releasing it. So I'd say it is CI but not necessarily CD. And that's okay.
Thanks, you correctly answered my arguments of pulling every day. I agree CI is the best way and committing to the trunk every day is the best idea. That graphics was useful.
Only point is we must have test suite there that covers well.
you cant just pick the worst examples of your opposing viewpoint and the best examples of your own and then say yours is better... sure if you are messy you can lose 12 months of work using feature branches.
Another thing: Your method introduces loads of dead code into the production branch. If I look into the production branch, I want to only see code that is active, or about to become active. Not half written code for a feature that will be released in five months. For me personally, a clean codebase is way more important than occasionally having a merge conflict or two.