Could there be something inherent in programming itself that causes estimates to be so unreliable? How are you coping with software teams that put too much confidence in estimating code? ►► Know your options! Access my FREE data hub for the top 25 software industry roles, TechRolepedia → jaymeedwards.com/access-techrolepedia/
I got out - after calling time estimates bs, because I had to defend myself over over again for not being fast enough, while running into multiple hidden "gems". Micromanaged / detailed tickets aside - they followed the time tracking vendors guide "how not to do time tracking" to the letter. 🤮
The best metaphor that describes development is "studying" (thanks Steve McConnel). The activity that mostly resembles development is that. 95% (yup, a random percentage I found somewhere in my brain) of the time spent when developing is learning. That is why it is impossible to provide accurate time measurements for any development task.
@@VortechBand in my particular case, they still think they've produced some piece of art; let's just say it was abstract and modern in it's own way... 😂
@@orzelg I did that last Tuesday. I can't fathom how someone can make themselves ask how long it takes to fix an issue when the guy they are talking to just told them that they have no idea how to fix it because he just got the whole project, using an unfamiliar framework running on a server system he's never even touched before, dumped in his lap less than a week ago. A project that he has to keep alive alongside 2 other inherited projects because the leads on those project either left, or was moved onto permanent positions hired out to product owners. Oh lord. I just realized I am the sole developer tasked with maintaining and further develop features for 3 major and 2 minor customers. At least I get to call myself a tech lead I guess. with only 4 years of experience. I feel grossly underpaid and overworked.
Sadly, programmers are to blame for everything, probably because we are usually introverts. As I've gotten older and more experienced, I have better 'luck' at pushing back on the silliness (when it is silly). #6 Stop Estimating is huge when there is not a business need to estimate. When an estimate is 'needed', I ask my mgmt: When do you need it? And let them know I can cut as many corners as it takes to meet any deadline. All the sudden, that 'business need' of an estimate just melts away.
The reason I think why estimates are hardly ever possible to make accurately is because coding can only be planned ahead so much and most often requires a lot of "as you go" problem solving. This increases with task complexity. The amount of complexity you can "fit" in your head at once is limited. I can give an accurate estimate on baking a cake because the complexity of the task is low and I have a clear vision on the entire process. The complexity of a coding task can very quickly become too much and as it increases, the vision of the task becomes less clear and thus harder to give an accurate estimate for because the amount of "as you go" problem solving increases. Another important aspect of the problem is that people with no hands on experience of coding, especially non-tech managers, cannot easily conceptualize this and are often taken aback by attempts at explaining and instead see it as an attempt at avoiding responsibility or general laziness on the part of the developer attempting to explain this reality. For this reason I think that dev teams are usually best managed by experienced devs with good people skills, because of the empathy and understanding stemming from their own experience and ability to communicate this reality to other management actors. Long story short, the traditional concept of "productivity" measured in time/energy put into a task against its yield sure is completely off with regards to coding, in my opinion. Any business hiring devs would avoid much disfunction by adjusting their model accordingly. My two cents as a dev of 20 years. Thank you for reading!
Agree completely. Accepting the fact that we don't write code during estimation, and the correct code is discovered as we actually do the work is the first step for any manager to be successful in this industry.
It's almost impossible for a jr dev to say this. Managers will imply they just suck at their job and jr devs won't have the confidence/experience to know that's a lie.
You can only divide an conquer to a certain resolution, no amount of preparation, talking, estimating and going over everything again will prevent you from problems which will be discovered during the actual implementation. But that's what makes good engineers, they overcome this obstacles without introducing massive amount of technical debt. Completely agree with your statement.
So sometimes our coworkers can shoot us in the foot here. I've experienced juniors and mids on my team who work outside regular work hours (they have PR reviews and such time stamped at 7pm for instance) to give the impression to management that they are "faster", which gives our PM a false idea of our true sprint capacity, and sometimes estimates are lower based on how little "time" they previously spent on a task just because they worked after hours to make the change seem "fast". Ultimately, it's more work for us who work normally because that's more PRs for us to review. And it's a tougher review since the work was rushed through. They haven't figured out that their only real "reward" here is just more work, and if they really need a raise that bad, their free time is better spent on interview prep and job hunting. In the meantime, our estimates suck.
Even worse: Cowboy coders are also often seen as most productive. They reach higher velocity by skipping all important engineering qualities. Especially juniors are often not yet aware of that.
@@vanlepthien6768 I agree, but I don't know how you make someone stop working late of their own free will. And management loves it, obviously. And I can't call it out because it's not really my business. I know I'm moving on in a couple years, though.
@@vanlepthien6768 LOL, dream on. I've not worked in an organisation that didn't force unpaid overtime in 20 years or more. And all those organisations deliberately underestimate projects to be able to give low quotes to customers in order to win bids.
@@vanlepthien6768 - I tell my team exactly that... and make a point of saying that if they are choosing to work silly hours just to appear productive, they are choosing to lower their hourly rate. Conversely, if they tell me they're struggling, I'll a) reduce their expected outcome and b) arrange training on whatever they're struggling with. No-one writes good code when tired.
A problem on managements end too, is they almost never ask you to re-evaluate estimates, they are determined to hold you to the estimate as if you were legally liable for it, rather than trying to figure out if the estimate needs adjusting. They'd rather blame you for the project being late, then on finding out when it'll actually be ready. Good managers ought to be regularly checking in for new estimates so they can better manage client expectations and have a better idea of where things are actually at, especially if they understand they're asking you to usually solve a problem that's never been solved before and there's no real way for you to know exactly what the future holds, but that you're likely to be be able to better predict it the more you work on the problem. But it seems so many would just rather be ignorant of the project's actual status and continue putting their faith in early estimates.
It's because the managers have already made promises to their bosses who've already made guarantees to customers: Development Team-to-Manager: "Based on our initial estimate, we think it might take about 6 months to complete if all goes as expected." Manager-to-Director: "The development team says it will be ready in 6 months." Director-to-Customer: "The product will absolutely be ready to go live in no more than 6 months. Guaranteed. Cross my heart and hope to die."
@@philw3039 if that's what actually happens (and sometimes it does) you're bang on. It doesn't have to be that way though. Customers accept estimated releases without certainty all the time. Look at the gaming industry for example!
I just found your channel and I'm blown away, this is all stuff us 'Veterans' have been trying to say for years. I'm still always shocked when a project goes south for these reasons, and then next project, management and the business repeat the same exact pattern. I'm on one right now with a 6 month development window, but we have not even finished gathering requirements or talking about MVP with the customer. But management has set the date and now we have to work from that... It just never gets better :) Preach it!! :)
What??, are you on my team? How did you know this? - this is my team. No design, requirements or usecase document and date was set and now we are scrambling to meetup
There is a good and very true statement that pops in my mind: "Software changes but people don't" I'd say you have to really make clear that whatever the hell management thought was possible, it isn't and they should undertand that. Worst projects in my career so far were always those where a PM was sitting alone in his office for couple of month, developing this "Master Plan" and how long things should take without ever talking to an engineer before.
Suppose we do without estimates: how do you deal with clients who absolutely want to know up-front how long it is going to take and how much it is going to cost? How do you convince them it is better if you provided a continuous stream of value (e.g. in sprints)? They'd always fear it's going to cost too much. Estimates give them the illusion of control. How do you replace that powerful illusion? Please make a video about this.
I found the best thing that kind of works is to split your tasks into smaller ones and estimate a couple of them. Just the first couple of them. The others cannot really be estimated good enough because they to far into the future and there may be some unforeseen things that can throw off the estimates of those. That way, the customer does have some kind of idea about what needs to be done, and what you think some tasks are going to take. Each time when you have made some progress that is relevant for the customer you could tell them what you’ve done so far. In that way the customer sees some progress. And progress is usually what the customer wants to see. Not all of them. But some. Hope this kind of perspective makes sense a bit? There are definitely more ways to look at it.
You might not be able to work with such people. Software today is not the same as 20 years ago. Businesses and technology itself is magnitudes more complex, making any sort of estimates at most guesses.
If you really wan't to understand why your customers care so much about estimated, start offering them fixed price solutions. Just quote them a price for the work, regardless of how long it takes. You'll soon find out that being able to predict how much something is going to cost you is the difference between running a solid business and going bust. And remember that in the software business margins are generally high enough to provide a cushion allowing you to fail a few times. Your customers probably don't have that luxury. There is a inherent cost risk in software development, and it all boils down to who is carrying the risk. Software companies prefer to move the risk to their customers, and the customers rather leave the risk with the suppliers.
I've worked both sides client and consultant. It is a common practice for consultants to give the estimates that the client is looking for knowing full well it is impossible. Use up the budget and ask for more.
Basically, you have to lie. Give them a list of bullshit tasks with estimates. Then do the real tasks and map them broadly to the bullshit ones. If you can pull that off and deliver on-time… you’ve made it as a developer.
You are forgetting one last tip: when you need to give an estimate, make one and multiple by three. Then defend that estimate from pressure to reduce it. If you give a low estimate, you are setting yourself up for failure as you will only reach the deadline in the best case scenario. Your original estimate is how much work you think it needs under good circumstances and covers that. The second part of your time is spent on the nasty bugs that appear and the third part on the things that need to be done that you didn't know about yet.
even when doing that - the estimate is hours on the task, not as the calendar ticks over.... "about 4hrs" means "4hrs need to be on that task" not off doing everything else...
Estimating also takes up a lot of time, especially when you have whole-team meetings to estimate a whole collection of user stories. We had a feature recently that we estimated in stories for its components across three meetings. When the business found out it would actually take a lot longer than they thought it would, it got deprioritised. That meant that about total 40-50 hours (they were most of several two-hour-long meetings of about 10 people) was spent on estimating work that's now only gone on the backlog and probably won't get picked up in a while.
There is almost no excuse for a 2 hour meeting... I'd never do this especially in such large groups, you will probably lose 50% of the room within the first 20 minutes. Disastrous....
Planning helps though. Unless you're a manager that has no problem spending 1000 development hours to save 10 hours of meetings because their KPI is to reduce time in meetings, so they just cut them at the expensive of any value it adds.
It has been a long time since I was in a project that is not estimated in hours, directly or indirectly through story points. The truth is hours really don’t matter. Because the measure is always blown up to proportions that don’t really take either complexity or effectivity into account. I’d say it is a “feel good” measure, especially in software development - which is supposed to be continuous and not in project form.
My standard answer for "how long will this project take" is one month. It's always one month. Why? That's long enough to get started, realize that there is no way its going to happen, and find a new job.
I know it's a joke but this would be highly unprofessional... I can even imagine that managers have been lied to by engineers for years and never understood what's wrong, why is it wrong, why people leave and why everybody hates the code and nobody wants to touch it. You'd be suprised how much impact you can create with a good dose of honesty and espeically if you have actionable ideas how to improve the project. Bottom line is, we're pretty much responsible for our own mess and even though it's easy to blame management for everything, the truth is that not a manager has written the messy code but another engineer did. Look at it like this, if you would come up with a brilliant idea to cut down OR time and cost by ordering the surgeon to not use rubber gloves and never count the utensils before and after surgery. He would simply refuse to do it. Because his profesionalism will not allow it, his work ethic will not allow it, his understanding of risk vs reward will not allow it and his hippocratic oath will not allow it. Same goes for engineering, management must not tell us how to write our code. If we as engineers decide that every piece of code should be tested (which it should) then management must not force us to drop testing because of deadlines. A certain code quality is always implied and you just do it.
@@PaladinJenkis Because you have never been fired for not being able to accomplish a task that was not possible to do in the given deadline, even though you told them it was not possible. I have. I even found out later through a coworker that they realized after I left it was not possible, and that I did a good job after all. I'm still waiting for them to call me and apologize[1]. [1] For the humor impaired, that is a joke.
@@scottfranco1962 Sorry to hear, that's a tremendously shitty move by the company that hierd you. To be honest, it's better getting fired instead of slowly burning out while trying to meet absurd deadlines in projects where everything is broken to the core but your overlords expect you to magically fix someone elses garbage. Almost killed my passion for software engineering. One of the best skills I've learned is to know when to go.
@@PaladinJenkis It was a situation of a bad boss who had never been an engineer. He drew several complaints from underlings and the group had a healthy turnover. I think that was the third time I had pushed back on schedules as a being unrealistic and the boss gave me an ultimatum about meeting the schedule or being dismissed ("this is your last chance"), so it was not unexpected. He picked an interesting subject to press on. There was a hardware issue, in fact one of several I had successfully gone after, which was why they assigned it to me. Thus the exact cause was unknown (it was a SPI interface to a ram that was being corrupted). I came up with several innovative diagnostic tools to determine what the issue was, but I knew on day one that there was no way that would yield an answer by the given deadline. My take on the situation is the same today as it was then. I said "no" too often to a boss who didn't like to hear the word. I think there is a continuous balance in programming about telling managers the unvarnished truth about what is going on vs. what they want to hear. I got my best feedback from a manager I talked to years after a company we both worked for failed and closed their doors. He said "you have no filter... you state things as they are without any sort of polish". I kinda like that actually.
@@scottfranco1962 Sounds like a very shitty boss to be honest. And the fact they failed speaks for itself. I understand that saying no too ofter could be taken the wrong way but there are times where you simply have to tell the truth because otherwise you will just sink in deeper into expectations you know you can't meet... It's for the best that they let you go, crunchtime is one thing but this to me sounds like a permanent working mode. I hope you're better of now wherever you are. Thanks for sharing!
Here are some reasons upper management has used to require that estimates *must* be given (not saying they are right or wrong): 1. This other team here has delivery deadlines to meet, they are dependent on your work, so you need to give them an estimate. 2. We need to start our marketing campaign on this product, so you need to give us the full roadmap with dates we can commit to. 3. Project management requires CPI and SPI metrics, its in our certification. You need to schedule things, period. 4. How can we even promote people if we cannot compare the time that person took to perform the task against the estimates. 5. Everyone in the company has to commit to work. What makes your software development team special? You think you're above everyone else? 6. I don't care about your arguments. You need to read Book X and Book Y, then you will see how estimating can be done, even in the case of software. ... I could probably go on after 15 years experience with this eternal battle. While every argument holds a kernel of truth, the part that bugs me is the blatant dismissal every time I mention one (or all) of the factors you enumerated in the video.
7) Your client needs a fixed timeframe and price. He probably has a short deadline to present the whole thing in front of a budget committee to acquire funds and internal resources for the project. While the developers still have questions because the requirements are too vague, your sales guys and your management are breathing down your neck to come up with estimates so they can send out a quote ASAP.
@@RedOchsenbein When you do this to a client or a potential client you run the risk that someone else who may even have much less understanding of the customers needs undercuts you. Happened to me several times. And it's no consolation when you hear on the grapevine that the project failed because it run out of budget and the management kicked out the company who undercut you. They may end up burning their money and maybe even committing to salvaging some software-monstrosity and sticking to a huge technical debt. If there is a second shot to make things work, sometimes it takes several years to be able to secure enough funds again.
@@petrisz Well, I don't say its a good practice. But by forcing the team to estimate everything with the reasons given it's what you get. Talking about clients: Todays clients are asking for 'agile' projects with fixed scope and fixed deadlines. How realistic is that? We all need to start to sell the process instead of a 'product'. It's called 'Software Development' and not 'Software Building' for a reason. If a client in todays world does not understand (and does not want to learn) why fixed scope and fixed deadlines are not working, and why a truly agile process is the solution for that, we should not work with the client. Period. When working as a composer I used to say: "If someone gets the job because he does it for free, I should be happy: He'll earn the same I do, but has to work for it."
@@RedOchsenbein I agree with you. Also, I'm happy if you are able to pick & choose clients and projects. I truly am. Unfortunately some have less space for manoeuvre and in order to feed the families of the developers uncomfortable deals must be accepted as well. Some sectors are well behind and have little experience with software development projects. When the client is for example in manufacturing and used to well calculable costs, he is just very far from the mindset needed for agile.
With this, you're effectively hiding the problem. Sprints are not meant to always get finished. It would be nice if they are, but if they're not, then the team needs to see what the problem was. If it turns out that there was too much in the sprint and the team couldn't finish everything in time, after a few such sprints, the velocity is reduced for the next one. If you keep hidden overtime, everyone would have the impression that the sprints are fine just as they are and nothing will change.
@@SuperPranx agree with SuperPranx here bigtime. You need to train management that estimates will not always be met. It is not a commitment, period. If they want the manage projects with fixed, known up-front activities, don't work in software!
why? do you get punished for not working free overtime? if you want to do overtime, cant you ask your team lead for overtime? or, you know, dont work for free and go home?
This comes under the concept of tasks failing successfully. The developer, or frankly any job that's in a position of being overworked needs to be willing to jump on the proverbial grenade of "I worked as long as you paid me to work and was not able to get this done" to create evidence that too much work exists to complete within the time constraints provided. Masking a lack of time behind working extra hours is only going to skew the results and leave management with bad info when it comes time to hiring a new staff or what have you.
I've had situations where there are still plenty of open questions about a task, yet the task needs an estimate and should be started ASAP. It's impossible to properly estimate a task if the task is unclear. Like you said, the UI might be clear, but the business logic and Amy conflicts with the current logic can be tough to tease out. Also, it seems like the business logic on the Backend should be finished a sprint before the Frontend should start work on that same feature. It's pretty tough to complete a full feature with BE and FE in one sprint, because something usually takes longer than expected.
for backend/frontend, you can make a standard for the backend API first. I found this helps with interconnecting components developed by separate people: first thing you do, work asynchronously on a document that determines the API to the best ability you can first (some stuff might change if you find you can't get some data and etc though, but generally rare). Think of it as part of the spec/design
Two solutions there James: 1) instead of estimating complexity/time, you can time-box an investigation to figure out the answers, for example "I'll know more about it in 2 days" is perfectly valid. 2) Backend before FrontEnd simply puts pressure on the team delivering the backend (assuming they're separated). Instead, the design phase should include an API spec agreement, so that the FrontEnd devs will know what to expect, and both can then happen in parallel. Of course, that risks BackEnd implementing slightly differently, but that's solved in the daily standups.
I still have a hard time with separate back end and front end teams. It makes sense in theory, hire specialists and get twice the work done. In reality, I almost never see the API spec/contract between the two just get created for an endpoint and once delivered, never change. As the project progresses, existing contracts between the two teams evolve regularly. Now you're essentially refactoring across two teams. Some of the worst waste of time spent in queues, with nobody able to move forward, happens in this situation. And then you've got the wonderful "hey, well since you're waiting on those changes, just work on this other new thing". Next thing you know - half baked code base.
Some points worth reiterating: - I have little to no idea what issues I might actually face implementing these changes. I might need to involve other domain experts who are too busy already. One seemingly minor issue might take weeks or even months to figure out. - I have never done this work before. It might be easy or it might be really hard. - I don't know if I will have to take on critical bugs, more tasks, or attend long meetings during this project - Lastly, time estimates are ESTIMATES not actual time to completion. I've had these happen to me a few times: - "X Weeks / Months!?!?! that's unacceptable! Upper management isn't going to like it. Come up with an estimate that's more reasonable". Well..., I always add extra padding to cover for unexpected issues. Besides, I'm sure they wanted it all yesterday so they can ship it out today. - "Shouldn't new project B take about the same amount of time as previous project A that you finished earlier since they are similar?" - "Other team members are good at coming up with time estimates. Why can't you?"
When they complain that the estimate is too big and you should lower it to please management, you learn right there that the team you're working with is immature.
re: estimates, I kept looking bad compared to the rest of my team, until I realized that my work almost never need to be revisited because I'd carefully think through the problem before coding. Stories the other team members worked on would always be under revision, discussion, re-revision, and often end up so broken or under-performing that they'd get tossed. My manager was obsessed with Jira and moving the little boxes around instead of everything else that was actually important. I'd often suggest something or point something out, and he'd get annoyed with me before later seeing how I was correct and apologizing, and then would keep doing that pattern no many how many times he'd apologize. The worst was how my estimates would always get wrecked by discovering some bug or inexcusable issue in the other parts I needed to integrate with, and then dealing with all the overhead of getting those problems fixed within our janky anti-agile practices.
I’ve had the exact same thing happen on a team, except my manager never apologized, but kept blaming me for being late on “my tasks”. When I finally convinced him that a lot of my time was spent helping juniors, he told everyone to stop talking to me! I still had to do this central component, though, just with no input.
Doing things properly in the 1st place is really the key for success, even if it looks like taking a bit more time initially. But it will pay off multiple times afterwards. Sadly, it's the problem that if things go well, nobody sees or asks how much time it would have cost if not going well, so you have nothing to compare about. Typical management issue these day's
@@mrx7956 My experience from 12 years of development, has been that the engineers that do the job sloppily, but quickly, are the ones who see the most rapid career advancement, and quick advancement allows them to avoid paying the price for their poor work ethic.
If you're doing contract work, particularly for smaller companies that want a single one-off kind of development project, you have to estimate. Ideally, you can get into a retainer or maybe monthly subscription kind of thing and then you get as much done as you can, but even then they'll want some kind of idea of what they will get at the end of the month. I can't say I blame them for wanting to know.
That's the most sane way to do it. Monthly budgeting. Choose your spend per month, and choose what metrics about the *business* you want to measure from each release to see if you're moving in the right direction. It's amazing how many companies I see measure the cost of development, but they don't measure whether it's actually increasing revenue, or signups, or cutting costs (whatever the business goals are) until the entire project is done. This is software people! We can measure any time.
Brilliant. I've been facing all the problems listed during my career and it's such a relief to know that I'm not the only one who understands the estimation as broken for software engineering - it's really hard to escalate, especially to non-technical managers. I do agree with each word. As for the need of estimation - I think that for any business it's an optimisation problem - to get the maximum outcome from the minimum input so it needs to understand velocity and build business plans. I'd say that estimation would work only in case of standardization of the Software Engineering - when we'll have standards for how to design architecture, write code, build and connect components etc. so we'll have a strong foundation of common tools and processes. Just like in architecting and building houses or skyscrapers. Maybe, SE will naturally evolve to that level with time.
As a business person that has taken up coding as a hobby, a good corollary is estimates for multi year financial models. So much can go wrong or right that we know models are mostly meant to be illustrative/directional. If you tell a business person “estimating how long this will take is like estimating revenue 2 years from now” it might help them understand that it might be a useful exercise but one that should be done relatively quickly. Its impossible to not estimate at all because the business needs to ultimately determine whether the cost will be worth the output, but estimates should be done relatively quickly, should include a range rather than a hard number, and should be understood to be directional only.
This was pure gold. I actually hate giving estimates. But I realize that this is because this is a feature of Agile that is hard to pin down. When you are working in a two-week sprint, and get tasked to do a lift on some technology or process that you have no prior knowledge of, the Agile method of the building actually fails here. Because agile is both design and Implementation lumped into one. Where you would've had a better idea about what you were going to be building if you had a true design step. Where you were able to list out to the best of your ability all of the functional and non-functional requirements, and the design approach and technology you were going to be using. I think the trick may be to estimate smaller tasks, and then go waterfall for the larger tasks. And maybe estimate individual parts of the larger task, after you have a true understanding of the design of the larger feature.
Agreed. Agile development and estimating are just a case of one wanting the best of both worlds. Waterfall projects can be even more wasteful than agile ones, but at least they don’t make ridiculously uninformed predictions.
@@HealthyDev lol. its quite funny in a dark way. And as a mid level engineer, i'm still trying to figure out whats the best way to get around this limitation of my work. Thanks for posting this piece of gold.
I find that so much advice in the software development space is focused on the ought side of the is-ought gap. No matter how much we try, the reality is that there are no agile software teams. There are a ton of teams that have 2-week sprints at the bottom of a waterfall. Yes estimates are ridiculous, but we give them. I'd love to hear more advice on how to deal with how software teams *actually* work, and not a vision of how software teams *ought* to work. I see that you're giving real-world, practical advice and I appreciate that. I don't believe business people can be convinced that we don't need estimates. I think the fundamental paradigms of business and the pressures they have regarding investors force them to (ironically) overvalue output and undervalue outcomes as a way of gauging progress. So they end up focusing on "how long will it take to create X" instead of asking "what have we learned about improving conversion rates?" The fact that business people have to be convinced by engineers to care more about business outcomes is a terrible reflection on the state of things in our industry. But breaking them out of these paradigms requires a change on the level of a revolution IMO.
I saw a different video of yours about "How to write code like a senior developer?" and immediately thought, "These things help for sure. But I suck at estimations as a developer and companies would expect senior developers to correctly estimate the development work, I wonder if it is learnable?" and then clicked on your channel to find that your latest video is on estimating development work :D Thanks a lot for the video! Really helpful.
i work in as a software engineer in a datacenter with customers in the financial sector and my experience is these customers are turning every penny twice before they want to spend something and at this point they want the estimates to calculate the cost of a feature/project and decide wether it will be ordered or not
Yeah that's unfortunate. I'm going to do some future videos to try and shed some light on the assumptions investors make in software and why costing the solution isn't as valuable as they think it is. Stay tuned.
Great observation about most software projects not being truly team oriented. I once worked for a company where a contest was held to reward the developer who could fix the most bugs during a sprint. I took the risk of being fired to tell management what a horrible idea this was, and how it did not foster a good team environment at all.
OMG. This literally just came up recently at my place. Management thought competition was a great idea. Most of the developers were like "we'd rather focus on getting the job done".
I became convinced that the NoEstimates movement was correct several years ago. The problems encountered by software development are such that you cannot possibly predict anything except the simplest work with any kind of accuracy until you've actually done the work. It's a common refrain to push more design left with the thought, "if we just thought about it *really hard*, we could predict the future," but it never pans out, for the reasons that you say in your video. Yet I still work in an environment where estimates are part of the culture. I chip away at it every now and then. I recently posed the formulation to managers: if estimates are part of commitments, then estimates will be padded subconsciously, because people don't like to look incompetent. Then, Parkinson's law takes hold and work expands to meet these padded estimates. Therefore, the act of estimating is causing us to develop more slowly. People -- product managers even -- seem to agree with this, and then carry on demanding estimates. I can only assume at this point that the actual value of estimates is as some kind of performance art that maintains confidence in stakeholders.
My manager literally told me few days ago that "if you want to be a senior developer, you have to unblock yourself". "You should not be asking for help (to the senior engineer on the team)". Also, he added "I estimated this to be ready in 2 days". Refering to a project with a new technology, internal to the company, with vague and obscure documentation, and with a set of requirements that nobody knows if are remotely achievable with that technology. He never asked for my estimate. I am super annoyed... (btw it is a FAANG)
Sounds like a toxic manager, start some interview prep on the side and prepare for the worst, at the very least you won’t be caught off guard when they try to dick you during performance reviews, leave with a smile on your face and a bigger salary bump.
Oh god, I could go on and on about this topic. Been in the industry for a decade and I've never seen a software project finishing by the date that was estimated. Estimating projects is easily my least favorite thing about this job and at times have me questioning my life choices. I'm currently involved in a project that was initially estimated for 4 months. We're currently closing in on the 12th month 🤣 I'd really appreciate more videos on this topic. Because whenever I oppose estimating, the counterargument I get is the same thing you mentioned. "We need to come up with a cost, plan out ahead with resource availability bla bla".
The problem I faced was we would give estimates and the managers or sales would say it needs to be done in a third of the time or less. A company I worked for once got the programmers and artists together to estimate a project and we had to give one. Programmers estimated 9 months, artists estimated 6 months so it'd take 9 month total vp of content was shocked it would take that long so I asked how long do you expect it to take he replied two weeks. That was a two hour meeting of 7 people time I'm excluding the vp of content and then we had a follow up hour meeting with me and the two art managers, where the vp is trying to say if we got more people we could do in a month. We just sat there with our heads in our hands telling him adding people will not bring down the estimates. That was a consistent problem with that company. They didn't end up taking on the project because it would have taken too long, this was at time when they had no projects planned.
I worked once at a company with a team that has multiple micro services, but quite a bit of legacy code. One piece of the legacy code was login, and multiple vulnerabilities came in which was set as priority for the next sprint(it was really bad😞). These issues were so intertwined, but after completing my spike my suggestion was a rewrite of these workflows affected, and when they asked how long that would take I wasn't comfortable giving a concrete answer so I said a sprint. Instead they deemed it too long to spend on writing any legacy code, and instead I was to bandaid the issue which ended up taking twice as long(and introduced a bug). I think management needs to enable developers to do the work they want done, instead of micromanaging the changes devs can make.
great discussion. I was pretty good at estimating some good specifications. My manager would tell me my estimate was too much and would cut the estimate in half. The whole process was dishonest. I started answering requests for estimates with it will take as long as it takes. I had another project where I was able to beat my estimate by 10k by designing a library to reuse common functionality in all the programs. I thought I was going to be a hero for saving 10k in billable hours. Boy was I wrong.
Constantly asked for estimates--the reasoning I get is "so the company knows how much of my time will be dedicated to this instead of other duties." or "so we can plan the rest of the project accordingly". As a "one man team" (architect, developer, admin and support) this is very counter intuitive--not only does it (make me) feel pressured to get the work done AND give a good estimate, it isn't a smart question based on our staffing -- if I'm waist-deep in a piece of logic for a new feature and then get a support call, I'm expected to take the call. This derails my development and it takes me a few minutes to "find my place again" (not including the delay fixing whatever support issue I have). None of my duties are actually freed up, so why do we need to know how much of my time needs to be dedicated to developing? It's directly dependant on all the other stuff I have to do. This has resulted in my estimates always being "give or take a week" -- sometimes the code flows with no interruptions, sometimes I have literal days where I'm in support hell for a different issue and can't make it back to code (IT Software Engineer [Supply Chain/MFG/Warehousing] for a Medium size company)
On committing to less and defining exactly WHAT you commit to, is imperative in big business. If you can refer to documentation signed off on by management that shows you delivered, the conversation stops dead. You are highly advised to use managements indecision and slight chaos to catch up on what you actually needed to do. I find management will happily leave a team 100% idle while 'they' play politics with projects. 'They' give estimtes, 'they' scale the scope and cost. Then, and only then do then hand it to a dev team and PM and ask them "When" it can be done by. They can be out by an order of magnitude. A 30 day contract already signed with the customer, selling the dream, is estimated as a 140 man day project by dev and their PM. That's a hard one to swallow, but it usually isn't dev swallowing that one, it's the sales reps. "Sell the dream". "Manage the disappointment."
This is part of what led me to resign from the consulting agency I worked for over 10 years. Tons of great people in general. But we had a few people in key leadership, only at a few locations but enough to impact me at least, who spent more time keeping a client from firing us after a bad sale then doing it right the first time with realistic expectations - exactly as you said. That got old real fast.
The biggest problem I have with estimates is that, as you have said there are way too many variables that will inevitably adversely affect the accuracy of the original estimate. I always cringe when I have to reluctantly give a ball park time range estimate because management always always interprets this as an estimate they can use. The only problem I have with agile estimates is what I am dealing with right now, because the customer has a deadline what we are doing is perceived as agile but upon closer inspection it feels like we are actually waterfall, scrumfall? It also doesn't help that our "devops" team is completely dysfunctional where they stubbornly refuse to break their own silo to embrace real devops culture of shared responsibility, our estimates go up because this silo is a bottleneck to getting builds into customer's hands.
Yeah agile literally means able to adapt to change. Which means you intend to change. Which means you can't lock yourself into a deadline. It's scrummerfall/waterscrum for sure. You're not the only one! Hang in there, I hope it gets better for you soon!
This is the eighth or so video of yours I’ve watched, and it pains me to know how right you are. I was once a proud and happy programmer. Now, my resume is basically 5 layoffs, and one termination…in that order. Agile, scrum, Kansan, teamwork, story points, estimations, hardly any reproduction steps, and telling QA how to test the software I just fixed or added to all add up to my utter failure. These videos should spark anger in me as a reminder, but you seem like a guy I would’ve enjoyed working with. By the way, that termination was only two months ago, and I can’t even look at a computer. Im that 48 year old, somewhere between junior and senior level developer who wishes he would’ve gone for that BFA.
I loved this video and agree woth most of it (except the part about team vs individual stuff and combining components but I work in BI report development where every report is basically its own island)! If I could convince my leadership of one thing I'd pick eliminating estimates. I have found it to be the source of so much wasted energy and emotion I can't count it. Estimates are always wrong so we either get blamed for taking too long or not bringing in enough work (we do scrum/sprints). I actually think the latter is worse. Estimating (along with the time box of Sprints) encourages overestimate and therefore bringing in less (the lesser evil) which creates a mindset of doing less and not pushing boundaries or taking risks. I personally prefer taking on more than I can handle because I know I will still achieve more than if I don't push myself even if I don't achieve my goal. Shoot for the stars, you might hit the moon kind of thing. Estimating creates continuous battles between the team, product owner, scrum masters (who shouldn't even be involved in this) and leadership that ultimately just makes the developers feel like children and always failing. It's very toxic.
An estimate should NEVER be taken to heart or something that's concrete. An estimate is defined as "roughly calculate or judge the value, number, quantity, or extent of.". Notice the use of "roughly calculate", that's not the same thing as "concretely calculate". Blaming anyone for BAD estimates doesn't get us anywhere. So let's just not bother with that. We can always go back and work out what was done well, and not so well. In my job as a web developer, if I can't get something done in 30 minutes, I have to provide an estimate. I take an estimate and multiply it by 3. Reason being is I never know what I'm going to run into that's going to be a hurdle in getting something done. It's worked for me for years. Thanks for the video Healthy Software Developer, much love
While I generally agree with you, after working at Kodak for 5 years, I got pretty good with estimates... But, most of the time I was doing repeatable things, and even when things were not repeatable, I got pretty good with WAGS (Wild Ass Guess) because of experience. Also, I got exceptionally good at breaking down the requirements into estimateable tasks. Frankly, I have never seen anyone do this since then... My estimates were always in time (hours, days, weeks, etc.). I have tried to use Story Points over the last 10 years, and my sense is Story Points are total bullshit and meaningless. It was a good idea, but we can never find consensus on what they really mean. Project Managers and Developers will never agree on what Story Points mean. My general sense now is that estimation is evil, but sometime necessary. Personally, I love the approach of projects like Eclipse, where you get one release per year, and you get what you get. However, if someone has hard deadlines, I can do it, but explaining how would be a serious essay or book...
From what I can read, you must have used story points wrong. Actually I agree, story points on their own are useless but if you correlate them with the team velocity, you will get a pretty decent time estimate without overburdening your team with estimating time. Let's say your team averages at a velocity of 20SP and your sprint is 2 weeks long, which means that a 10SP story is done within a week and so on. I personally thing Story Points are a godsent, I don't have to abstract the complexity and estimate how many days I will be working on, I simply focus on the complexity which makes estimation much easier (for me ta least). If you found a good way to give reliable time estimates, that's a pretty good skill to have. I simpy suck at this so I am happy to have story points.
What I have seen is mostly that estimates declared as required because - management wants to allocate budgets to features (funnily in my experience the upside is way more volatile then the cost/dev effort but nobody cares about that one as in my eyes due to power positions that volatility is accepted) - to create long roadmaps showing what one planes to do and get stakeholder buy in for it - software dev companies want to sell stuff as fixed price and thus need to be able to give reliable estimates as they can get a higher margin out of that then with time and material and the client wants to remove uncertainty for his cost end (though i have never seen that in case there are severe issues on the cost end for the supplioer the client doesn have to deal with the blow-up) - honestly i believe a lot has to do with management judging teams by delivering what the team said it would do iin the time it said it would take as managemnt has trouble of easily finding other good ways of judging team performance and ultimately that is still a lot of what mgmt does in most iorganizations - not supporting teams but "managing" team or individual "performance" - estimates are required often becuase senior managemnt has personal goals that some project with some feature set goes live until some date.... Also i think it s pretty ridiculous to expect great estimates for sth you never need before with technologies that are new/changed etc and lots of uncertainty. I mean even going from Munich to Frankfurt by car - a journey I travelled a lot of times I cannot give you a great estimate. Could be 3.5 to 5h - i.e. > 30% difference cause of the uncertainty and that is a well defined road, no change of the target, experience of 40x travels, detailed knowledge of used technology (my car)....
As a project manager, I use historical data to determine an estimated time for coding projects whether waterfall or Agile. I then use this information to verify what the developers indicate as reasonable means. I look at the use cases that the developers are required to build using history and personal expertise. Then there is testing and dev fix debt to think about which may be intangible but using errors per line of code could help. Using the development standard especially the secure development standard as a baseline, you can start getting an idea that most stakeholders are clueless as to the mentality behind coding. You can also look at Bayes Theory as a tool. Understanding Psychology is an added benefit. Great video!
@iumpieces did I mention I was a developer? I use the information gathered for duration as a indicator for average time to code, added to the testing requirement, the mean time is always about 25% out. That is why I use other reporting methods to determine the reasons why there is a delay, a junior developer has more leeway than an medium or senior, but even senior developers have issues. When time starts getting short, and when there is little slack, I pull senior and the delayed programmer into a meeting, with his code to share knowledge and expertise. Most PMs, I know don’t do this. I have project managed many different projects, waterfall is not agile, one builds houses and infrastructure using waterfall, why? Quality gates and approvals. Software can be done using waterfall but does not deliver a workable product in the short term. My approach is not perfect as it does not cater for complexities within the use cases or user stories that are not evident, that is why I have the BA rationalize the complexity to determine which level of developer gets it, juniors always get the crappy work with minimal complexity. This is not a construction mindset, it is a balance between stakeholder requirements, cost, effort (time) and quality. Using a System Mindset allows you to determine a functionality drop that improves functionality over time starting with the foundation which the solution architect determines. The brain is an organ, and sometimes it does not want to work, writers block is real, it is critical to understand this. The issue I am finding lately is that senior developers are hesitant in assisting junior developers for whatever personal or professional reason.
Just had this discussion yesterday. I'm a Software Manager and I'm getting pressure to estimate to see what we can commit to deliver to customers by given dates.
I always advise people, if there's any way to do business to your customers without deadlines, do everything in your power to avoid them. It produces better products, a healthier work environment, and prevents customers from being in control of the company's pace of innovation. Unfortunately it's just expected most places. If you have to estimate because you can't get management to see that, you have to estimate. I've been there. Sometimes people just can't understand how software being knowledge work changes the reliability of estimates - no matter how hard we try.
I agree that estimation on a software development, or to be fair, feature development can be really difficult, especially on doing something that's completely new to the developer. In consulting work however, customers would tend to refer to the amount of hours that you've estimated where it then would be justified to pay you. This is also how (so far in my firm) we send out offers to potential customers and how they'd agree with that estimation in the first place. Tbf, I feel like it's the wrong approach all together. Creative work can take 5 min, or 2 whole weeks until it is "done" but it's difficult to convince the customer to derive the value that we're putting in with that 2 weeks work, especially when they just wanna see the business output of our feature development, and have no care about the creative work/trial and error that "brings no value" but it's the only way in coding when mostly it is "as you go" Do you or anyone here see, how we can approach to change the mindset of the customers in general for the future? Or is there a way to mitigate this?
I work for a company that insists on estimates. Their reason is for "billing purposes." But here is what confuses me. I've once estimated a project would take up to 4 weeks. they said it should not take that long and was asked to reduce it. if they already knew how long it should take then why ask me to estimate in the first place?
Red flag. As stated in the video, only the developer doing the work should say how long something takes. If they think it can be done quicker. "That's fine, if you can show me how it can be done quicker I will agree to it. Until then, I'm the one doing the work. This is what it's going to take". Never accept being pressured into someone else's estimate!
latest project I was on - converting some APIs running locally C#/.NET/Sql and using rabbitMQ to Azure functions using topic/subscription/CosmosDB/Data lakes model - "shouldn't take you guys long, it's mostly copy pasting the existing code into the Azure function right?". Even if you know nothing about programming I think seeing how many of those words are different is a clue to it not being super straightforward. Also we were expected to make up for a 3 month delay in the project start and finish by the original date. I quit 3 months in. Deadline was not met nor is it even close to being met. After I left they started realizing maybe I wasn't being 'slow' after all... maybe it's more complex than they thought. I am loving your videos. I am fairly junior and I was beginning to think I just wasn't cut out for this. Hopefully I will have the tools to stand up to unrealistic expectations in my next job!
Yesterday I was doing a fairly 'routine' task, add an excel export button to a report in our old legacy ERP software. I've probably done this close to a dozen times now, so thought it would only take about half an hour to do the work. This instead took me over half the day, because the problem was these forms run off some old ActiveX component that is accessed through a module that is a complete blackbox (and even the docs are missing so even RTFMing wasn't an option). It was only after sitting back with a cup of tea and staring at the report for an age that I noticed there was an extra column on it and got a hunch that I needed to add an extra offset to get the data exporting correctly. It is always silly things like this that derail a project's estimate, they stack up quickly.
After 28 years coding, I follow the rule of 3 and I usually get the estimated time right. This is how I do: 1) brake the project as much parts as possible. 2) make a estimation for each part. 3) multiply the time by 3. I know, people will call you crazy for taking so long to complete the project. In the end this time is not for the thing you see, they are for the thing you don't see, like a random tuesday morning when you got a kernel update and you computer won't start anymore or the pipe of your kitchen brake and you need to wait for the plumber to fix it. Remember one thing, I can get someone working more the 4 liquid hours a day when you have a emergency, but our brain is not made to have more the 4 hours liquid time of full concentration (talking about coding). You may be in front of the computer for 10 hours, it doesn't mean you being productive for 10 hours.... The 3 value give you enough time to plan well, do and test, and redo it if necessary, also I have never see a manager or a client complaining for delivery before the estimated time. The time give the company a estimate about how they need to plan about sales, marketing, hiring people.... finance... a bad estimation can really bankrupt a company. I learned this in Japan. There, when you ask for a service they give you a really bad and long estimation time but deliver before.
Defining a common language between business and dev team, such as BPMN diagrams (or simple flow charts, if that suits you), is now my go-to approach when I see too many "surprises" during development. That's an overall good practice, if you ask me: you can define into one artifact the overall scope, the happy paths and alternative flows, the acceptance criteria, the use cases, etc. Additional information can be added in text when needed and ultimately that will serve as a contract between devs and Product Owner. Other than that, every development team should factor in non functional requirements and their effort, such as testing, code quality parameters, demos, documentation, tooling, etc. Even so, we all know things happen all the time, that's why those are called "estimates", not "deadlines".
I think managers want estimates because software companies usually bill by the hour, like if developers were some kind of labor workers that has hourly shifts. Because of that, I think some companies/clients like to have estimates to know how much money (estimate by the hour because they bill by the hour) they will have to pay for a specific feature/product. Besides the hourly billing reason, I could think that managers/clients want estimates to know when the feature will be ready, the problem is when they turn the estimate into a deadline.
Great discussion. One thing as a developer that has always bothered me is when PMs and clients try to project a completion date based on software estimates. There never is an account for variables. Thankfully I work at a company that tries to follow the Statement of Work and budget as close as possible. If a client wants an extra feature, then we assess whether it is simple enough to do and if we have money left over.
Here's my experience about mainly doing software for companies that's then used in-house (not re-sold, or re-billed): Usually when asked "how long will this take" I give a range that seems reasonable to me, and state that I have also other work to do, and will have to see how things can be prioritized. People asking these questions usually need to be able to arrange things with other departments, customers, internal communication, etc. -- so need are often happy with a ballpark number that helps them to tell if they need to work on their end this month, this quarter or next year. The trick then comes later: when you realize the time or date you promised isn't realistic for whatever reason, get in touch with the person(s) that are affected by a changing date and let them know ASAP. The earlier you tell them, the better. Add the reasons why it will be later, or ask them if they agree on the priorities of all the tasks/projects you're currently on. Or state the obstacles that are out of your hand. Doing this respects their need to arrange things on their end, and establishes a trustworthy relationship. If it's something that is supposed to be finished in a longer while anyhow, I also like to communicate in between to give an update of the current state of affairs -- even when I'm on track. So far I only made good experiences with that. It's the communication after giving an estimate that allows to manage expectations. If you keep the communication channel open the expectations will gradually approximate reality. An estimate is only a starting point, not a factual reality.
My calendar estimates usually go out of the window when I have to take new tasks that no other team member are willing/able to take. The problem is that if my work would be needed for some additional feature in e.g. 4 weeks, the choice is actually between prioritizing the task that no other member of the team can do and the additional feature that initially scheduled 4 weeks in the future being delayed in addition of my current work getting delayed. And it's often these dependencies that quickly get too complex for management to understand the big picture. I've taken a habit of saying "If I don't get interrupted with other work, this will probably take me 8 work days" to more clearly communicate the risks to the management. I think the bigger problem in my team is that way too often I'm the only member that can effectively work in many parts of the software. Other members would be able to do the work but take 4x the time I would need so they end up avoiding the task and the management may end up agreeing on that.
Estimations are required not only for PM or Stakeholders to know when they can expect to have a deliverables, but they are also a measure implicitly to know if everyone understand what they are doing. depend on the type of project as you mentioned a unique integration, we can have a preliminary implementation to test an idea VS a huge complicated integrated SW with multiple components which at the end leads to not implement the idea due to some other issues. it is also depend on the way of working, I have experience with Kanban and Scrum, there are different dependencies which leads to an impediments or blocking other member of team if estimations going off the range. although I agree with you knowledge work and not repeatable work can leads to unreliable estimations, but as a SW expert I also expect an ability of picturing the path they need to take before start doing the work. at the end nice points to share thanks.
Another useful video. Thank you so much for putting this effort into these. It's really helpful to newer software developers like myself navigating work issues
I am just crawling out of another estimated simple just-add-this-one-thing project. I could've used your articulation in working with the client. I am putting this somewhere in the early stages of any future project. Thanks so much.
I work for a software contracting firm. My client insists on an output driven contractual model, meaning each User Story will get assigned a price based on its estimation before work on it can begin. Basically each Story is treated as its own little fixed-price project. As long as the customer wants this contractual model, I don't see any way around using estimates (and at least retroactively, connecting story points to hours worked).
I’ve experienced this. I had a coworker trying to do agile in a consulting context and this is what he came up with. I explained to him why it didn’t make sense but he decided to try it anyway. Luckily I didn’t have to work on that project. I’m not sure how much you can do in this situation once you’ve established that way of working. The client probably won’t want to give up illusions of predictability once they’ve “had them”.
Our manager compares this often to a visit to a doctor. You come to a doctor, they tell you "I will take you in in 3 hours", you wait roughly 2 hours (rarely ever exactly, but it's at least close-ish), get taken in, or you get told "sorry, another 30 mins please". And you are fine with that. BUT if you visit a doctor, and they tell you "wait here for some time" and 3 hours later, without any update, they suddenly take you in... Your nerves are already boiling by then, because you were left for 3 hours in uncertainty and by then you probably start wondering, whether you are even going to be taken in or not at all. And you could have spent 2-3 hours by doing something else, going out or whatever. This is also the reasoning why we are using estimates. As a product owner or any business side employee... If you have 100 tasks/stories in backlog, you want to see progress, you want to see them being worked on. You want to be sure your issues are being taken with due seriousness and appropriate priority. You want to make sure there isn't something the team is ignoring or forgetting about. You want transparent and up to date information. You may also have something else to do and want to set up your schedule accordingly to the release date of what you have requested. Getting estimates and due dates helps achieving all that, or rather I should say, (un)successfully leads to having an illusion of having that. The estimates also provide some idea of the costs in manpower, and that can be compared to the estimated added value - being then used to determine whether some other, more valuable features shouldn't be made instead, or whether this feature shouldn't be cancelled before even trying to make it work as it most likely won't ever pay off. For some context, I'd like to add that we cannot really be called agile, although we are running sort of a hybrid. We do collect feedback as we go, but we don't really ensure its empirically measurable at all times (rarely ever really). We reorder the stories in the backlog and change their priorities or even cancel them as we go, and we don't shy away from releasing only core features first, then waiting with their future expansions until we get some life experience from actual production usage of them. We don't use story points though and instead run estimates. We also run several teams where several members may overlap. Which leads to, as I have already suggested, a very uncertain system which produces only a thin illusion of knowing when things are actually going to be finished. We do keep a very big margin for errors in our estimates though, so it's usually fine at the end of the day.
You're very welcome! These videos are definitely meant to be of value to all people involved in delivering software - not just programmers. Unfortunately nobody watches them unless I put "programmer" or "software developer" in the title so the RUclips algorithm picks it up ;).
Dude, i just discovered ur channel, and I love it... Literally every single frustration I have had working with clients you have pointed out here... If you can share some methods of communication for dealing with clients that have lacking technical skills. You know, how to go about communicating necessities in a comprehensible way, without being obnoxious.
About familiar technologies: also plan for running the software for at least 10 years. I've made decisions to avoid otherwise promising open source projects simply because they are maintained by a single human. If anything would happen to that maintainer, our team would need to take over the whole project and if that feature is not the central point for our product, that risk is not worth taking.
Estimates usually become important when the business case is too weak. Potential business value should be big enough to account for uncertainty and risks.
Your wording is much better than mine. I always said that software businesses were driven from the revenue end, and not the cost end. In other words, worry about making enough money to cover whatever costs may come. I didn't have a succinct way of putting it. Thanks for this.
Let me ask for help here....I was frequently asked estimate the time depending on complexity and I would never understand why my estimates were thought to be over estimates. The worst part is my team had notions of time to be spent on each piece as they categorized it based on earlier projects. The problem was our project was unique and I could never understand how they could equate the time for the functionality which was written in completely different method , language and was different in the first place.
It's oversimplified software project management. "It's similar to previous work, and similar in complexity, so it will take a similar amount of time". That sounds great on paper, but is absolutely false. If software development were that straightforward, it would be easy. It's sad when you hear these tropes said over, and over, and over again to the point that most management think they are absolutely normal. You are in the right. Previous experience RARELY translates to equivalent estimates. It's a lack of self awareness and wishful thinking that lead to these types of things on their part.
Many projects involve work that is totally new to the team - and possibly new to the company overall. It's hard to ask someone how long it will take to "figure something out" or "learn something new." How long would it take you to learn Latin? How long would it take you, right now - to knit a quilt? If you've never done each, how would you know?
Hehe totally. Scott Nimrod and I talked about exactly this in my first interview I did with him. Here's a short where we talk about that part (in case you haven't already seen it): ruclips.net/user/shorts8d6d7wjSZt0
You are really growing on me, I must admit my initial reaction to you was a little on the negative side, but the more I have listened to you the more I like you.
Estimating is not that hard, after some time working in same domain/company. 4 simple rules to get it right: 1- Think the time you would take to do it 100% 8h/day than x4 or x5. 2- Don't estimate a value. Estimate a range. For example you think you would take 2 days to do something. Estimate 5-10 days. If you keep getting it right, try to narrow the range, If you are falling behind or ahead on your estimates, increase the window. 3- If you are blocked, time is not counting. If you estimated 2-4 days to do something, and it took 7 days to get a contract/API/Docs and then you took 3 days after that. you are within the estimate. You shouldn't estimate somelse's work, they should estimate their own work. 4- The one doing the work, should be the one to estimate. If managers don't agree with your estimate, leave yours in writing and flag their's as wrong. When their estimate fails and they blame you, show them the written message where you flagged your estimate and their's as wrong. and a bonus one: Don't estimate bugs. It makes zero sense. That was not suppose to be happening. If you knew what needed to be done to fix it, you wouldn't have the bug in the first place. Create a task to investigate bug. and that investigation produces a hotfix it is easy, if the bug is complex and requires lot of work the investigation work generates new ticket(s) to implement fix.
I'm a product owner for a dev team in a startup company and this is a big problem here! We are building a product and the owners want to release it ASAP althought, there are so many problems that are holding up the development process on both businesses and technical sides. Now because of the pressure the company put on the team, the PM decided to forced the team to estimate their tasks for each sprint for a while and that didn't go well. Currently the PM is the one who's doing the estimate without getting back to the devs and it's a complete mess! I think it's a different situation when you're building a product than building a service where you can estimate by subscription duration or a contract not the whole project.
Hey Bisan! Here are some more videos on the channel that may help you explain the pitfalls I've seen threaten many companies that try to build software without experience doing it before (or being developers). There should be some good arguments in here, and considerations you can wrestle with, to help you work through the issues with the owners. Specifically: "Minimum Viable Product: Letting Customers Help You Profit": ( ruclips.net/video/mj1neLc7swA/видео.html ), "An Agile Budget Keeps You From Being a Code Monkey": ( ruclips.net/video/pG4wNLopMZA/видео.html ), "Is Your Software Company A Feature Factory Or A Lean Startup?": ( ruclips.net/video/gnk86UQ0j0I/видео.html ), "How a Business Model Canvas Helps Agile Teams": ( ruclips.net/video/S0VXZWlckIM/видео.html ), "Impact Mapping: What's Your Software Development Worth?": ( ruclips.net/video/oqsFdmvj3hM/видео.html ), and "Spot a Fake Agile Team in Under 7 Minutes!": ( ruclips.net/video/H6GdK-dChtY/видео.html ) are good ones to start with. I've also got playlists in the "Playlists" tab on Agile, Scrum, and Lean that may be of help to you. Lastly, the playlist "Life Cycle of a Software Product" can be helpful for understanding the confusion between startup and enterprise culture, and how that impacts companies at various stages of growth: ( ruclips.net/p/PL32pD389V8xuDgMEjLOIQhb_V4RZAN6Gg ). Hope some of that helps!
Not a developer, not even a worker, but I find your content so interesting that I am adopting your advices in my artistic projects!! Having health issues I have to deal with uncertainty, my performance is impredictable but at the same time I am very ambitious and I want to do a good job no matter what... and I can't figure out the time I will need to finish a task. It's a big challenge!! With Scrum (by the guide and the usual videos, too simplistic) I was still in struggle. Thank you for giving me a more useful and doable way to apply the method!!! 😃
I'm in the automotive industry. The Aspice standard requires estimation from start to finish. I as SW project manager have to do the estimation which of course will not reflect the reality. The main goal of estimation is to show whether you have enough people to deliver in time. You need too show that you have overload in the team and dorisk assessment. Assessors are looking to see estimation and risk management. If this is not done, there is no chance of level 2 Aspice. Hopefully the new standards will integrate agile with scrum because as of now the industry has only a fake scrum estimation where story points become hours.
I think most estimates are actually related to marketing decisions. Marketing often wants to tell about new features in progress that they can start advertising before the release and they way estimates for scheduling the advertising. I personally think it would make more sense to start talking about the features when they are in QA phase but many marketing people seem to think that vaporware is not bad PR. And if you constantly request for estimates and then do scheduled marketing based on those estimates, you'll end up marketing vaporware sooner or later. Another thing that estimates are needed for is management level decisions about prioritizing new features. If management thinks that feature X will generate them $100k per year and Y will generate them $150k per year, they want to know if Y takes 150% of the effort of building X? (Assuming both X and Y would roughly need the whole team.) In many agile real world implementations, nobody does accurate enough acceptance criteria for high level features to make this actually possible but the management and marketing still believes it's possible so they keep asking for estimates.
Manager here, in answer to your request at the end, we need estimates because our customers work through tendering processes. Engineers are asked to estimate because they are the experts and have the best understanding of what needs to be done. This is core to the current business model. That doesn’t mean things can’t change or that we’ve given up trying. I’d love for my teams never to have to estimate again, I spent decades as a software developer before collecting my management lobotomy and I know it’s a big ask. I just think that sometimes development teams don’t realise quite how much change needs to happen for these ideas to be adopted. In our case it’s an entire market sector. Our approach has been to achieve success with one customer first then use that success to demonstrate how things can be better for everyone in the future by adopting more modern development methods. To the commenters below, we’re not all evil y’know. 😀
Thanks for sharing. My goal on the channel isn’t to bash management and it’s frustrating that I do have some viewers that fall into that. And I’ve definitely experienced mismanagement that burned people out and caused failed projects. But I also know most developers aren’t given enough insight to the business so we tend to second guess and complain when we don’t understand why things are the way they are. Appreciate your feedback and it’s always valuable to hear from someone who’s done work in both sides of these issues!
A lot of the articles talking about the latest and greatest thing are completely inorganic. How many people who used a thing and found it useful, took the time to write an article for free? A lot of these articles are written by sales partners and employees that get paid commissions/ consulting hours if the product is adopted.
I can't tell you how many times I estimate a project at (let's say 400 hours - of design, development, testing, documentation, etc.) and that project takes the estimated hours. However, management decides to add on more smaller projects during that time and then expects the 400 hour project to be completed at the already aggressive deadline. I think it really comes down to the fact that most managers do not have any development experience; as an actual developer or project manager for developers. Therefore, these managers have no idea how to estimate how long development will take which leads to poorly designed and executed codebases.
As a single developer one a small project I was never comfortable giving my pm a single number. I always gave him three numbers best case, worst case and most likely. He was able to deduce from those three figures how certain or confident I was about a certain estimate. On a routine task those figures were close, on a novel use case those figures spread more. Averaging over many tasks/estimates he was able to project somewhat accurately to the customer what effort was required. It was a "waterfally" project, but giving a range also helped to communicate that estimates are guesstimates ;)
OMG, found your vidos by accident, but you are so right with what you say. I myself am in a company where all bad things you mentioned are applied by our managers. Sadly, they don't listen to the development workforce, but instead come to the conclusion that product X or technology Y is not profitable or too risky instead of just implementing the teams correctly and let the developers do their job properly. I totally understand that profit is the overall goal of a big company, but even if the workforce shows this to their management but they are all unable to understand that they get more work done with higher quality for less money if they would act differently then canceling whole products because they think they are too risky/expensive/unpredictable or to just increase pressure on why estimates are not met... this is very very frustrating. I really begin loosing hope, because i see a decline of knowledge and common sense mostly in the management level. Products become more buggy, it's bad for the customer and most of the time only short term budget benefits are made. Knowledgebase within a company declines as good developers leaves to other places. I begin to realize, that management attitude these day's is the pure evil and reason for companies going a bad route and maybe lead to the fact that people loose their jobs because of childish and short term management decisions. Caused by the fact that managers tend to know nothing any more about their products these days.
I want to share this with my current team so bad, just finished the planning session hours ago, and ended being frustrated because of the decisions that were made
I'm gonna try to "estimate" an answer before watching: My trick that I assume is in the video somewhere, but just in case, is: Come up with what I think is a pretty reasonable estimate, and then I consider my level of confidence and then just add on more, baseline add 30%, but I'm not afraid to add more time if the task seems particularly hairy and or woolly. PS. I've always disliked the way 'estimate' is used, not just in my dev work but just in life generally, because any kind of project of nearly any complexity (from creative to crafting to housework) is always at best extraordinarily difficult to predict. Creative work is inherently complex which is compounded by the fact that humans are generally incredibly bad at predicting the future, which is what estimates like these functionally are. when asked for estimates, I'll often refer to them as predictions or guesses (or prophecies if i'm feeling particularly geeky) and I'll often include layers of confidence in an estimate, like "I haven't spent as much time in that system so that'll probably take at least two days" or "assuming it goes as similar stuff has gone in the past it should only take 6 hours (as a general rule the only time I'll commit to doing something in one day is if I've already done the actual work requested, or something extremely similar.)
In my current job we have to estimate by hours (I know horrendous), but then the business people take away those estimates to the clients and say, this is how long it's going to take and so we're billing you for those hours. So when we come to do the tasking we create a bunch of little tasks that can be done in parallel and give them slightly inflated estimates so we're not feeling like we're under pressure to stick to those hours all the time. Just the other day I knocked out about "5 hours" worth of tasks in an hour because we had tasked out each individual component of writing an API endpoint.
The act of estimating as a team is a critical opportunity to catch gaps in the expected deliverable or share known solutions. The points are basically useless because of management tyranny, but the discussion about the estimation is everything.
The reason people I've worked for/with want estimates is because they want to be able to 1) estimate how long a project will take so they can schedule other projects and 2) bill customers. However, when the estimates are ultimately wrong, project schedules are ultimately extended and customers are ultimately forced to pay more than the original estimate or the company has to eat those hours if they locked themselves into a particular price. This video was spot on, especially point #3.
If your estimate ever includes anything by any 3rd party vendor, just multiply whatever result you would otherwise estimate by 10. In my experience, 3rd party vendors may suddenly stop even answering you without any warning due some internal problems. It's very rare that you get such transparency from the 3rd party vendor that would know about their future internal problems arising. If you don't have a plan ready not being able to use that 3rd party vendor to implement the feature you need, you either need to decide that if the 3rd party vendor issue may cause the whole feature to fail or you include enough time to fully implement alternative solution. If you use 3rd party open source library, make sure that you include the risk that you become the sole maintainer of the project. I personally think that's lesser risk than trusting closed source "supported" 3rd party library in long run but that's still a risk you should consider.
In my company my upper management is forced to impose estimations on us. Upper management works in waterfall and expects us to implement scrum practices. An example is we want to build out this new feature and we need to give man month estimates on how long it will take to management and then we are expected as scrum teams to break down our tasks in estimation and work in a scrum/agile way, its insanity. I'm part scrum master part developer, but this way of doing things doesn't work I skip the burn down usually because there is no way it actually reflects the work we are doing. Estimation tend to be pretty bogus for us as developers.
Reason 8. Management has already decided what the estimate will be. I've seen this so many times! I'll never forget this time we had to implement a feature for retail stores, and my team was asked for an estimate, and we said it would be about five months and get done in February. The development manager said that's wrong, it will take two months and be done in November... which was important because we never roll out changes between Thanksgiving and Christmas. We went back and forth for a while, but the dev manager gets to be the dev manager, and the "official" estimate was November. Because it had to be. We got the thing done in February like I said. The scope of the problem was what it was.
A big thing I notice is managers and leaders being all "optimistic" when choosing what projects they convince the government or customers to give our firm. Project we have no idea how to estimate. A boss I had a few years back just took on a project with no idea whether we could deliver and it was contingent upon consuming an API from the government in my country that was in the alpha stage. During the project multiple times my code would stop working because the government team changed their API. I never understood as a junior developer (and the only dev in the company) how I was supposed to make forms sending into an API that was continually changing. The right thing to do as I saw it would be to wait until the API was completed. Or we'd have to continually change our code as the API changes. But the boss was like "no lets just keep it going, optimism YEY!"
It's easy to be optimistic when you spend time cheerleading the customer and still get to spend time with your family, while the programmer stays up at late keeping your impossible promises. Sorry you had to go through that, it's complete B.S. I've worked with those types of managers. They sell false hope. I've also worked with managers who are incredibly honest, transparent, and realistic about the complexity and difficulty of this work with their customers. I find the latter always earn more respect from the client and gain a long term relationship. Blind optimism is short-term thinking in professional IT services.
Thanks for putting this on video. Exactly my thoughts for many years in the industry seeing things done wrong. I am just not so good bringing it up but now I can just use the share button to make my point :D
ima send this vid some love. If you are to go full agile and most likely some CI towards some non-prod environment there needs to be a certain level of understanding and maturity between both parties. I have yet to come across that level of maturity in either clients or management. Heck last time I told management that using estimated story points to optimize a scrum sprint is basically a miniature version of waterfall I got reprimanded for not knowing agile or scrum -.-' cool, I'll leave you guys to bickering about estimates and deadlines not being met then. Great video as per usual, I realize it's 2 years old, still relevant though.
Was stuck with a pool-based "project manager" once - had a task of investigating, assessing and mitigating nearly 100 audit issues in a major ERP landscape (HUGE task) with *highly* variable amounts of work required for each. He insisted that I provide a rough estimate for each and that I wouldn't be held to it, etc. Then I see they're trying to force me to those estimates and that the briefing I'd written on why this was impossible was carefully omitted to senior management... This guy spent all his time telling everyone how great he was but for my project he was merely an impediment and time waster. Thankfully he "moved on" to another company who bought his sales pitch. I find the more formulaic they are the more finger-pointy they tend to be and this makes them exponentially less effective.
Imagine this scenario, someone asks you to estimate the task before doing it and how many hours it will be completed. Each week you have to peek which tasks you will do, and if you go off the estimate you won't be paid for the hours, any extra hours will be on you, the stress is high and you can't change or extend the estimate or replace the tasks.
That's literally treating estimates as commitments. Rule #1 in software is work for people who understand the definition of the word "estimate". If a mechanic tells you a repair will be $500, and comes back and says they discovered it's worse than they thought and it's now $2000, NOBODY does 4x the work for free.
@@HealthyDev And the mechanic tells you that the cost is going to increase. That's why I always managed to time left to complete. Managing to estimates is at the best stupid.
To get funding, we submit proposals for what will be done between the start of the next fiscal year and the end of the next. And we have little time to prepare that plan. This year, I'll have 6 days from learning the customer representatives' list of priorities until I have to submit the deliverables for the following year. The "estimates" are wild guesses, but I will still be held to them. At least it's better than last year when I had 2 days.
Thanks for making a video about this. It is so tedious to discuss this with management over and over again. Now I can just post a link to your video where everything is perfectly explained 👌
Im a product mgr. I never ask for estimates because as an ex transformation specialist - i know - we dont produce fluglebinders off an assembly line. its just silly. we give a guestimate with a huge contingency margin and a month of delivery.
Our clients are familiar with waterfall and they’re used to hours, so we have to convert our tasks into hours because that’s what their project management office is used to.
what's also beautiful, is when a client tries to be smart, and instead asks if i can first figure out whether something is possible, before paying to have me build it. it sounds reasonable at first. however, the result is usually that basically almost the whole thing has to be built to get that answer. and even if i record the time spent, it can still becomes a negotiation about how much they are willing to pay, and if the negotiation fails then the time spent is lost. admittedly it this doesn't happen often, but it's frustrating when it does
Businesses always want some form of estimate. So what I do is break bigger projects down into smaller ones then I delve I to the component level detail and give a reasonable estimate. I then add about 25% on to my estimate for each component for mucking around. On top of all this I make the estimate fuzzy. +/- 25% is good, and +/- 50% is reasonable. I then give an estimate window of +/- 25%, this means realistically, I believe something can be delivered by myself or a team with a range of +/- 50%. I would rather under promise and over deliver and it's been working great so far, and if people question why there is a range I just explain much of the same reasons outlined in this video.
I once learned an important lesson about estimates. I started working in a state company that did street cleaning stuff, and at some point I was requested to clean a huge open sewer pipe that was all clogged up with some 5x3 meters of waist-high dirt from heavy rains. The boss asked me and my co-worker how long it would take, and he said about three weeks. When the boss went away he told me this: _"We could probably get this done in under a week, worse case two weeks, but never give the boss your best estimates. You never know what's gonna happen, and if you don't get it done on that time, the boss will look down on you. But if you promise three weeks and get it done in one or two, he'll be pleased with you."_ We got it done in nearly 2 weeks, with the help of a 3rd guy who joined us at some point. There was a freaking tree under the dirt, which was one of those time consuming things we could have never predicted.
Great pitch but if the backlog is to re-platform a website, using your example of the blog review component how do you get customer / user feedback(metrics) when users won’t see the finished feature until the whole project is complete? I guess it depends on what we define as a customer …
I agree. Replatforming projects are some of the most wasteful I’ve worked on over the years. Yeah, let’s try to duplicate a product that already works with no changes. Always better to use it as an opportunity to re-envision the product and deliver something more streamlined and without features people didn’t use as much in the previous one. Doing a incremental rewrite where you keep the legacy product running and start rolling out partial new experiences is even better than a replatform!
In my experience, since acceptance criteria is so often lacking, asking for estimates is same as asking "how much does building a bridge cost". Correct answer might be something along like this: "Somewhere between 5 billion dollars and 1 dollar. If you want better estimate, you have to give more details. Where will the bridge be located? Will it be used by ants or are you going to land fully loaded cargo airplanes on it? If it's used by cars, how many lanes you need and should the estimate include all the ramps in both ends? How long bridge do you need? When do you need it ready? Are you able to pay it fully up front?" As you explained in point 2, there are a lot of variables and many don't have clear answers even if the developer actually tried to acquire the information.
When you said "we had a 3rd party tool that had a memory leak in a specific situation" - well the biggest bug I ever made in production software was when I was using 3rd party tool in a way I assumed it was supposed to be used and that created a thread leak in the 3rd party tool. On the positive note, I now know what's the limit for thread number in windows applications.
I always push for a POC up front where you can explore the problem domain, which then makes estimating a lot more accurate. Convincing management a POC is required isn't always easy though...
Could there be something inherent in programming itself that causes estimates to be so unreliable? How are you coping with software teams that put too much confidence in estimating code?
►► Know your options! Access my FREE data hub for the top 25 software industry roles, TechRolepedia → jaymeedwards.com/access-techrolepedia/
I got out - after calling time estimates bs, because I had to defend myself over over again for not being fast enough, while running into multiple hidden "gems".
Micromanaged / detailed tickets aside - they followed the time tracking vendors guide "how not to do time tracking" to the letter. 🤮
Software is art, not an assembly line.
The best metaphor that describes development is "studying" (thanks Steve McConnel). The activity that mostly resembles development is that. 95% (yup, a random percentage I found somewhere in my brain) of the time spent when developing is learning. That is why it is impossible to provide accurate time measurements for any development task.
@@VortechBand in my particular case, they still think they've produced some piece of art; let's just say it was abstract and modern in it's own way... 😂
@@asap8426 "It's a classy, rustic piece with a framework of abstract ideas, set on a canvas of functional moments and pure methods"
When a product manager asks how long it will take to develop a piece of code, say "About as long as it takes to catch a fish."
The only reasonable answer is: I don't know, give me time to estimate the task.
I say sometimes about range - "from 2 days to 2 months"...
@@orzelg I did that last Tuesday.
I can't fathom how someone can make themselves ask how long it takes to fix an issue when the guy they are talking to just told them that they have no idea how to fix it because he just got the whole project, using an unfamiliar framework running on a server system he's never even touched before, dumped in his lap less than a week ago.
A project that he has to keep alive alongside 2 other inherited projects because the leads on those project either left, or was moved onto permanent positions hired out to product owners.
Oh lord. I just realized I am the sole developer tasked with maintaining and further develop features for 3 major and 2 minor customers.
At least I get to call myself a tech lead I guess. with only 4 years of experience.
I feel grossly underpaid and overworked.
@@pablog5738 I've given estimates of over a hundred hours for doing an estimate.
@@vanlepthien6768 yeah!!! That the correct attitude!!!!
Sadly, programmers are to blame for everything, probably because we are usually introverts.
As I've gotten older and more experienced, I have better 'luck' at pushing back on the silliness (when it is silly).
#6 Stop Estimating is huge when there is not a business need to estimate.
When an estimate is 'needed', I ask my mgmt: When do you need it? And let them know I can cut as many corners as it takes to meet any deadline.
All the sudden, that 'business need' of an estimate just melts away.
I can cut as many corners as necessary to meet any deadline! lol 😂 I’m using that one.
The reason I think why estimates are hardly ever possible to make accurately is because coding can only be planned ahead so much and most often requires a lot of "as you go" problem solving. This increases with task complexity.
The amount of complexity you can "fit" in your head at once is limited. I can give an accurate estimate on baking a cake because the complexity of the task is low and I have a clear vision on the entire process. The complexity of a coding task can very quickly become too much and as it increases, the vision of the task becomes less clear and thus harder to give an accurate estimate for because the amount of "as you go" problem solving increases.
Another important aspect of the problem is that people with no hands on experience of coding, especially non-tech managers, cannot easily conceptualize this and are often taken aback by attempts at explaining and instead see it as an attempt at avoiding responsibility or general laziness on the part of the developer attempting to explain this reality. For this reason I think that dev teams are usually best managed by experienced devs with good people skills, because of the empathy and understanding stemming from their own experience and ability to communicate this reality to other management actors.
Long story short, the traditional concept of "productivity" measured in time/energy put into a task against its yield sure is completely off with regards to coding, in my opinion. Any business hiring devs would avoid much disfunction by adjusting their model accordingly.
My two cents as a dev of 20 years. Thank you for reading!
Agree completely. Accepting the fact that we don't write code during estimation, and the correct code is discovered as we actually do the work is the first step for any manager to be successful in this industry.
Amen!
It's almost impossible for a jr dev to say this. Managers will imply they just suck at their job and jr devs won't have the confidence/experience to know that's a lie.
You can only divide an conquer to a certain resolution, no amount of preparation, talking, estimating and going over everything again will prevent you from problems which will be discovered during the actual implementation. But that's what makes good engineers, they overcome this obstacles without introducing massive amount of technical debt.
Completely agree with your statement.
@@credman It's not much easier for a senior dev to do this in a volatile industry where there's always someone else that can be hired to replace you.
So sometimes our coworkers can shoot us in the foot here. I've experienced juniors and mids on my team who work outside regular work hours (they have PR reviews and such time stamped at 7pm for instance) to give the impression to management that they are "faster", which gives our PM a false idea of our true sprint capacity, and sometimes estimates are lower based on how little "time" they previously spent on a task just because they worked after hours to make the change seem "fast". Ultimately, it's more work for us who work normally because that's more PRs for us to review. And it's a tougher review since the work was rushed through.
They haven't figured out that their only real "reward" here is just more work, and if they really need a raise that bad, their free time is better spent on interview prep and job hunting. In the meantime, our estimates suck.
Even worse: Cowboy coders are also often seen as most productive. They reach higher velocity by skipping all important engineering qualities. Especially juniors are often not yet aware of that.
All hours should be paid, and the company culture should enforce that.
@@vanlepthien6768 I agree, but I don't know how you make someone stop working late of their own free will. And management loves it, obviously. And I can't call it out because it's not really my business. I know I'm moving on in a couple years, though.
@@vanlepthien6768 LOL, dream on.
I've not worked in an organisation that didn't force unpaid overtime in 20 years or more.
And all those organisations deliberately underestimate projects to be able to give low quotes to customers in order to win bids.
@@vanlepthien6768 - I tell my team exactly that... and make a point of saying that if they are choosing to work silly hours just to appear productive, they are choosing to lower their hourly rate.
Conversely, if they tell me they're struggling, I'll a) reduce their expected outcome and b) arrange training on whatever they're struggling with.
No-one writes good code when tired.
A problem on managements end too, is they almost never ask you to re-evaluate estimates, they are determined to hold you to the estimate as if you were legally liable for it, rather than trying to figure out if the estimate needs adjusting. They'd rather blame you for the project being late, then on finding out when it'll actually be ready. Good managers ought to be regularly checking in for new estimates so they can better manage client expectations and have a better idea of where things are actually at, especially if they understand they're asking you to usually solve a problem that's never been solved before and there's no real way for you to know exactly what the future holds, but that you're likely to be be able to better predict it the more you work on the problem. But it seems so many would just rather be ignorant of the project's actual status and continue putting their faith in early estimates.
Yep. This is the classic treating estimates as commitments problem.
It's because the managers have already made promises to their bosses who've already made guarantees to customers:
Development Team-to-Manager: "Based on our initial estimate, we think it might take about 6 months to complete if all goes as expected."
Manager-to-Director: "The development team says it will be ready in 6 months."
Director-to-Customer: "The product will absolutely be ready to go live in no more than 6 months. Guaranteed. Cross my heart and hope to die."
@@philw3039 if that's what actually happens (and sometimes it does) you're bang on. It doesn't have to be that way though. Customers accept estimated releases without certainty all the time. Look at the gaming industry for example!
I just found your channel and I'm blown away, this is all stuff us 'Veterans' have been trying to say for years. I'm still always shocked when a project goes south for these reasons, and then next project, management and the business repeat the same exact pattern. I'm on one right now with a 6 month development window, but we have not even finished gathering requirements or talking about MVP with the customer. But management has set the date and now we have to work from that...
It just never gets better :)
Preach it!! :)
Welcome to the channel! Glad to have more vets on here!
Ah the old “here we go again…” project kickoffs :)
What??, are you on my team? How did you know this? - this is my team.
No design, requirements or usecase document and date was set and now we are scrambling to meetup
There is a good and very true statement that pops in my mind: "Software changes but people don't"
I'd say you have to really make clear that whatever the hell management thought was possible, it isn't and they should undertand that.
Worst projects in my career so far were always those where a PM was sitting alone in his office for couple of month, developing this "Master Plan" and how long things should take without ever talking to an engineer before.
Are you on my project? Because rinse and repeat same pattern.
Suppose we do without estimates: how do you deal with clients who absolutely want to know up-front how long it is going to take and how much it is going to cost? How do you convince them it is better if you provided a continuous stream of value (e.g. in sprints)? They'd always fear it's going to cost too much. Estimates give them the illusion of control. How do you replace that powerful illusion? Please make a video about this.
I found the best thing that kind of works is to split your tasks into smaller ones and estimate a couple of them. Just the first couple of them. The others cannot really be estimated good enough because they to far into the future and there may be some unforeseen things that can throw off the estimates of those. That way, the customer does have some kind of idea about what needs to be done, and what you think some tasks are going to take. Each time when you have made some progress that is relevant for the customer you could tell them what you’ve done so far. In that way the customer sees some progress. And progress is usually what the customer wants to see. Not all of them. But some.
Hope this kind of perspective makes sense a bit? There are definitely more ways to look at it.
You might not be able to work with such people. Software today is not the same as 20 years ago. Businesses and technology itself is magnitudes more complex, making any sort of estimates at most guesses.
If you really wan't to understand why your customers care so much about estimated, start offering them fixed price solutions. Just quote them a price for the work, regardless of how long it takes. You'll soon find out that being able to predict how much something is going to cost you is the difference between running a solid business and going bust. And remember that in the software business margins are generally high enough to provide a cushion allowing you to fail a few times. Your customers probably don't have that luxury.
There is a inherent cost risk in software development, and it all boils down to who is carrying the risk. Software companies prefer to move the risk to their customers, and the customers rather leave the risk with the suppliers.
I've worked both sides client and consultant. It is a common practice for consultants to give the estimates that the client is looking for knowing full well it is impossible. Use up the budget and ask for more.
Basically, you have to lie. Give them a list of bullshit tasks with estimates. Then do the real tasks and map them broadly to the bullshit ones. If you can pull that off and deliver on-time… you’ve made it as a developer.
You are forgetting one last tip: when you need to give an estimate, make one and multiple by three. Then defend that estimate from pressure to reduce it. If you give a low estimate, you are setting yourself up for failure as you will only reach the deadline in the best case scenario.
Your original estimate is how much work you think it needs under good circumstances and covers that. The second part of your time is spent on the nasty bugs that appear and the third part on the things that need to be done that you didn't know about yet.
Thank you. Nice tip
even when doing that - the estimate is hours on the task, not as the calendar ticks over.... "about 4hrs" means "4hrs need to be on that task" not off doing everything else...
@@NicholasOrr exactly this!
Estimating also takes up a lot of time, especially when you have whole-team meetings to estimate a whole collection of user stories. We had a feature recently that we estimated in stories for its components across three meetings. When the business found out it would actually take a lot longer than they thought it would, it got deprioritised. That meant that about total 40-50 hours (they were most of several two-hour-long meetings of about 10 people) was spent on estimating work that's now only gone on the backlog and probably won't get picked up in a while.
There is almost no excuse for a 2 hour meeting... I'd never do this especially in such large groups, you will probably lose 50% of the room within the first 20 minutes.
Disastrous....
Planning helps though. Unless you're a manager that has no problem spending 1000 development hours to save 10 hours of meetings because their KPI is to reduce time in meetings, so they just cut them at the expensive of any value it adds.
It has been a long time since I was in a project that is not estimated in hours, directly or indirectly through story points. The truth is hours really don’t matter. Because the measure is always blown up to proportions that don’t really take either complexity or effectivity into account. I’d say it is a “feel good” measure, especially in software development - which is supposed to be continuous and not in project form.
My standard answer for "how long will this project take" is one month. It's always one month. Why? That's long enough to get started, realize that there is no way its going to happen, and find a new job.
I know it's a joke but this would be highly unprofessional... I can even imagine that managers have been lied to by engineers for years and never understood what's wrong, why is it wrong, why people leave and why everybody hates the code and nobody wants to touch it. You'd be suprised how much impact you can create with a good dose of honesty and espeically if you have actionable ideas how to improve the project.
Bottom line is, we're pretty much responsible for our own mess and even though it's easy to blame management for everything, the truth is that not a manager has written the messy code but another engineer did. Look at it like this, if you would come up with a brilliant idea to cut down OR time and cost by ordering the surgeon to not use rubber gloves and never count the utensils before and after surgery. He would simply refuse to do it. Because his profesionalism will not allow it, his work ethic will not allow it, his understanding of risk vs reward will not allow it and his hippocratic oath will not allow it.
Same goes for engineering, management must not tell us how to write our code. If we as engineers decide that every piece of code should be tested (which it should) then management must not force us to drop testing because of deadlines. A certain code quality is always implied and you just do it.
@@PaladinJenkis Because you have never been fired for not being able to accomplish a task that was not possible to do in the given deadline, even though you told them it was not possible. I have. I even found out later through a coworker that they realized after I left it was not possible, and that I did a good job after all. I'm still waiting for them to call me and apologize[1].
[1] For the humor impaired, that is a joke.
@@scottfranco1962 Sorry to hear, that's a tremendously shitty move by the company that hierd you. To be honest, it's better getting fired instead of slowly burning out while trying to meet absurd deadlines in projects where everything is broken to the core but your overlords expect you to magically fix someone elses garbage.
Almost killed my passion for software engineering.
One of the best skills I've learned is to know when to go.
@@PaladinJenkis It was a situation of a bad boss who had never been an engineer. He drew several complaints from underlings and the group had a healthy turnover. I think that was the third time I had pushed back on schedules as a being unrealistic and the boss gave me an ultimatum about meeting the schedule or being dismissed ("this is your last chance"), so it was not unexpected. He picked an interesting subject to press on. There was a hardware issue, in fact one of several I had successfully gone after, which was why they assigned it to me. Thus the exact cause was unknown (it was a SPI interface to a ram that was being corrupted). I came up with several innovative diagnostic tools to determine what the issue was, but I knew on day one that there was no way that would yield an answer by the given deadline.
My take on the situation is the same today as it was then. I said "no" too often to a boss who didn't like to hear the word.
I think there is a continuous balance in programming about telling managers the unvarnished truth about what is going on vs. what they want to hear. I got my best feedback from a manager I talked to years after a company we both worked for failed and closed their doors. He said "you have no filter... you state things as they are without any sort of polish". I kinda like that actually.
@@scottfranco1962 Sounds like a very shitty boss to be honest. And the fact they failed speaks for itself. I understand that saying no too ofter could be taken the wrong way but there are times where you simply have to tell the truth because otherwise you will just sink in deeper into expectations you know you can't meet... It's for the best that they let you go, crunchtime is one thing but this to me sounds like a permanent working mode.
I hope you're better of now wherever you are.
Thanks for sharing!
Here are some reasons upper management has used to require that estimates *must* be given (not saying they are right or wrong):
1. This other team here has delivery deadlines to meet, they are dependent on your work, so you need to give them an estimate.
2. We need to start our marketing campaign on this product, so you need to give us the full roadmap with dates we can commit to.
3. Project management requires CPI and SPI metrics, its in our certification. You need to schedule things, period.
4. How can we even promote people if we cannot compare the time that person took to perform the task against the estimates.
5. Everyone in the company has to commit to work. What makes your software development team special? You think you're above everyone else?
6. I don't care about your arguments. You need to read Book X and Book Y, then you will see how estimating can be done, even in the case of software.
...
I could probably go on after 15 years experience with this eternal battle.
While every argument holds a kernel of truth, the part that bugs me is the blatant dismissal every time I mention one (or all) of the factors you enumerated in the video.
And in the end they just get teams massively overestimating everything and making sure they don't deliver faster...
7) Your client needs a fixed timeframe and price. He probably has a short deadline to present the whole thing in front of a budget committee to acquire funds and internal resources for the project. While the developers still have questions because the requirements are too vague, your sales guys and your management are breathing down your neck to come up with estimates so they can send out a quote ASAP.
@@RedOchsenbein When you do this to a client or a potential client you run the risk that someone else who may even have much less understanding of the customers needs undercuts you. Happened to me several times. And it's no consolation when you hear on the grapevine that the project failed because it run out of budget and the management kicked out the company who undercut you. They may end up burning their money and maybe even committing to salvaging some software-monstrosity and sticking to a huge technical debt. If there is a second shot to make things work, sometimes it takes several years to be able to secure enough funds again.
@@petrisz Well, I don't say its a good practice. But by forcing the team to estimate everything with the reasons given it's what you get. Talking about clients: Todays clients are asking for 'agile' projects with fixed scope and fixed deadlines. How realistic is that? We all need to start to sell the process instead of a 'product'. It's called 'Software Development' and not 'Software Building' for a reason. If a client in todays world does not understand (and does not want to learn) why fixed scope and fixed deadlines are not working, and why a truly agile process is the solution for that, we should not work with the client. Period.
When working as a composer I used to say: "If someone gets the job because he does it for free, I should be happy: He'll earn the same I do, but has to work for it."
@@RedOchsenbein I agree with you. Also, I'm happy if you are able to pick & choose clients and projects. I truly am. Unfortunately some have less space for manoeuvre and in order to feed the families of the developers uncomfortable deals must be accepted as well. Some sectors are well behind and have little experience with software development projects. When the client is for example in manufacturing and used to well calculable costs, he is just very far from the mindset needed for agile.
In my company, Sprints are treated as two-week deadlines, so I often do free, hidden-overtime as a Java Developer to deliver, because of that.
Did you also have the issue of regulary having deadlines within your sprint? Like "x needs to be deployed till Friday EOB".
With this, you're effectively hiding the problem. Sprints are not meant to always get finished. It would be nice if they are, but if they're not, then the team needs to see what the problem was. If it turns out that there was too much in the sprint and the team couldn't finish everything in time, after a few such sprints, the velocity is reduced for the next one. If you keep hidden overtime, everyone would have the impression that the sprints are fine just as they are and nothing will change.
@@SuperPranx agree with SuperPranx here bigtime. You need to train management that estimates will not always be met. It is not a commitment, period. If they want the manage projects with fixed, known up-front activities, don't work in software!
why? do you get punished for not working free overtime? if you want to do overtime, cant you ask your team lead for overtime? or, you know, dont work for free and go home?
This comes under the concept of tasks failing successfully. The developer, or frankly any job that's in a position of being overworked needs to be willing to jump on the proverbial grenade of "I worked as long as you paid me to work and was not able to get this done" to create evidence that too much work exists to complete within the time constraints provided. Masking a lack of time behind working extra hours is only going to skew the results and leave management with bad info when it comes time to hiring a new staff or what have you.
I've had situations where there are still plenty of open questions about a task, yet the task needs an estimate and should be started ASAP. It's impossible to properly estimate a task if the task is unclear. Like you said, the UI might be clear, but the business logic and Amy conflicts with the current logic can be tough to tease out.
Also, it seems like the business logic on the Backend should be finished a sprint before the Frontend should start work on that same feature. It's pretty tough to complete a full feature with BE and FE in one sprint, because something usually takes longer than expected.
for backend/frontend, you can make a standard for the backend API first. I found this helps with interconnecting components developed by separate people: first thing you do, work asynchronously on a document that determines the API to the best ability you can first (some stuff might change if you find you can't get some data and etc though, but generally rare). Think of it as part of the spec/design
Two solutions there James:
1) instead of estimating complexity/time, you can time-box an investigation to figure out the answers, for example "I'll know more about it in 2 days" is perfectly valid.
2) Backend before FrontEnd simply puts pressure on the team delivering the backend (assuming they're separated). Instead, the design phase should include an API spec agreement, so that the FrontEnd devs will know what to expect, and both can then happen in parallel. Of course, that risks BackEnd implementing slightly differently, but that's solved in the daily standups.
I still have a hard time with separate back end and front end teams. It makes sense in theory, hire specialists and get twice the work done. In reality, I almost never see the API spec/contract between the two just get created for an endpoint and once delivered, never change. As the project progresses, existing contracts between the two teams evolve regularly. Now you're essentially refactoring across two teams. Some of the worst waste of time spent in queues, with nobody able to move forward, happens in this situation. And then you've got the wonderful "hey, well since you're waiting on those changes, just work on this other new thing". Next thing you know - half baked code base.
Some points worth reiterating:
- I have little to no idea what issues I might actually face implementing these changes. I might need to involve other domain experts who are too busy already. One seemingly minor issue might take weeks or even months to figure out.
- I have never done this work before. It might be easy or it might be really hard.
- I don't know if I will have to take on critical bugs, more tasks, or attend long meetings during this project
- Lastly, time estimates are ESTIMATES not actual time to completion.
I've had these happen to me a few times:
- "X Weeks / Months!?!?! that's unacceptable! Upper management isn't going to like it. Come up with an estimate that's more reasonable". Well..., I always add extra padding to cover for unexpected issues. Besides, I'm sure they wanted it all yesterday so they can ship it out today.
- "Shouldn't new project B take about the same amount of time as previous project A that you finished earlier since they are similar?"
- "Other team members are good at coming up with time estimates. Why can't you?"
When they complain that the estimate is too big and you should lower it to please management, you learn right there that the team you're working with is immature.
re: estimates, I kept looking bad compared to the rest of my team, until I realized that my work almost never need to be revisited because I'd carefully think through the problem before coding. Stories the other team members worked on would always be under revision, discussion, re-revision, and often end up so broken or under-performing that they'd get tossed. My manager was obsessed with Jira and moving the little boxes around instead of everything else that was actually important. I'd often suggest something or point something out, and he'd get annoyed with me before later seeing how I was correct and apologizing, and then would keep doing that pattern no many how many times he'd apologize.
The worst was how my estimates would always get wrecked by discovering some bug or inexcusable issue in the other parts I needed to integrate with, and then dealing with all the overhead of getting those problems fixed within our janky anti-agile practices.
I’ve had the exact same thing happen on a team, except my manager never apologized, but kept blaming me for being late on “my tasks”. When I finally convinced him that a lot of my time was spent helping juniors, he told everyone to stop talking to me! I still had to do this central component, though, just with no input.
Doing things properly in the 1st place is really the key for success, even if it looks like taking a bit more time initially. But it will pay off multiple times afterwards.
Sadly, it's the problem that if things go well, nobody sees or asks how much time it would have cost if not going well, so you have nothing to compare about.
Typical management issue these day's
@@mrx7956 My experience from 12 years of development, has been that the engineers that do the job sloppily, but quickly, are the ones who see the most rapid career advancement, and quick advancement allows them to avoid paying the price for their poor work ethic.
it might come to light if management were to track time spent on issues. but good luck trying to convince them to (i haven't been able to)
If you're doing contract work, particularly for smaller companies that want a single one-off kind of development project, you have to estimate.
Ideally, you can get into a retainer or maybe monthly subscription kind of thing and then you get as much done as you can, but even then they'll want some kind of idea of what they will get at the end of the month.
I can't say I blame them for wanting to know.
That's the most sane way to do it. Monthly budgeting. Choose your spend per month, and choose what metrics about the *business* you want to measure from each release to see if you're moving in the right direction. It's amazing how many companies I see measure the cost of development, but they don't measure whether it's actually increasing revenue, or signups, or cutting costs (whatever the business goals are) until the entire project is done. This is software people! We can measure any time.
Brilliant. I've been facing all the problems listed during my career and it's such a relief to know that I'm not the only one who understands the estimation as broken for software engineering - it's really hard to escalate, especially to non-technical managers. I do agree with each word.
As for the need of estimation - I think that for any business it's an optimisation problem - to get the maximum outcome from the minimum input so it needs to understand velocity and build business plans. I'd say that estimation would work only in case of standardization of the Software Engineering - when we'll have standards for how to design architecture, write code, build and connect components etc. so we'll have a strong foundation of common tools and processes. Just like in architecting and building houses or skyscrapers. Maybe, SE will naturally evolve to that level with time.
As a business person that has taken up coding as a hobby, a good corollary is estimates for multi year financial models. So much can go wrong or right that we know models are mostly meant to be illustrative/directional. If you tell a business person “estimating how long this will take is like estimating revenue 2 years from now” it might help them understand that it might be a useful exercise but one that should be done relatively quickly.
Its impossible to not estimate at all because the business needs to ultimately determine whether the cost will be worth the output, but estimates should be done relatively quickly, should include a range rather than a hard number, and should be understood to be directional only.
This was pure gold. I actually hate giving estimates. But I realize that this is because this is a feature of Agile that is hard to pin down. When you are working in a two-week sprint, and get tasked to do a lift on some technology or process that you have no prior knowledge of, the Agile method of the building actually fails here. Because agile is both design and Implementation lumped into one. Where you would've had a better idea about what you were going to be building if you had a true design step. Where you were able to list out to the best of your ability all of the functional and non-functional requirements, and the design approach and technology you were going to be using. I think the trick may be to estimate smaller tasks, and then go waterfall for the larger tasks. And maybe estimate individual parts of the larger task, after you have a true understanding of the design of the larger feature.
Agreed. Agile development and estimating are just a case of one wanting the best of both worlds. Waterfall projects can be even more wasteful than agile ones, but at least they don’t make ridiculously uninformed predictions.
@@HealthyDev lol. its quite funny in a dark way. And as a mid level engineer, i'm still trying to figure out whats the best way to get around this limitation of my work. Thanks for posting this piece of gold.
I find that so much advice in the software development space is focused on the ought side of the is-ought gap. No matter how much we try, the reality is that there are no agile software teams. There are a ton of teams that have 2-week sprints at the bottom of a waterfall. Yes estimates are ridiculous, but we give them. I'd love to hear more advice on how to deal with how software teams *actually* work, and not a vision of how software teams *ought* to work. I see that you're giving real-world, practical advice and I appreciate that.
I don't believe business people can be convinced that we don't need estimates. I think the fundamental paradigms of business and the pressures they have regarding investors force them to (ironically) overvalue output and undervalue outcomes as a way of gauging progress. So they end up focusing on "how long will it take to create X" instead of asking "what have we learned about improving conversion rates?" The fact that business people have to be convinced by engineers to care more about business outcomes is a terrible reflection on the state of things in our industry. But breaking them out of these paradigms requires a change on the level of a revolution IMO.
I saw a different video of yours about "How to write code like a senior developer?" and immediately thought, "These things help for sure. But I suck at estimations as a developer and companies would expect senior developers to correctly estimate the development work, I wonder if it is learnable?" and then clicked on your channel to find that your latest video is on estimating development work :D
Thanks a lot for the video! Really helpful.
i work in as a software engineer in a datacenter with customers in the financial sector
and my experience is these customers are turning every penny twice before they want to spend something
and at this point they want the estimates to calculate the cost of a feature/project and decide wether it will be ordered or not
Yeah that's unfortunate. I'm going to do some future videos to try and shed some light on the assumptions investors make in software and why costing the solution isn't as valuable as they think it is. Stay tuned.
Great observation about most software projects not being truly team oriented. I once worked for a company where a contest was held to reward the developer who could fix the most bugs during a sprint. I took the risk of being fired to tell management what a horrible idea this was, and how it did not foster a good team environment at all.
OMG. This literally just came up recently at my place. Management thought competition was a great idea. Most of the developers were like "we'd rather focus on getting the job done".
I became convinced that the NoEstimates movement was correct several years ago. The problems encountered by software development are such that you cannot possibly predict anything except the simplest work with any kind of accuracy until you've actually done the work. It's a common refrain to push more design left with the thought, "if we just thought about it *really hard*, we could predict the future," but it never pans out, for the reasons that you say in your video.
Yet I still work in an environment where estimates are part of the culture. I chip away at it every now and then. I recently posed the formulation to managers: if estimates are part of commitments, then estimates will be padded subconsciously, because people don't like to look incompetent. Then, Parkinson's law takes hold and work expands to meet these padded estimates. Therefore, the act of estimating is causing us to develop more slowly.
People -- product managers even -- seem to agree with this, and then carry on demanding estimates. I can only assume at this point that the actual value of estimates is as some kind of performance art that maintains confidence in stakeholders.
Great insights. I'll be talking about #noestimates more as the channel progresses.
My manager literally told me few days ago that "if you want to be a senior developer, you have to unblock yourself". "You should not be asking for help (to the senior engineer on the team)". Also, he added "I estimated this to be ready in 2 days". Refering to a project with a new technology, internal to the company, with vague and obscure documentation, and with a set of requirements that nobody knows if are remotely achievable with that technology. He never asked for my estimate.
I am super annoyed... (btw it is a FAANG)
Sounds like a toxic manager, start some interview prep on the side and prepare for the worst, at the very least you won’t be caught off guard when they try to dick you during performance reviews, leave with a smile on your face and a bigger salary bump.
Oh god, I could go on and on about this topic. Been in the industry for a decade and I've never seen a software project finishing by the date that was estimated. Estimating projects is easily my least favorite thing about this job and at times have me questioning my life choices. I'm currently involved in a project that was initially estimated for 4 months. We're currently closing in on the 12th month 🤣
I'd really appreciate more videos on this topic. Because whenever I oppose estimating, the counterargument I get is the same thing you mentioned. "We need to come up with a cost, plan out ahead with resource availability bla bla".
The problem I faced was we would give estimates and the managers or sales would say it needs to be done in a third of the time or less.
A company I worked for once got the programmers and artists together to estimate a project and we had to give one. Programmers estimated 9 months, artists estimated 6 months so it'd take 9 month total vp of content was shocked it would take that long so I asked how long do you expect it to take he replied two weeks. That was a two hour meeting of 7 people time I'm excluding the vp of content and then we had a follow up hour meeting with me and the two art managers, where the vp is trying to say if we got more people we could do in a month. We just sat there with our heads in our hands telling him adding people will not bring down the estimates. That was a consistent problem with that company. They didn't end up taking on the project because it would have taken too long, this was at time when they had no projects planned.
I worked once at a company with a team that has multiple micro services, but quite a bit of legacy code. One piece of the legacy code was login, and multiple vulnerabilities came in which was set as priority for the next sprint(it was really bad😞). These issues were so intertwined, but after completing my spike my suggestion was a rewrite of these workflows affected, and when they asked how long that would take I wasn't comfortable giving a concrete answer so I said a sprint. Instead they deemed it too long to spend on writing any legacy code, and instead I was to bandaid the issue which ended up taking twice as long(and introduced a bug). I think management needs to enable developers to do the work they want done, instead of micromanaging the changes devs can make.
great discussion. I was pretty good at estimating some good specifications. My manager would tell me my estimate was too much and would cut the estimate in half. The whole process was dishonest. I started answering requests for estimates with it will take as long as it takes. I had another project where I was able to beat my estimate by 10k by designing a library to reuse common functionality in all the programs. I thought I was going to be a hero for saving 10k in billable hours. Boy was I wrong.
Constantly asked for estimates--the reasoning I get is "so the company knows how much of my time will be dedicated to this instead of other duties." or "so we can plan the rest of the project accordingly". As a "one man team" (architect, developer, admin and support) this is very counter intuitive--not only does it (make me) feel pressured to get the work done AND give a good estimate, it isn't a smart question based on our staffing -- if I'm waist-deep in a piece of logic for a new feature and then get a support call, I'm expected to take the call. This derails my development and it takes me a few minutes to "find my place again" (not including the delay fixing whatever support issue I have). None of my duties are actually freed up, so why do we need to know how much of my time needs to be dedicated to developing? It's directly dependant on all the other stuff I have to do. This has resulted in my estimates always being "give or take a week" -- sometimes the code flows with no interruptions, sometimes I have literal days where I'm in support hell for a different issue and can't make it back to code (IT Software Engineer [Supply Chain/MFG/Warehousing] for a Medium size company)
On committing to less and defining exactly WHAT you commit to, is imperative in big business. If you can refer to documentation signed off on by management that shows you delivered, the conversation stops dead. You are highly advised to use managements indecision and slight chaos to catch up on what you actually needed to do.
I find management will happily leave a team 100% idle while 'they' play politics with projects. 'They' give estimtes, 'they' scale the scope and cost. Then, and only then do then hand it to a dev team and PM and ask them "When" it can be done by. They can be out by an order of magnitude. A 30 day contract already signed with the customer, selling the dream, is estimated as a 140 man day project by dev and their PM.
That's a hard one to swallow, but it usually isn't dev swallowing that one, it's the sales reps. "Sell the dream". "Manage the disappointment."
This is part of what led me to resign from the consulting agency I worked for over 10 years. Tons of great people in general. But we had a few people in key leadership, only at a few locations but enough to impact me at least, who spent more time keeping a client from firing us after a bad sale then doing it right the first time with realistic expectations - exactly as you said. That got old real fast.
The biggest problem I have with estimates is that, as you have said there are way too many variables that will inevitably adversely affect the accuracy of the original estimate. I always cringe when I have to reluctantly give a ball park time range estimate because management always always interprets this as an estimate they can use.
The only problem I have with agile estimates is what I am dealing with right now, because the customer has a deadline what we are doing is perceived as agile but upon closer inspection it feels like we are actually waterfall, scrumfall?
It also doesn't help that our "devops" team is completely dysfunctional where they stubbornly refuse to break their own silo to embrace real devops culture of shared responsibility, our estimates go up because this silo is a bottleneck to getting builds into customer's hands.
Yeah agile literally means able to adapt to change. Which means you intend to change. Which means you can't lock yourself into a deadline. It's scrummerfall/waterscrum for sure. You're not the only one! Hang in there, I hope it gets better for you soon!
This is the eighth or so video of yours I’ve watched, and it pains me to know how right you are. I was once a proud and happy programmer. Now, my resume is basically 5 layoffs, and one termination…in that order. Agile, scrum, Kansan, teamwork, story points, estimations, hardly any reproduction steps, and telling QA how to test the software I just fixed or added to all add up to my utter failure. These videos should spark anger in me as a reminder, but you seem like a guy I would’ve enjoyed working with. By the way, that termination was only two months ago, and I can’t even look at a computer. Im that 48 year old, somewhere between junior and senior level developer who wishes he would’ve gone for that BFA.
I loved this video and agree woth most of it (except the part about team vs individual stuff and combining components but I work in BI report development where every report is basically its own island)! If I could convince my leadership of one thing I'd pick eliminating estimates. I have found it to be the source of so much wasted energy and emotion I can't count it. Estimates are always wrong so we either get blamed for taking too long or not bringing in enough work (we do scrum/sprints). I actually think the latter is worse. Estimating (along with the time box of Sprints) encourages overestimate and therefore bringing in less (the lesser evil) which creates a mindset of doing less and not pushing boundaries or taking risks. I personally prefer taking on more than I can handle because I know I will still achieve more than if I don't push myself even if I don't achieve my goal. Shoot for the stars, you might hit the moon kind of thing. Estimating creates continuous battles between the team, product owner, scrum masters (who shouldn't even be involved in this) and leadership that ultimately just makes the developers feel like children and always failing. It's very toxic.
Unfortuantely, my estimates are only wild guesses. Multiply my hilariously wrong estimates by 12 weeks of "SAFe" - "agile" commitments, yikes.
An estimate should NEVER be taken to heart or something that's concrete. An estimate is defined as "roughly calculate or judge the value, number, quantity, or extent of.". Notice the use of "roughly calculate", that's not the same thing as "concretely calculate".
Blaming anyone for BAD estimates doesn't get us anywhere. So let's just not bother with that. We can always go back and work out what was done well, and not so well.
In my job as a web developer, if I can't get something done in 30 minutes, I have to provide an estimate. I take an estimate and multiply it by 3. Reason being is I never know what I'm going to run into that's going to be a hurdle in getting something done. It's worked for me for years.
Thanks for the video Healthy Software Developer, much love
While I generally agree with you, after working at Kodak for 5 years, I got pretty good with estimates... But, most of the time I was doing repeatable things, and even when things were not repeatable, I got pretty good with WAGS (Wild Ass Guess) because of experience. Also, I got exceptionally good at breaking down the requirements into estimateable tasks. Frankly, I have never seen anyone do this since then...
My estimates were always in time (hours, days, weeks, etc.). I have tried to use Story Points over the last 10 years, and my sense is Story Points are total bullshit and meaningless. It was a good idea, but we can never find consensus on what they really mean. Project Managers and Developers will never agree on what Story Points mean.
My general sense now is that estimation is evil, but sometime necessary. Personally, I love the approach of projects like Eclipse, where you get one release per year, and you get what you get. However, if someone has hard deadlines, I can do it, but explaining how would be a serious essay or book...
From what I can read, you must have used story points wrong. Actually I agree, story points on their own are useless but if you correlate them with the team velocity, you will get a pretty decent time estimate without overburdening your team with estimating time.
Let's say your team averages at a velocity of 20SP and your sprint is 2 weeks long, which means that a 10SP story is done within a week and so on.
I personally thing Story Points are a godsent, I don't have to abstract the complexity and estimate how many days I will be working on, I simply focus on the complexity which makes estimation much easier (for me ta least).
If you found a good way to give reliable time estimates, that's a pretty good skill to have. I simpy suck at this so I am happy to have story points.
What I have seen is mostly that estimates declared as required because
- management wants to allocate budgets to features (funnily in my experience the upside is way more volatile then the cost/dev effort but nobody cares about that one as in my eyes due to power positions that volatility is accepted)
- to create long roadmaps showing what one planes to do and get stakeholder buy in for it
- software dev companies want to sell stuff as fixed price and thus need to be able to give reliable estimates as they can get a higher margin out of that then with time and material and the client wants to remove uncertainty for his cost end (though i have never seen that in case there are severe issues on the cost end for the supplioer the client doesn have to deal with the blow-up)
- honestly i believe a lot has to do with management judging teams by delivering what the team said it would do iin the time it said it would take as managemnt has trouble of easily finding other good ways of judging team performance and ultimately that is still a lot of what mgmt does in most iorganizations - not supporting teams but "managing" team or individual "performance"
- estimates are required often becuase senior managemnt has personal goals that some project with some feature set goes live until some date....
Also i think it s pretty ridiculous to expect great estimates for sth you never need before with technologies that are new/changed etc and lots of uncertainty. I mean even going from Munich to Frankfurt by car - a journey I travelled a lot of times I cannot give you a great estimate. Could be 3.5 to 5h - i.e. > 30% difference cause of the uncertainty and that is a well defined road, no change of the target, experience of 40x travels, detailed knowledge of used technology (my car)....
Thanks Thorsten these are all great points I can address. Most of these will be covered in various future episodes!
As a project manager, I use historical data to determine an estimated time for coding projects whether waterfall or Agile. I then use this information to verify what the developers indicate as reasonable means. I look at the use cases that the developers are required to build using history and personal expertise. Then there is testing and dev fix debt to think about which may be intangible but using errors per line of code could help.
Using the development standard especially the secure development standard as a baseline, you can start getting an idea that most stakeholders are clueless as to the mentality behind coding.
You can also look at Bayes Theory as a tool. Understanding Psychology is an added benefit.
Great video!
@iumpieces did I mention I was a developer? I use the information gathered for duration as a indicator for average time to code, added to the testing requirement, the mean time is always about 25% out. That is why I use other reporting methods to determine the reasons why there is a delay, a junior developer has more leeway than an medium or senior, but even senior developers have issues. When time starts getting short, and when there is little slack, I pull senior and the delayed programmer into a meeting, with his code to share knowledge and expertise. Most PMs, I know don’t do this. I have project managed many different projects, waterfall is not agile, one builds houses and infrastructure using waterfall, why? Quality gates and approvals. Software can be done using waterfall but does not deliver a workable product in the short term.
My approach is not perfect as it does not cater for complexities within the use cases or user stories that are not evident, that is why I have the BA rationalize the complexity to determine which level of developer gets it, juniors always get the crappy work with minimal complexity. This is not a construction mindset, it is a balance between stakeholder requirements, cost, effort (time) and quality. Using a System Mindset allows you to determine a functionality drop that improves functionality over time starting with the foundation which the solution architect determines. The brain is an organ, and sometimes it does not want to work, writers block is real, it is critical to understand this. The issue I am finding lately is that senior developers are hesitant in assisting junior developers for whatever personal or professional reason.
@iumpieces, what is the role of a project manager? Not to micromanage but to deliver the project on time, in budget with quality.
Just had this discussion yesterday. I'm a Software Manager and I'm getting pressure to estimate to see what we can commit to deliver to customers by given dates.
I always advise people, if there's any way to do business to your customers without deadlines, do everything in your power to avoid them. It produces better products, a healthier work environment, and prevents customers from being in control of the company's pace of innovation. Unfortunately it's just expected most places. If you have to estimate because you can't get management to see that, you have to estimate. I've been there. Sometimes people just can't understand how software being knowledge work changes the reliability of estimates - no matter how hard we try.
I agree that estimation on a software development, or to be fair, feature development can be really difficult, especially on doing something that's completely new to the developer. In consulting work however, customers would tend to refer to the amount of hours that you've estimated where it then would be justified to pay you. This is also how (so far in my firm) we send out offers to potential customers and how they'd agree with that estimation in the first place.
Tbf, I feel like it's the wrong approach all together. Creative work can take 5 min, or 2 whole weeks until it is "done" but it's difficult to convince the customer to derive the value that we're putting in with that 2 weeks work, especially when they just wanna see the business output of our feature development, and have no care about the creative work/trial and error that "brings no value" but it's the only way in coding when mostly it is "as you go"
Do you or anyone here see, how we can approach to change the mindset of the customers in general for the future? Or is there a way to mitigate this?
I work for a company that insists on estimates. Their reason is for "billing purposes."
But here is what confuses me. I've once estimated a project would take up to 4 weeks.
they said it should not take that long and was asked to reduce it.
if they already knew how long it should take then why ask me to estimate in the first place?
Red flag. As stated in the video, only the developer doing the work should say how long something takes. If they think it can be done quicker. "That's fine, if you can show me how it can be done quicker I will agree to it. Until then, I'm the one doing the work. This is what it's going to take". Never accept being pressured into someone else's estimate!
Oh yes, out of experience: THAT is how it normally goes
latest project I was on - converting some APIs running locally C#/.NET/Sql and using rabbitMQ to Azure functions using topic/subscription/CosmosDB/Data lakes model - "shouldn't take you guys long, it's mostly copy pasting the existing code into the Azure function right?". Even if you know nothing about programming I think seeing how many of those words are different is a clue to it not being super straightforward. Also we were expected to make up for a 3 month delay in the project start and finish by the original date. I quit 3 months in. Deadline was not met nor is it even close to being met. After I left they started realizing maybe I wasn't being 'slow' after all... maybe it's more complex than they thought. I am loving your videos. I am fairly junior and I was beginning to think I just wasn't cut out for this. Hopefully I will have the tools to stand up to unrealistic expectations in my next job!
Yesterday I was doing a fairly 'routine' task, add an excel export button to a report in our old legacy ERP software.
I've probably done this close to a dozen times now, so thought it would only take about half an hour to do the work.
This instead took me over half the day, because the problem was these forms run off some old ActiveX component that is accessed through a module that is a complete blackbox (and even the docs are missing so even RTFMing wasn't an option).
It was only after sitting back with a cup of tea and staring at the report for an age that I noticed there was an extra column on it and got a hunch that I needed to add an extra offset to get the data exporting correctly.
It is always silly things like this that derail a project's estimate, they stack up quickly.
After 28 years coding, I follow the rule of 3 and I usually get the estimated time right.
This is how I do:
1) brake the project as much parts as possible.
2) make a estimation for each part.
3) multiply the time by 3.
I know, people will call you crazy for taking so long to complete the project. In the end this time is not for the thing you see, they are for the thing you don't see, like a random tuesday morning when you got a kernel update and you computer won't start anymore or the pipe of your kitchen brake and you need to wait for the plumber to fix it.
Remember one thing, I can get someone working more the 4 liquid hours a day when you have a emergency, but our brain is not made to have more the 4 hours liquid time of full concentration (talking about coding). You may be in front of the computer for 10 hours, it doesn't mean you being productive for 10 hours....
The 3 value give you enough time to plan well, do and test, and redo it if necessary, also I have never see a manager or a client complaining for delivery before the estimated time.
The time give the company a estimate about how they need to plan about sales, marketing, hiring people.... finance... a bad estimation can really bankrupt a company.
I learned this in Japan. There, when you ask for a service they give you a really bad and long estimation time but deliver before.
Defining a common language between business and dev team, such as BPMN diagrams (or simple flow charts, if that suits you), is now my go-to approach when I see too many "surprises" during development. That's an overall good practice, if you ask me: you can define into one artifact the overall scope, the happy paths and alternative flows, the acceptance criteria, the use cases, etc. Additional information can be added in text when needed and ultimately that will serve as a contract between devs and Product Owner. Other than that, every development team should factor in non functional requirements and their effort, such as testing, code quality parameters, demos, documentation, tooling, etc. Even so, we all know things happen all the time, that's why those are called "estimates", not "deadlines".
I think managers want estimates because software companies usually bill by the hour, like if developers were some kind of labor workers that has hourly shifts. Because of that, I think some companies/clients like to have estimates to know how much money (estimate by the hour because they bill by the hour) they will have to pay for a specific feature/product.
Besides the hourly billing reason, I could think that managers/clients want estimates to know when the feature will be ready, the problem is when they turn the estimate into a deadline.
Great discussion. One thing as a developer that has always bothered me is when PMs and clients try to project a completion date based on software estimates. There never is an account for variables.
Thankfully I work at a company that tries to follow the Statement of Work and budget as close as possible. If a client wants an extra feature, then we assess whether it is simple enough to do and if we have money left over.
Here's my experience about mainly doing software for companies that's then used in-house (not re-sold, or re-billed):
Usually when asked "how long will this take" I give a range that seems reasonable to me, and state that I have also other work to do, and will have to see how things can be prioritized. People asking these questions usually need to be able to arrange things with other departments, customers, internal communication, etc. -- so need are often happy with a ballpark number that helps them to tell if they need to work on their end this month, this quarter or next year.
The trick then comes later: when you realize the time or date you promised isn't realistic for whatever reason, get in touch with the person(s) that are affected by a changing date and let them know ASAP. The earlier you tell them, the better. Add the reasons why it will be later, or ask them if they agree on the priorities of all the tasks/projects you're currently on. Or state the obstacles that are out of your hand.
Doing this respects their need to arrange things on their end, and establishes a trustworthy relationship.
If it's something that is supposed to be finished in a longer while anyhow, I also like to communicate in between to give an update of the current state of affairs -- even when I'm on track.
So far I only made good experiences with that. It's the communication after giving an estimate that allows to manage expectations. If you keep the communication channel open the expectations will gradually approximate reality. An estimate is only a starting point, not a factual reality.
My calendar estimates usually go out of the window when I have to take new tasks that no other team member are willing/able to take. The problem is that if my work would be needed for some additional feature in e.g. 4 weeks, the choice is actually between prioritizing the task that no other member of the team can do and the additional feature that initially scheduled 4 weeks in the future being delayed in addition of my current work getting delayed.
And it's often these dependencies that quickly get too complex for management to understand the big picture.
I've taken a habit of saying "If I don't get interrupted with other work, this will probably take me 8 work days" to more clearly communicate the risks to the management.
I think the bigger problem in my team is that way too often I'm the only member that can effectively work in many parts of the software. Other members would be able to do the work but take 4x the time I would need so they end up avoiding the task and the management may end up agreeing on that.
Estimations are required not only for PM or Stakeholders to know when they can expect to have a deliverables, but they are also a measure implicitly to know if everyone understand what they are doing. depend on the type of project as you mentioned a unique integration, we can have a preliminary implementation to test an idea VS a huge complicated integrated SW with multiple components which at the end leads to not implement the idea due to some other issues. it is also depend on the way of working, I have experience with Kanban and Scrum, there are different dependencies which leads to an impediments or blocking other member of team if estimations going off the range.
although I agree with you knowledge work and not repeatable work can leads to unreliable estimations, but as a SW expert I also expect an ability of picturing the path they need to take before start doing the work. at the end nice points to share thanks.
Another useful video. Thank you so much for putting this effort into these. It's really helpful to newer software developers like myself navigating work issues
Glad it was helpful!
I am just crawling out of another estimated simple just-add-this-one-thing project. I could've used your articulation in working with the client. I am putting this somewhere in the early stages of any future project.
Thanks so much.
I work for a software contracting firm. My client insists on an output driven contractual model, meaning each User Story will get assigned a price based on its estimation before work on it can begin. Basically each Story is treated as its own little fixed-price project. As long as the customer wants this contractual model, I don't see any way around using estimates (and at least retroactively, connecting story points to hours worked).
I’ve experienced this. I had a coworker trying to do agile in a consulting context and this is what he came up with. I explained to him why it didn’t make sense but he decided to try it anyway. Luckily I didn’t have to work on that project. I’m not sure how much you can do in this situation once you’ve established that way of working. The client probably won’t want to give up illusions of predictability once they’ve “had them”.
Our manager compares this often to a visit to a doctor. You come to a doctor, they tell you "I will take you in in 3 hours", you wait roughly 2 hours (rarely ever exactly, but it's at least close-ish), get taken in, or you get told "sorry, another 30 mins please". And you are fine with that. BUT if you visit a doctor, and they tell you "wait here for some time" and 3 hours later, without any update, they suddenly take you in... Your nerves are already boiling by then, because you were left for 3 hours in uncertainty and by then you probably start wondering, whether you are even going to be taken in or not at all. And you could have spent 2-3 hours by doing something else, going out or whatever.
This is also the reasoning why we are using estimates. As a product owner or any business side employee... If you have 100 tasks/stories in backlog, you want to see progress, you want to see them being worked on. You want to be sure your issues are being taken with due seriousness and appropriate priority. You want to make sure there isn't something the team is ignoring or forgetting about. You want transparent and up to date information. You may also have something else to do and want to set up your schedule accordingly to the release date of what you have requested. Getting estimates and due dates helps achieving all that, or rather I should say, (un)successfully leads to having an illusion of having that. The estimates also provide some idea of the costs in manpower, and that can be compared to the estimated added value - being then used to determine whether some other, more valuable features shouldn't be made instead, or whether this feature shouldn't be cancelled before even trying to make it work as it most likely won't ever pay off.
For some context, I'd like to add that we cannot really be called agile, although we are running sort of a hybrid. We do collect feedback as we go, but we don't really ensure its empirically measurable at all times (rarely ever really). We reorder the stories in the backlog and change their priorities or even cancel them as we go, and we don't shy away from releasing only core features first, then waiting with their future expansions until we get some life experience from actual production usage of them. We don't use story points though and instead run estimates. We also run several teams where several members may overlap. Which leads to, as I have already suggested, a very uncertain system which produces only a thin illusion of knowing when things are actually going to be finished. We do keep a very big margin for errors in our estimates though, so it's usually fine at the end of the day.
I'm a project manager type, and I'm learning a lot from your videos.
You're very welcome! These videos are definitely meant to be of value to all people involved in delivering software - not just programmers. Unfortunately nobody watches them unless I put "programmer" or "software developer" in the title so the RUclips algorithm picks it up ;).
Dude, i just discovered ur channel, and I love it... Literally every single frustration I have had working with clients you have pointed out here... If you can share some methods of communication for dealing with clients that have lacking technical skills.
You know, how to go about communicating necessities in a comprehensible way, without being obnoxious.
About familiar technologies: also plan for running the software for at least 10 years. I've made decisions to avoid otherwise promising open source projects simply because they are maintained by a single human. If anything would happen to that maintainer, our team would need to take over the whole project and if that feature is not the central point for our product, that risk is not worth taking.
Estimates usually become important when the business case is too weak. Potential business value should be big enough to account for uncertainty and risks.
Bingo!
Your wording is much better than mine. I always said that software businesses were driven from the revenue end, and not the cost end. In other words, worry about making enough money to cover whatever costs may come. I didn't have a succinct way of putting it. Thanks for this.
Let me ask for help here....I was frequently asked estimate the time depending on complexity and I would never understand why my estimates were thought to be over estimates. The worst part is my team had notions of time to be spent on each piece as they categorized it based on earlier projects. The problem was our project was unique and I could never understand how they could equate the time for the functionality which was written in completely different method , language and was different in the first place.
It's oversimplified software project management. "It's similar to previous work, and similar in complexity, so it will take a similar amount of time". That sounds great on paper, but is absolutely false. If software development were that straightforward, it would be easy. It's sad when you hear these tropes said over, and over, and over again to the point that most management think they are absolutely normal. You are in the right. Previous experience RARELY translates to equivalent estimates. It's a lack of self awareness and wishful thinking that lead to these types of things on their part.
Many projects involve work that is totally new to the team - and possibly new to the company overall. It's hard to ask someone how long it will take to "figure something out" or "learn something new." How long would it take you to learn Latin? How long would it take you, right now - to knit a quilt? If you've never done each, how would you know?
Hehe totally. Scott Nimrod and I talked about exactly this in my first interview I did with him.
Here's a short where we talk about that part (in case you haven't already seen it): ruclips.net/user/shorts8d6d7wjSZt0
You are really growing on me, I must admit my initial reaction to you was a little on the negative side, but the more I have listened to you the more I like you.
Estimating is not that hard, after some time working in same domain/company.
4 simple rules to get it right:
1- Think the time you would take to do it 100% 8h/day than x4 or x5.
2- Don't estimate a value. Estimate a range. For example you think you would take 2 days to do something. Estimate 5-10 days. If you keep getting it right, try to narrow the range, If you are falling behind or ahead on your estimates, increase the window.
3- If you are blocked, time is not counting. If you estimated 2-4 days to do something, and it took 7 days to get a contract/API/Docs and then you took 3 days after that. you are within the estimate. You shouldn't estimate somelse's work, they should estimate their own work.
4- The one doing the work, should be the one to estimate. If managers don't agree with your estimate, leave yours in writing and flag their's as wrong. When their estimate fails and they blame you, show them the written message where you flagged your estimate and their's as wrong.
and a bonus one:
Don't estimate bugs. It makes zero sense. That was not suppose to be happening. If you knew what needed to be done to fix it, you wouldn't have the bug in the first place. Create a task to investigate bug. and that investigation produces a hotfix it is easy, if the bug is complex and requires lot of work the investigation work generates new ticket(s) to implement fix.
I'm a product owner for a dev team in a startup company and this is a big problem here!
We are building a product and the owners want to release it ASAP althought, there are so many problems that are holding up the development process on both businesses and technical sides.
Now because of the pressure the company put on the team, the PM decided to forced the team to estimate their tasks for each sprint for a while and that didn't go well. Currently the PM is the one who's doing the estimate without getting back to the devs and it's a complete mess!
I think it's a different situation when you're building a product than building a service where you can estimate by subscription duration or a contract not the whole project.
Hey Bisan!
Here are some more videos on the channel that may help you explain the pitfalls I've seen threaten many companies that try to build software without experience doing it before (or being developers). There should be some good arguments in here, and considerations you can wrestle with, to help you work through the issues with the owners.
Specifically:
"Minimum Viable Product: Letting Customers Help You Profit": ( ruclips.net/video/mj1neLc7swA/видео.html ), "An Agile Budget Keeps You From Being a Code Monkey": ( ruclips.net/video/pG4wNLopMZA/видео.html ), "Is Your Software Company A Feature Factory Or A Lean Startup?": ( ruclips.net/video/gnk86UQ0j0I/видео.html ), "How a Business Model Canvas Helps Agile Teams": ( ruclips.net/video/S0VXZWlckIM/видео.html ), "Impact Mapping: What's Your Software Development Worth?": ( ruclips.net/video/oqsFdmvj3hM/видео.html ), and "Spot a Fake Agile Team in Under 7 Minutes!": ( ruclips.net/video/H6GdK-dChtY/видео.html ) are good ones to start with.
I've also got playlists in the "Playlists" tab on Agile, Scrum, and Lean that may be of help to you.
Lastly, the playlist "Life Cycle of a Software Product" can be helpful for understanding the confusion between startup and enterprise culture, and how that impacts companies at various stages of growth: ( ruclips.net/p/PL32pD389V8xuDgMEjLOIQhb_V4RZAN6Gg ).
Hope some of that helps!
Not a developer, not even a worker, but I find your content so interesting that I am adopting your advices in my artistic projects!! Having health issues I have to deal with uncertainty, my performance is impredictable but at the same time I am very ambitious and I want to do a good job no matter what... and I can't figure out the time I will need to finish a task. It's a big challenge!! With Scrum (by the guide and the usual videos, too simplistic) I was still in struggle. Thank you for giving me a more useful and doable way to apply the method!!! 😃
I'm in the automotive industry. The Aspice standard requires estimation from start to finish. I as SW project manager have to do the estimation which of course will not reflect the reality.
The main goal of estimation is to show whether you have enough people to deliver in time. You need too show that you have overload in the team and dorisk assessment.
Assessors are looking to see estimation and risk management. If this is not done, there is no chance of level 2 Aspice.
Hopefully the new standards will integrate agile with scrum because as of now the industry has only a fake scrum estimation where story points become hours.
I think most estimates are actually related to marketing decisions. Marketing often wants to tell about new features in progress that they can start advertising before the release and they way estimates for scheduling the advertising.
I personally think it would make more sense to start talking about the features when they are in QA phase but many marketing people seem to think that vaporware is not bad PR. And if you constantly request for estimates and then do scheduled marketing based on those estimates, you'll end up marketing vaporware sooner or later.
Another thing that estimates are needed for is management level decisions about prioritizing new features. If management thinks that feature X will generate them $100k per year and Y will generate them $150k per year, they want to know if Y takes 150% of the effort of building X? (Assuming both X and Y would roughly need the whole team.)
In many agile real world implementations, nobody does accurate enough acceptance criteria for high level features to make this actually possible but the management and marketing still believes it's possible so they keep asking for estimates.
Manager here, in answer to your request at the end, we need estimates because our customers work through tendering processes. Engineers are asked to estimate because they are the experts and have the best understanding of what needs to be done. This is core to the current business model.
That doesn’t mean things can’t change or that we’ve given up trying. I’d love for my teams never to have to estimate again, I spent decades as a software developer before collecting my management lobotomy and I know it’s a big ask. I just think that sometimes development teams don’t realise quite how much change needs to happen for these ideas to be adopted. In our case it’s an entire market sector.
Our approach has been to achieve success with one customer first then use that success to demonstrate how things can be better for everyone in the future by adopting more modern development methods.
To the commenters below, we’re not all evil y’know. 😀
Thanks for sharing. My goal on the channel isn’t to bash management and it’s frustrating that I do have some viewers that fall into that. And I’ve definitely experienced mismanagement that burned people out and caused failed projects. But I also know most developers aren’t given enough insight to the business so we tend to second guess and complain when we don’t understand why things are the way they are. Appreciate your feedback and it’s always valuable to hear from someone who’s done work in both sides of these issues!
A lot of the articles talking about the latest and greatest thing are completely inorganic. How many people who used a thing and found it useful, took the time to write an article for free? A lot of these articles are written by sales partners and employees that get paid commissions/ consulting hours if the product is adopted.
I can't tell you how many times I estimate a project at (let's say 400 hours - of design, development, testing, documentation, etc.) and that project takes the estimated hours. However, management decides to add on more smaller projects during that time and then expects the 400 hour project to be completed at the already aggressive deadline. I think it really comes down to the fact that most managers do not have any development experience; as an actual developer or project manager for developers. Therefore, these managers have no idea how to estimate how long development will take which leads to poorly designed and executed codebases.
As a single developer one a small project I was never comfortable giving my pm a single number. I always gave him three numbers best case, worst case and most likely. He was able to deduce from those three figures how certain or confident I was about a certain estimate. On a routine task those figures were close, on a novel use case those figures spread more. Averaging over many tasks/estimates he was able to project somewhat accurately to the customer what effort was required. It was a "waterfally" project, but giving a range also helped to communicate that estimates are guesstimates ;)
OMG, found your vidos by accident, but you are so right with what you say.
I myself am in a company where all bad things you mentioned are applied by our managers.
Sadly, they don't listen to the development workforce, but instead come to the conclusion that product X or technology Y is not profitable or too risky instead of just implementing the teams correctly and let the developers do their job properly.
I totally understand that profit is the overall goal of a big company, but even if the workforce shows this to their management but they are all unable to understand that they get more work done with higher quality for less money if they would act differently then canceling whole products because they think they are too risky/expensive/unpredictable or to just increase pressure on why estimates are not met... this is very very frustrating.
I really begin loosing hope, because i see a decline of knowledge and common sense mostly in the management level. Products become more buggy, it's bad for the customer and most of the time only short term budget benefits are made.
Knowledgebase within a company declines as good developers leaves to other places.
I begin to realize, that management attitude these day's is the pure evil and reason for companies going a bad route and maybe lead to the fact that people loose their jobs because of childish and short term management decisions.
Caused by the fact that managers tend to know nothing any more about their products these days.
I want to share this with my current team so bad, just finished the planning session hours ago, and ended being frustrated because of the decisions that were made
I'm gonna try to "estimate" an answer before watching:
My trick that I assume is in the video somewhere, but just in case, is: Come up with what I think is a pretty reasonable estimate, and then I consider my level of confidence and then just add on more, baseline add 30%, but I'm not afraid to add more time if the task seems particularly hairy and or woolly.
PS. I've always disliked the way 'estimate' is used, not just in my dev work but just in life generally, because any kind of project of nearly any complexity (from creative to crafting to housework) is always at best extraordinarily difficult to predict. Creative work is inherently complex which is compounded by the fact that humans are generally incredibly bad at predicting the future, which is what estimates like these functionally are.
when asked for estimates, I'll often refer to them as predictions or guesses (or prophecies if i'm feeling particularly geeky) and I'll often include layers of confidence in an estimate, like "I haven't spent as much time in that system so that'll probably take at least two days" or "assuming it goes as similar stuff has gone in the past it should only take 6 hours (as a general rule the only time I'll commit to doing something in one day is if I've already done the actual work requested, or something extremely similar.)
In my current job we have to estimate by hours (I know horrendous), but then the business people take away those estimates to the clients and say, this is how long it's going to take and so we're billing you for those hours. So when we come to do the tasking we create a bunch of little tasks that can be done in parallel and give them slightly inflated estimates so we're not feeling like we're under pressure to stick to those hours all the time. Just the other day I knocked out about "5 hours" worth of tasks in an hour because we had tasked out each individual component of writing an API endpoint.
The act of estimating as a team is a critical opportunity to catch gaps in the expected deliverable or share known solutions. The points are basically useless because of management tyranny, but the discussion about the estimation is everything.
The reason people I've worked for/with want estimates is because they want to be able to 1) estimate how long a project will take so they can schedule other projects and 2) bill customers. However, when the estimates are ultimately wrong, project schedules are ultimately extended and customers are ultimately forced to pay more than the original estimate or the company has to eat those hours if they locked themselves into a particular price. This video was spot on, especially point #3.
If your estimate ever includes anything by any 3rd party vendor, just multiply whatever result you would otherwise estimate by 10. In my experience, 3rd party vendors may suddenly stop even answering you without any warning due some internal problems. It's very rare that you get such transparency from the 3rd party vendor that would know about their future internal problems arising. If you don't have a plan ready not being able to use that 3rd party vendor to implement the feature you need, you either need to decide that if the 3rd party vendor issue may cause the whole feature to fail or you include enough time to fully implement alternative solution.
If you use 3rd party open source library, make sure that you include the risk that you become the sole maintainer of the project. I personally think that's lesser risk than trusting closed source "supported" 3rd party library in long run but that's still a risk you should consider.
In my company my upper management is forced to impose estimations on us. Upper management works in waterfall and expects us to implement scrum practices. An example is we want to build out this new feature and we need to give man month estimates on how long it will take to management and then we are expected as scrum teams to break down our tasks in estimation and work in a scrum/agile way, its insanity. I'm part scrum master part developer, but this way of doing things doesn't work I skip the burn down usually because there is no way it actually reflects the work we are doing. Estimation tend to be pretty bogus for us as developers.
Reason 8. Management has already decided what the estimate will be.
I've seen this so many times! I'll never forget this time we had to implement a feature for retail stores, and my team was asked for an estimate, and we said it would be about five months and get done in February. The development manager said that's wrong, it will take two months and be done in November... which was important because we never roll out changes between Thanksgiving and Christmas.
We went back and forth for a while, but the dev manager gets to be the dev manager, and the "official" estimate was November. Because it had to be.
We got the thing done in February like I said. The scope of the problem was what it was.
A big thing I notice is managers and leaders being all "optimistic" when choosing what projects they convince the government or customers to give our firm.
Project we have no idea how to estimate.
A boss I had a few years back just took on a project with no idea whether we could deliver and it was contingent upon consuming an API from the government in my country that was in the alpha stage.
During the project multiple times my code would stop working because the government team changed their API.
I never understood as a junior developer (and the only dev in the company) how I was supposed to make forms sending into an API that was continually changing.
The right thing to do as I saw it would be to wait until the API was completed.
Or we'd have to continually change our code as the API changes.
But the boss was like "no lets just keep it going, optimism YEY!"
It's easy to be optimistic when you spend time cheerleading the customer and still get to spend time with your family, while the programmer stays up at late keeping your impossible promises. Sorry you had to go through that, it's complete B.S. I've worked with those types of managers. They sell false hope. I've also worked with managers who are incredibly honest, transparent, and realistic about the complexity and difficulty of this work with their customers. I find the latter always earn more respect from the client and gain a long term relationship. Blind optimism is short-term thinking in professional IT services.
Thanks for putting this on video. Exactly my thoughts for many years in the industry seeing things done wrong. I am just not so good bringing it up but now I can just use the share button to make my point :D
ima send this vid some love. If you are to go full agile and most likely some CI towards some non-prod environment there needs to be a certain level of understanding and maturity between both parties. I have yet to come across that level of maturity in either clients or management. Heck last time I told management that using estimated story points to optimize a scrum sprint is basically a miniature version of waterfall I got reprimanded for not knowing agile or scrum -.-' cool, I'll leave you guys to bickering about estimates and deadlines not being met then.
Great video as per usual, I realize it's 2 years old, still relevant though.
Was stuck with a pool-based "project manager" once - had a task of investigating, assessing and mitigating nearly 100 audit issues in a major ERP landscape (HUGE task) with *highly* variable amounts of work required for each. He insisted that I provide a rough estimate for each and that I wouldn't be held to it, etc. Then I see they're trying to force me to those estimates and that the briefing I'd written on why this was impossible was carefully omitted to senior management... This guy spent all his time telling everyone how great he was but for my project he was merely an impediment and time waster. Thankfully he "moved on" to another company who bought his sales pitch. I find the more formulaic they are the more finger-pointy they tend to be and this makes them exponentially less effective.
Imagine this scenario, someone asks you to estimate the task before doing it and how many hours it will be completed. Each week you have to peek which tasks you will do, and if you go off the estimate you won't be paid for the hours, any extra hours will be on you, the stress is high and you can't change or extend the estimate or replace the tasks.
That's literally treating estimates as commitments. Rule #1 in software is work for people who understand the definition of the word "estimate". If a mechanic tells you a repair will be $500, and comes back and says they discovered it's worse than they thought and it's now $2000, NOBODY does 4x the work for free.
@@HealthyDev And the mechanic tells you that the cost is going to increase. That's why I always managed to time left to complete. Managing to estimates is at the best stupid.
To get funding, we submit proposals for what will be done between the start of the next fiscal year and the end of the next. And we have little time to prepare that plan. This year, I'll have 6 days from learning the customer representatives' list of priorities until I have to submit the deliverables for the following year. The "estimates" are wild guesses, but I will still be held to them. At least it's better than last year when I had 2 days.
Thanks for making a video about this. It is so tedious to discuss this with management over and over again. Now I can just post a link to your video where everything is perfectly explained 👌
Ha! I'm trying...still got a lot more to come.
reason to have estimates, according to our management - to predict a date when requested by a customer feature can be delivered.
Yes, my question would be one level deeper - why do they need to predict the date when a customer feature can be delivered?
Im a product mgr. I never ask for estimates because as an ex transformation specialist - i know - we dont produce fluglebinders off an assembly line. its just silly. we give a guestimate with a huge contingency margin and a month of delivery.
Our clients are familiar with waterfall and they’re used to hours, so we have to convert our tasks into hours because that’s what their project management office is used to.
what's also beautiful, is when a client tries to be smart, and instead asks if i can first figure out whether something is possible, before paying to have me build it. it sounds reasonable at first. however, the result is usually that basically almost the whole thing has to be built to get that answer. and even if i record the time spent, it can still becomes a negotiation about how much they are willing to pay, and if the negotiation fails then the time spent is lost. admittedly it this doesn't happen often, but it's frustrating when it does
Businesses always want some form of estimate. So what I do is break bigger projects down into smaller ones then I delve I to the component level detail and give a reasonable estimate. I then add about 25% on to my estimate for each component for mucking around.
On top of all this I make the estimate fuzzy. +/- 25% is good, and +/- 50% is reasonable.
I then give an estimate window of +/- 25%, this means realistically, I believe something can be delivered by myself or a team with a range of +/- 50%.
I would rather under promise and over deliver and it's been working great so far, and if people question why there is a range I just explain much of the same reasons outlined in this video.
I think the problem with story points is that they look like numbers. We're internally using small, medium, large, x-large as the effort needed.
I once learned an important lesson about estimates. I started working in a state company that did street cleaning stuff, and at some point I was requested to clean a huge open sewer pipe that was all clogged up with some 5x3 meters of waist-high dirt from heavy rains. The boss asked me and my co-worker how long it would take, and he said about three weeks. When the boss went away he told me this: _"We could probably get this done in under a week, worse case two weeks, but never give the boss your best estimates. You never know what's gonna happen, and if you don't get it done on that time, the boss will look down on you. But if you promise three weeks and get it done in one or two, he'll be pleased with you."_
We got it done in nearly 2 weeks, with the help of a 3rd guy who joined us at some point. There was a freaking tree under the dirt, which was one of those time consuming things we could have never predicted.
Great pitch but if the backlog is to re-platform a website, using your example of the blog review component how do you get customer / user feedback(metrics) when users won’t see the finished feature until the whole project is complete? I guess it depends on what we define as a customer …
I agree. Replatforming projects are some of the most wasteful I’ve worked on over the years. Yeah, let’s try to duplicate a product that already works with no changes. Always better to use it as an opportunity to re-envision the product and deliver something more streamlined and without features people didn’t use as much in the previous one. Doing a incremental rewrite where you keep the legacy product running and start rolling out partial new experiences is even better than a replatform!
In my experience, since acceptance criteria is so often lacking, asking for estimates is same as asking "how much does building a bridge cost". Correct answer might be something along like this: "Somewhere between 5 billion dollars and 1 dollar. If you want better estimate, you have to give more details. Where will the bridge be located? Will it be used by ants or are you going to land fully loaded cargo airplanes on it? If it's used by cars, how many lanes you need and should the estimate include all the ramps in both ends? How long bridge do you need? When do you need it ready? Are you able to pay it fully up front?"
As you explained in point 2, there are a lot of variables and many don't have clear answers even if the developer actually tried to acquire the information.
When you said "we had a 3rd party tool that had a memory leak in a specific situation" - well the biggest bug I ever made in production software was when I was using 3rd party tool in a way I assumed it was supposed to be used and that created a thread leak in the 3rd party tool.
On the positive note, I now know what's the limit for thread number in windows applications.
Hey at least you learned something from it! That must have been frustrating.
I always push for a POC up front where you can explore the problem domain, which then makes estimating a lot more accurate. Convincing management a POC is required isn't always easy though...
There's an article floating around that estimates are usually the median, but what really counts is the mean