As the head of DevOps at some of the biggest companies in the world, all I can say is... "Thanks for reaching out, Please raise a ticket through the correct channels for CICD support."
I am in a fortune 200 tier company and its hard to find DevOps to help us with basic OPS shit. I am Dev and we always struggle to get help to lubricate the devops the philosophy of devops was supposed to be to connect dev and ops, but they proceeded with disconnecting dev team from ops team like its nothing
I come from a sys admin, and data center management background, and I have noticed a savage divide between developers and operations. As a DevOps eng, I'm constantly battling dev's requests to manually run code in production; individual user accounts with write access to DBs, or "just give me X resource I need publically available." My background forces me to look at costs, security, and performance, and developers DO NOT care. It is the same thing as it was with on-prem resources, bad code? add more CPU and or mem, app needs to communicate with X, just open the firewall, need a new service no documentation so round and round with requests they didn't know they needed. My solution has been self-service tools but I'm CONSTANTLY catching dev's miss using these tools and putting the company at risk! So I have to pull access and limit what they can do to protect the company. For my WoW players, DevOps is the healer of the raid no one cares until it affects them. Our job is to keep you from destroying yourself and the company.
This is an interesting point. If devs have the tools to provision their own resources, how do you avoid them throwing more resources at a problem without oversight? How do you prevent opening a resource to the entire world “but it works” situations? Someone needs to be auditing these things somehow. Maybe having a security auditing team and a performance auditing team rolling between all internal projects?
I've been in your situation and what I ended up doing was to sit down with management (eventually even the CTO) and explain that there need to be standard hard and fast security practices that nobody deviates from without approval (security team will be your ally on this), approved services with bulk buys to keep costs down and a team budget for deploying anything else, all enforced via the self-service tooling. We then revoked dev write access to all cloud APIs and ran workshops to help the dev teams onboard. Took us 2 years in a company of about 100 engineers. The thing is, we needed a catalyst event (most companies will because you'll be working against entrenched culture). In our case, we had an R&D project that pulled retail customer PII into ElasticSearch but the devs on that project went quick and easy and just deployed the AWS service. The problem was that this was new at the time and not supported in-VPC which meant open to the Internet. Project wound down and they didn't clean up. I come in, look at AWS billing, notice the line item and decide to investigate. Single document on cluster read something to the effect of "we have your data hostage, pay us BTC or we publish it". Well, there goes the quarter, call InfoSec, they hurry over and there's jaws on the floor. Eventually, the GDPR fine hit, CEO and CTO walk onto the engineering floor and declare "the cowboy dev era is over, everything goes through DevOps and InfoSec". Good luck.
I also come as a sysadmin role and the only memorable experience I've had from our devops tooling was that it was constantly deleting or misconfiguring my configuration files and when I spoke to the devops team they tried to explain to me that they have a policy. Their policy was ad-hoc and went both against the documentation and the empirical evidence of how things actually work for that specific system. I showed them both the documentation and the evidence, they didn't fix it. I fixed it with a workaround so it will break again on the next reboot, because I literally had no other option. I had to explain to the customer why the goddamn app randomly stops working as if it's my fault, since I'm responsible for it. And yet I don't have access to the devops tooling that directly affects it and the devops team doesn't respect my input. Devops to me is more like that healer in raids that keeps crying how bad the tank and DPS are, crying how the healer's should never DPS and then misses all the healing timings and gets you killed. I'm not saying all devops is like that, but I feel like rule #1 of big companies is that nobody knows wtf they are doing.
@@russellsanders4535 true, but also takes the reactor 10 seconds to link the article in their description. Helping boost the articles click through and adding text based credit
I'm fully on board that a part of my job is bullshit. Now imagine moving this bullshit from one type of system to another. and then expect anything to work at all.
As a DevOps practitioner I obviously have some opinions on this article - mostly positive, tbh. I totally agree that too many businesses treat roles like SRE/so called 'DevOps teams' as an excuse to palm off responsibilities that should be shared. Also fully support the call for more platform engineering in place of a bajillion teams with a bajillion different cloud environments, all managed with whatever hand-rolled solution that team came up with. The reality is we've known about, and tried to steer away from, these problems within the DevOps community for a while now. A big part of the problem is the fact that, if you get into DevOps through a bootcamp, the only experience you get of implementing these engineering practices comes in the form of individual/small group projects, where the inherent problem in trying to coordinate these practices between tens or hundreds of development teams remains hidden. We really need to do more to expose trainee practitioners to problems of scale early on - DevOps is, after all, a philosophy for improving organisational performance, and a bootcamp simply can't do that justice.
This is also an issue in software engineering. There is no way to simulate scale-like work without actually working in an environment that has big data. And there isn’t really anyone in CS trying to get an internship in DevOps because from what I’ve seen, it’s treated as a pivot for an engineer who doesn’t want to code as much. So the only options are: Hire more juniors and train them accordingly to take over once someone leaves Continue as we are now
@@jonathanalonso6492 I think we need to increase the 'jerk' on the 'acceleration' of our 'velocity' otherwise we just won't reach the 'position' we aspire to 😂
Someone (boogibugs) asked a great question in the Twitch chat: "Why is the operations team not classed as an engineering team? That's one of the issues right there." Boogibugs is correct. Just look at Star Trek. Engineering and operations always wore the same division colors. #justsayin
DevOps was never intended to be a role. It's a concept. Have your Ops integrated into the development process.. influence the product architecture from the design stage. Not a place where you show up on Friday saying "I need this whole new thing deployed on Monday." Then go to the nearest member of management and cry you didn't deliver on time because those "damn devOps are blocking you". DevOps is a concept of enablement, not a role of blockage.
Compare a java dev w/1-2 years of experience to one with 10+. Pretty drastic difference, no? (sql injection? idor? session fixation? txn boundaries?) Same thing applies to 'devops'; k8s isn't just some 'configs', there's a whole ecosystem of best practices, frameworks rising/falling, architecture and so on. That equally applies to tracing/logging, etc The reality is you end up with devs doing stuff in application code that should really be infra, but that's not the teams expertise. They are just copy/pasting the top SO results and effectively `chmod 777`ing your cloud account to get the server to work. Generally, theres no checks/balances (that's a silo, yo!) until you're "hacked".
@@freesoftwareextremist8119 humans are imperfect, it's not about writing code with 0 vulnerabilities, it expecting them to be there and to put a practice in place to catch them and to not trust the human.
I think it is more a byproduct of how at many companies the backlog is owned by product. Product rarely makes time for the types of work that are often considered "devops", but the works needs doing. Eventually there is a break-off group of devs that get to operate a backlog without the oversight of product. They are going to try to get on top of all the "devops" issues that have piled up and therefore turn into a _DevOPS team_. Companies that give engineering more influence over team's backlogs may be more likely to evolve towards platforms, because there is less of a build up of toilsome "devops" work.
Devops != Tech Debt, nor is it a result of tech debt. Devops have their own tech debt, development teams or products have their own. In some cases there is convergence. At the end of the day, someone has to handle (build, maintain, support, scale, secure, cost optimize, observability) infrastructure, build and maintain pipelines, ensure some form of security best practices are being implemented along the way and lastly - I've rarely seen a developer answer compliance questions that didn't only apply to their product, let a lone gather and submit evidence. Lastly, most devs in a large number of orgs are paid to code and deliver on something(s) with in their allotted time (sprint) and it's usually a feature. Rarely they're afforded the time to worry about anything outside of the features they're working on because its not in the business best interest when they want to crank out features. Think v1. V2 is typically when companies start to scale and a redesign is in in the works, well. Devs are now business on the re-arch. Not Devops! That's where we come in.
my guy, all internal politics aside: undelivered product is UNFINISHED product. period. no way you can tell a customer "works on my machine" and call it a day.
@@davidthomson3575 I'd consider "devops" tech debt created by devs to be things like: - our deployment process is someone manually running a script - we are using a dev's IAM keys to give a server access to another service And so many more. It is the brittle and insecure things that happen when corners are being cut that work until they don't.
When publishing videos like this, reacting to articles I mean, if you can share the link of the article in the description section that would be great... just saying... Great videos by the way
Devops != Tech Debt, nor is it a result of tech debt. Devops have their own tech debt, development teams or products have their own. In some cases there is convergence. Granted this can be a group of product devs or a separate team, eventually the former won't scale very long. At the end of the day, someone has to handle (build, maintain, support, scale, secure, cost optimize, observability) infrastructure, build and maintain pipelines, ensure some form of security best practices are being implemented along the way and lastly - I've rarely seen a product developer answer compliance questions that didn't only apply to their product, let a lone gather and submit evidence. Lastly, most devs in a large number of orgs are paid to code and deliver on something(s) within their allotted time (sprint) and it's usually a feature. Rarely they're afforded the time to worry about anything outside of the features they're working on because its not in the business best interest when they want to crank out features. Think v1. V2 is typically when companies start to scale and a redesign/re-arch is in in the works, well. Devs are now busy on the re-arch, not Devops! That's where we come in.
"Devops does not work because the hired devops guy cuts corners and engineers should not work with ops. The solution? Build your own PaaS platform." ??????
Yeah, exactly. Bob, when do you think we can make that change to the frontend I was asking you about. Hey Joe, that will take me about 45 more days because I am busy building out the whole Platform. If you are a 1 man shop or a few dudes doing a startup, I understand. But when you have a scaling organization what you are referring to is not ideal.
@@davidthomson3575 Not only that. Even if you have the resources, you are still just rebuilding an inferior version of already existing AWS Kubernetes/Terraform style hosting services just to virtue signal that you are "doing deployment right".
@@Fernando-ry5qt If easy enough to put behind dashboard buttons then there are likely already tools to automate that process within the same service or with something like github actions. The best button is no button, the best app is no app.
Yeah I agree, worked at a place where everything on infra was repo driven, you just sent a PR and post it on a channel to get a review, the checked if everything was fine, merge, wait a min and done It was pretty good in contrast with the old ticket system I am used to work
I'm "doing devops" in a small company and I like it. Important part is the end-to-end-responsibility, devs learning from ops problems. But it does require management understands "OPS > DEV" and you can't have 100% schedule management. We plan for 2w sprints (~scrumban) but it's just setting priority, what tracks is everyone focusing on. Tickets are done when they are done. Most important part of "my devops" is that once I hit the same thing Nth time, I will either do a bug fix (if I think I know the code cause) or add logging or setup monitoring. In traditional Development this can't happen because the guy looking at logs doesn't understand the source line behind the log line.
In trying to solve waterfall development's "problems" we have created a hydra of problems. Waterfall was simple and could apply to anything. Software was less brittle that way. "Real Agile" is also good, but the common way it's done and this whole DevOps stuff is just a ton of make work for very little actual work that gets done. Most of the time, I'm just left shocked what people call work. It's akin to bureaucracy and paper pushing than actually building stuff.
yeah honestly the amount of filler meetings to discuss work, or the amount of "investigate blah blah blah" tickets where the scope of the ticket is just to figure out how to do the actual work - it's just too much. the team would be much more productive if we cut the filler and just built the shit. sometimes the sheer volume of meetings feels like excuses for people to avoid getting their hands dirty with code
DevOps is a stupid buzzword name, but at least the point of taking the best practices of software development and applying to infrastructure delivery makes some sense. The challenge is mostly a people thing, nobody wants to learn how to do their job over again of let others take over their kingdom, and they resist in the most stubborn ways possible. Technology usually fails at the hands of an inflexible mind.
Exactly, every DevOps person (yeah yeah or whatever title you want) has to not only shoulder the bitch work the Devs were too LAZY to do but then catch a bunch of shit about how “useless” they are at the same time, fun fun
My main problem with DevOps is that I don't even know what exactly are they supposed to even do. Where do their responsibilities start and mine end? Why are they refusing to give me a read only prod db access, when I do really need it for analytical stuff and our app isn't even close to being released yet?
@@JeyPeyy idk what happen to my comment. devops is a production methodology for agile management. devops, right now, work very well for web ressources . Due to tech debt (or aging workforce), the average tech company is based on ITIL management. where each step of the devops cycle is managed by a different team. In those company the job title "devops" can mean a lot of thing (depending on HR). @CHE6yp is talking to an infrastructure/platform admin .
Because read only DB access to a prod database can send the sucker crashing with a single query. This is not a best practice, because we learned long ago no one should do this. You need a data pipelines or atleast a read replica. You write code, you should worry only about writing code. If you want a shit ton of distractions that keep you from writing code, start worrying about all of the non code writing responsibilities that fall under the guise of devops.
@@davidthomson3575 Oh I am so glad you said this! Illustrates my main PITA in modern development perfectly. In general, I would say giving any access to prod is highly dubious and not a "best practice". However, "best practice" is often just a massive distraction in itself when applied to a concrete example. In my case, we have 2 backend devs and our project basically just agregates a bunch of data from a third party. All of that data is completely recoverable, and the amount of users is currently zero. The reason I needed a prod db is because prod is the only place that does have actual data, and my task was to basically mutate some field values to make them all confirm to the same standart. Nobody has a full list of actual values that need mutation, so basically I needed to make a bunch of SELECT DISTINCT queries just to find out what exactly is needed. So, whats the path of least resistance and the best practice? Waiting for our DevOps/Platform Admin to make a replica, going to our prod frontend and manually cataloging all 2 million records, or, getting a prod (not even read-only necessarily) db access?
This is unrelatable to me because our team develops, deploys end to end, writes the CDK infrastructure, handles oncall, etc. I think separate dev ops teams just encourage engineers to be lazy tbh
I used to do that. The problem with generalization of that sort is that you’ll keep tripping over amateur problems as it takes forever to develop expertise in any one area. I love the concept, but in practice it can lead to burnout.
@@samgould8567Yes. I too believe it’s better for every organization, with capacity, to engage individuals who have specialized in solving specific problems, like devops as a practice especially in the sense of ensuring reliability. You can’t have everybody doing everything, it will end in chaos. 😅
So who's managing the cloud then? Presuming you're on AWS, who's managing organization itself? Security, compliance, fin ops? Who's going to be waken up if CloudTrail logs have something suspicions - on-call from full-stack Team A or Team B? Who's gonna call you out on unused resources?
I agree. The same goes for full stack garbage. Civilization is built on specialization and the key to succeeding is the collaboration of many specialists. Imagine this in any other industry: “we’re looking for someone who can work on every single aspect of the building: electrical, heating, plumbing, concrete, etc. We will assign each full stack trades person a room, and they are responsible for the whole thing.” You’d end up with inferior, inconsistent garbage, just like today’s software!
We should all more aspire to be jacks in all trades than masters in just one. It is very important to have overlapping skill sets and at least know what other roles are doing to enable communication and a mindset of shared responsibility and belonging. Unfortunately big corps are usually driven by automated processes and use business units, which separate people into well defined boxes. To say it with the words of Melvin Conway `Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.`
Article just sounded like the guy was complaining about everything. “DevOps done by engineers is bad because the engineers hate it and suck at it.” Then he goes “DevOps done by the DevOps engineers is bad because they are not good at their jobs and just copy paste things” Then his solution is to rename the DevOps engineers to platform engineers and magically everything will be better. The reality is that hard things are hard and random hat swapping is not gonna fix it. This is fundamentally a technology problem. When the tech changes then the situation will change.
The problem is that "dev" is creation, and "ops" is maintaining. Completely opposite perspectives (and cultures). Very few people can do both, let alone well (with the right mindset).
And the very disturbing things is that bad to mediocre DevOps earn on average MORE than good software developer. I didn't believe it until a guy I know, who doesn't even know how to write a for-loop, who worked at RedHat told me his salary and showed me their repos... He was writing yaml files and introducing very lil amount of "code", I think that three of them full time for about a year still didn't produce as much code as I alone do in just a week. Another thing is that most of them are clowns and liked to be called engineers while they can't solve the easiest equation ever.
@FredoCorleone is right, it's common that newbies are hired for DevOps on high salaries. Managers are impressed by the DevOps term... it's all a big crap and the results are shitty.
My company writes applications in PHP with a very simple stack. The concept of 'devops' is foreign to us because we don't need to compile, transpile, restart services, etc. The dev hits save, then refreshes the browser to test. Deploying to production is a single line command. DevOps sounds like a nightmare, glad we aren't needing any of that (yet). :)
"we don't need to ... restart services" - what does this mean? You clearly don't understand the stack you're running on, how can you understand an issue of a higher order paradigm?
@@cloudboogie It means our application doesn't have services to restart during development. PHP doesn't work that way. Been writing PHP for 15 years, i think i know what i'm talking about by now.
I'm MLOps. It's just software engineering good practices around ML. Introducing CI/CD for ML. Introducing model monitoring. Just like DevOps, it's mostly about the philosophy and requires "organization buy-in". It's just that instead of having unit tests and monitoring for (just) code, we do it for the model performance and data. And you don't make that work with a vague MLOps / DevOps title. You do that by writing code that enables that. Therefore, it's closer to the concept of SWE imo. (provided I'm not mixing terms)
Platform engineering is just applying DevOps principles to all of the tasks that get roped into DevOps enablement (the stuff DevOps teams tend to get farmed out to do). Make a clear boundry define your interface, everything in that is your baby, craddle to grave, everything else is a customer. This is true for your k8s/serverless/vm platforms, your CI/CD pipeline infrastructure, as well as the actual user facing applications.
I remember years ago when I was a little spud starting out no software developer would touch anything related to devops or continuous integration or continuous delivery. They wouldn't even do any SCCM work if pushed. Then Google came along and created the site reliability engineer role.....Suddenly the software developers who swore against anything relating to chef, maven or ansible suddenly wanted to be knee deep in this shite. It was the best marketing ploy in tech in the last twenty years and tonnes of people fell hook line and sink for it.
The biggest problem no one thinks about is that there is a time gradient across the software you create and deploy. DevOps only assumes that all 1000 different services are always up to date and deployed on a regular basis. Good working services will rarely be modified and updated or deployed, so you create a rift between your latest deployed services and the oldest deployed services that may contain technology that does not even exist anymore.
I would find it very helpful if the read article was in the video description. I often find myself the need to re read this or share it and it's kind of tough right now. I also think it's fair for the writer of course.
Half the problem is that DevOps is introduced as a theory (we've all read The Phoenix Project) and a responsibility (Make automation to support the new thing on the new platform). Ultimately sounds like this individual is a bit jaded because they have taken on the company's miscellaneous catch-all requests and that becomes frustrating for everyone involved. I've participated in a startup where everyone had 'DevOps' in front of their title early on (and everyone considered their automation and infrastructure needs) and slowly migrated to a dedicated DevOps team. Fortunately those teams stayed connected and DevOps would constantly assess what problems Engineers would face and provide secure solutions and a few neat little observation tools along the way. 🤷♂ I don't know man, every time I read a new title introduced as the solution - I can't help but think that it stems from communication issues that leadership failed to address.
If there is a special team named as "DevOps team", and if other teams don't care about how to filter the online status etc. - As there is no feedback with a proper Closed-Loop, it is possible that it is already on the way to a failure. Most important, DevOps or DevSecOps is a culture for engineering at first. Tools & process are only the method to implement that culture, to make each software engineer be a better one.
This is why heroku is amazing. No one wants to deal with DevOps. I just started at a job where they moved from heroku to AWS and no budget for DevOps. The moment I finish updating their old rails app to a newer version I'm moving everything back to heroku. This business wants new features and app bugs fixes for a super old app with 10% test coverage. I don't have enough time to also be dealing with DevOps crap from AWS.
To get started! Sure, it's great. But so many have blazed that trail before you. Why do you think cloud exists because there's a market. Why don't all services run on Heroku? It's great to get your ideas off the ground and you can run it for a longs as you want, at some point with real growth, you won't be able to.
Heroku is only good for small apps, as soon as you reach a certain threshold, I guarantee you gonna run out of memory or CPU regardless of the node scaling being used
Thanks for sharing your gripes. I finally feel like I'm not the only one who has devops issues (mine was Docker compose, which I fixed w/ Swarm) - heck, you've shared issues I didn't even know were such a big thing (Terraform and autoscaler).
DevOps internal tooling is the way to go, even though it has downsides in terms of job flexibility for devops engineers and developers in general. Thing is, if you want a naming scheme - enforce it! Create an internal tool that validates those inputs to ad nauseum to leave 0 flexibility in naming to devs, otherwise, they will just NAH-prod_2-eu-west-2_test-2_July_momLookAtMeNoHands all over the place...
1:30 12 years of data engineering, bi-dev and analytics .. you have no idea how true this line is xD took a sip and laughed for half a minute fighting not to spill something xD
While this author isn't doing a great job acknowledging their biases (you know, the whole "I'm selling a product in this space" thing) there are some really good points there. I've worked as a member of the DevOps team in a company with about 70 devs and what we'd have to do was assign each devops guy to a few dev teams based on how impactful their work was on production. In the beginning we did little more than Terraform all AWS shit and build CI/CD but I don't wanna be repeating the same task over and over so I built something to let the devs self-service. The biggest limitation for me was always my deep-seated hatred for the web frontend stuff so I built them a Python CLI tool to do the whole and published a Docker container on the internal registry. That tool and the automation around quickly became my job. Fast forward to big tech now and we are gradually moving towards platform-building; I am liking the trend in my current employer of building internal tools to solve problems by enabling self-service for the different engineers. Our problem is we're not really building a cohesive platform, instead we're building a bunch of disjointed internal self-service tools that don't talk to each other and still need to be orchestrated by hand or another layer of tooling on top of them. What I'd suggest is to start with dedicated platform teams that can own the whole platform stack consisting of, at least, some kind of workflow engine, generic compute, a modular/pluggable architecture, network and a consistent UI. I don't care if you're building a big monolith CLI, a massive web app console thing or just some tooling that consumes our beloved YAML, just have a consistent one-stop shop for people to launch and manage what they need and then get out of their way.
From my perspective working with DC infrastructure, DevOps seems like crap that either a) breaks my configuration files or b) deletes my configuration files... automatically. or c) doesn't do anything useful anyway because it can't handle servers or highly unpredictible new releases of enterprise software.
Nobody is actually doing CI/CD right. At least technically speaking. If I’m not mistaken, CI/CD imply pushing directly to master and then building and deploying the new version on every push. In reality this is a ridiculous approach, so some sort of change management is used. So instead we open PRs and all that.
@@HominisLupis No. Ci -> integrate changes of everyone in team often. Multiple times a day. That it. Everyone is working with the same project state without hiding anything. CD -> always have working build at every step. Release to world or client is separate business decision.
We had a DevOps team. After a while, they renamed themselves to SRE. After we got used to the name, they renamed themselves to Production. Their lead even gave a presentation explaining why we should call them Production, not DevOps or SRE. It felt like an identity crisis :)
If you feel like you're useless as a devops engineer, you can install some new UI, like rancher or something. Or you can just limit access to devs so they can have new blockers. 😂
This author could have saved us all some time by saying "Good engineering is hard, time consuming and expensive." Overall he contradicts himself. He says "DevOps bad", but then says you need people with skills in both areas. 👍🤦♂️ Here's the "secret" of DevOps: it's just good Systems Engineering. We just call it DevOps for the benefit of management types who don't know DNS from a runaway chain reaction. So much of what he complains about, is a sign that there are architectural issues or that the system has outgrown the staffing levels. Also, his advocating for "engineers should be able to do whatever the hell they like" is about the most retarded thing I've heard in a loooong time. Holy sweet Jesus! Have you ever had to support a heterogeneous environment? Ever had to take one over? Yes, you need to be using the most appropriate tools for a job, but what tools you have *must* be used consistently, so that others can unpick your mess when you leave. Does this mean that larger organisations become more stiltified? YES, OF COURSE IT DOES, because that's the fucking DEFINITION of a "big business". Frankly, this guy comes across as "just" a programmer (not least of all because of his "hobbyist" comment!!!), who doesn't like doing serious engineering. I'm a serious professional, fuck-you-very-much. I'm not a "hobbyist". This shit is HARD and made harder by tools like this that refuse to accept that engineering is about compromise. Further, devs who have "no interest in the platform" are, by definition, amateurs; F1 drivers know almost as much about the cars as the people that designed and built them. This is the attitude that software engineers *need* to have about platform.
Thank you, well said and makes far better sense. I couldn't see the sense in how the entirety of software was bundled in as inherently hobbyist in nature when there's nothing about what we do that's different to... well, any other profession that takes time to learn and has specializations. Just sounds like someone who thinks that what we do is holy rather than it being a highly technically rigorous profession.
NGL - this one flew over my head a little bit. An example can go a long way in explaining things, but this article is only an abstract wall of text. And I still don't know what ADP means.
DevOps is fine tbh. Kubernetes is a great tool but sometimes overkill for small startups but if you are willing to create k8s manifest files and create helm charts, it makes things so much easier to deploy applications to your clusters. There is a lot to handle though and once you bring devsecops in, it immediately fucks up your CI/CD pipeline
fr. I think each product should have an engineer making sure the infrastructure for that product is set in place. By using existing tools or makin new ones. I always found it weird that DevOps is a completely separated team. Maybe have a team that is Product-agnostic that deals with the infra to deploy the pipelines and tools of each product, but at most that. Each product has different needs, by having a team that deals with requirements for many projects just because "it's backend/infra stuff" makes their work extra hard, and in my opinion, breaks the organizational structure of the business
DevOps is a perfect example of how our industry fucks everything up. Development and Operations are *two different* teams. DevOps is the practice of these two teams working collaboratively instead of being at odds with each other. If you've founded a "DevOps Team", you've failed to do even basic studies to learn what DevOps is and deserve every bit of pain and suffering you experience. Don't get me started on Scrum and how utterly fucked up that is.
My market analyis shows Devops & Kubernetes are actually growing 20% per year, while sometimes an ungrateful position there's a huge positive outlook for people in DevOps & Kubernetes.
I was stuck waiting for DevOps to set up a Jenkins slave for over a year. Turns out he just had to fill out a 5 minute form. Management tried escalating this repeatedly during that year.
As someone who is a 'security oriented devops' person, I used to build platforms that help speed up the devs, and help automate things for the SoC, and other stuff, I can say that with where my org has gone in the last two years: the people driving us have no idea how to use us, or even understand what we do. Half my team has quit, they're not backfilling, and it feels like drowning. Where we once had a beautiful self service portal for engineering resources, today, I have to ask ops for a vmz and wait ten days. RIP
As someone that comes from a different sector and now entering devops that I have quickly realized that DevOps is a organizational philosophy that the entire company needs to adhere to. You can just hire two devops engineers and hope that will fix things.
"DevOps" is just another desperate label for "Quality Assurance", AkA "QA". As we all who suffered as "QA engineers" for decades before we finally wised up know, no software company currently in existence gives a flying f---- about whether their product is any good, much less even works at all. They only care that they get their customer's money. Accordingly, no software company currently in existence wants to do QA or fund it in any way at all. So, given that QA is a totally dead-end affair because of the foregoing total apathy about SW "quality", and DevOps is just another name for QA, well...... you know what they say about 2 + 2 equaling 4........
devops named the prod and the QA system PROD and non-PROD. this is terror inducing when you get an alert or are e.g. rebooting QA and read PROD in your peripheral visions
DevOps here, people would call DevOps anything but DevOps :))). Platform Engineering is still just DevOps. What dude says works well if you start project from scratch, but let see how well this applies to real world scenario where you have legacy apps, you have to do lift and shift and also have to do training for devs. At the end of the day this dude only wants to sell his platform, which is fine, but yeah DevOps is not a job is a a set of practices. Also to me seems like a business idea: why pay a 'DevOps' guy x-amount of money when you can force a developer to do the work and increase his salary with 10% so he doesnt hate you. What works best in large companies is to have teams that are made up from devs, qa, devops etc. and each team member is embedded into the team. It's still some kind of silo but not in the traditional way.
Wasn't the idea of DevOps to make the Devs not only responsible for building new stuff, but also for continuity of the application while Ops would not only be responsible for the stability, but also to ease changing it. In our company, it sort of meant that Ops people learned how to code and developers learned responsibility (of course they already knew how to maintain a platform, just had to learn it at scale) I still like it.
The simple issue I have with most of the "Dev Ops" that I did was that the system I was forced to use that was described as a "Simple, opinionated declarative language" was that it forced you to copy pasta code that otherwise would have been simply handed with a variable. For example, for each deployment, I had to create several yaml files that had values embedded in them that had to be the same - stuff that was easily generated with a simple script.
So, so many people saying “X is bad” and they are right because their interpretation of X makes it bad. DevOps bad? No. The idea op DevOps is that the developer is also responsible for operating it. No one in their right mind says that the developer must design and build the operations infrastructure, tools, procedures. There is no contradiction in doing DevOps AND have engineering teams/IT4IT/IT4DEV/SREs. Developers in DevOps must be given a comfortable platform to use to Operate their software on in development and in production.
Please create a ticket and we will get to it in our monday morning intake... code for leave me alone I am doing something that was dropped on my lap this morning that was previously techdebt and now is a fire. No one gives the ops people high fives unless they are on late getting the release fixed.
Devops isn't terrible, cloud isn't terrible, multi-cloud deployment aren't terrible. Tools for doing these things are terrible. Using AWS to deploy application is a nightmare, the UI/UX is a nightmare, using YAML for configuration + programming is a nightmare Using Jenkins for more complex staff is a nightmare. Maybe I'm to old, but Installing software from floppy disk or CD had better UX then deploying application to AWS. Don't blame the concept, blame the implementation The current solution to these problems is rename the concept and use the same bad tools and repeat this every 3-4 years. This is how agile/TDD/DDD survive for 20+ years
"The current solution to these problems is rename the concept and use the same bad tools and repeat this every 3-4 years." They are doing it I guess. Platform Engineer is the new thing. Now Enterprise want to cut the cost and repeat the cycle by rebuilding as self hosted services
This is sort of what our DevOps team does. They build a platform and set the templates up to have proper security infra so that the dev teams can easily create what they need in a scalable and secure way. We control our github actions based on the runners that devops provides with the appropriate iam roles. We can even spin up ephemeral environments for testing including spin up and tear down of databases in an RDS postgres cluster, all automated using strong security and vetted templates.
For me sounds it all like organizational problem, the employees were not prepared well and there was like no real plan for implementation step by step. Normally companies should start from docker and the slowly try out kubernetes, but yes some software vendors can sell Kubernetes very fast, me and my company we always ask our clients what are theirs real goal of achievement , when they have particular tech in mind.
Sorry but the fact the writer actually wrote a product in the area of Platform engineering and thinks DevOps should be thrown out of the window shows he has ZERO understanding what DevOps is and instead thinks that creating silos / titles changes the role. He has a VERY good reason to push A over B as that is his living. He chooses to wrap his ideas with "nice-words" and promotes his pink glasses. Platform Engineering uses DevOps practices', DevOps methodologies, DevOps tools. Every single "Platform engineering" topic he mentioned is basically DevOps. The term "platform engineering" has been around for almost 2 decade, and as you mentioned PE is just a new marketing-fused new term to present same ideas we have been having for 2 decades but because of BAD implementations, our indesutry things that it needs to "reinvent" itself, get new titles, new budgets, new silos. If you really want to have a discussion about DevOps @Primeagen, feel free. Our industry really needs to start talking and have good leadership to stop this nonsense of new names every 3 years. it started with cyber, then DevOps, now PE and in 3 years well have something new as AI is shaping up the landscape.
Canaries is something most companies that are really serious about DevOps and CI/Cd (for that reason) cannot do. We as people that generate energy and deliver it to the grid only have 1 nuclear power plant and 1 coal power plant and 3 gas tribune plants. We can’t phase it out. We role it out and it need to work. If we fail, we disrupt the grid, causing power outages and when we are fully inlicht a blackout. Banking is my other customer we deliver required reports to the ECB and Dutch Bank those are the only two and it needs to be correct! And it needs to be there in time. And for accounts you can’t have a new feature running on X customers because you can’t role back financial transactions! It’s not allowed by law! This is why you can’t book back a transaction you did. Payments are final. And even according to SOX law you need to be able to prove how any transaction worked at any time. So your audit would and reproduction or evidence will be a hell!
I'm a Devops and I think it is about to game over now. Because I don't see there''s no more openings for the DevOps jobs in my market since the last year. The fate of DevOps role is just like the EA role back in 2 decades ago. The Enterprise Architect thrived only a couple years (2005 - 2010). Then it sank forever. And I saw no more Enterprise Architect jobs available in the market in the past decade anymore. The Devops is on the same path as the EA now. So Devops will be history in a couple years later. Just a waste of time (I earned the title but no future to sustain).
I have to work occasionally with CI/CD pipelines that take 10 hours to complete. And it’s constantly failing because of some dependencies. And all I want to do is just add a new region to the deployment config and deploy that stupid services there (don’t ask why we deploy services that aren’t ours, our processes are… weird). Oh when you’re doing this you can block devs working on their stuff, fucking beautiful. Every time I have to do this, I really want to meet mfs who designed and developed that bullshit. Oh, the pain, I’m getting mad even thinking about it. CI/CD my ass.
My problem with Devops is that the line responsibly is not clear at my company. I end up having to learn just enough to get the thing done every time I need anything done, but never get enough reps to feel comfortable doing it
I was writing in php and (ohh my good) yes, it doesn't matter in the end. The main thing is that I'm able to deliver features fast, in reliable way, with only 20 lines deploy script. Im not using kubernetes terraform and so on. Why? Because product owner needs a good product, not terraform or kubernetes. If i can make things in a most simple way, I will do it that way.
Question: How many of you are doing devops without a college degree? I have no education beyond high school diploma, but my job history is sort of all over the place. I bounced around between C++ developer, IT Support, System Administrator, Python developer, PLC programmer, electronics manufacturing. Always used Linux and have been spinning up VPS to run my own git, mail, websites, etc. Where do I go, what do I do? Certifications? Should I go to college? Both ideally obviously?
Devops is what you get when you do OOP. Not a critique of OOP that's just the nature of things. FP makes you keep system design in mind in aess myopic way.
As the head of DevOps at some of the biggest companies in the world, all I can say is...
"Thanks for reaching out, Please raise a ticket through the correct channels for CICD support."
I need this at my work
@@replikvltyoutube3727 just build an auto response bot for the ticket system. It's fine
Just to get ignored
Oh god, I'm so glad I don't have to deal with this
I am in a fortune 200 tier company and its hard to find DevOps to help us with basic OPS shit. I am Dev and we always struggle to get help to lubricate the devops the philosophy of devops was supposed to be to connect dev and ops, but they proceeded with disconnecting dev team from ops team like its nothing
I come from a sys admin, and data center management background, and I have noticed a savage divide between developers and operations. As a DevOps eng, I'm constantly battling dev's requests to manually run code in production; individual user accounts with write access to DBs, or "just give me X resource I need publically available." My background forces me to look at costs, security, and performance, and developers DO NOT care. It is the same thing as it was with on-prem resources, bad code? add more CPU and or mem, app needs to communicate with X, just open the firewall, need a new service no documentation so round and round with requests they didn't know they needed. My solution has been self-service tools but I'm CONSTANTLY catching dev's miss using these tools and putting the company at risk! So I have to pull access and limit what they can do to protect the company. For my WoW players, DevOps is the healer of the raid no one cares until it affects them. Our job is to keep you from destroying yourself and the company.
I really like your take. Thank you for sharing.
This is an interesting point. If devs have the tools to provision their own resources, how do you avoid them throwing more resources at a problem without oversight?
How do you prevent opening a resource to the entire world “but it works” situations? Someone needs to be auditing these things somehow.
Maybe having a security auditing team and a performance auditing team rolling between all internal projects?
I've been in your situation and what I ended up doing was to sit down with management (eventually even the CTO) and explain that there need to be standard hard and fast security practices that nobody deviates from without approval (security team will be your ally on this), approved services with bulk buys to keep costs down and a team budget for deploying anything else, all enforced via the self-service tooling. We then revoked dev write access to all cloud APIs and ran workshops to help the dev teams onboard. Took us 2 years in a company of about 100 engineers.
The thing is, we needed a catalyst event (most companies will because you'll be working against entrenched culture).
In our case, we had an R&D project that pulled retail customer PII into ElasticSearch but the devs on that project went quick and easy and just deployed the AWS service. The problem was that this was new at the time and not supported in-VPC which meant open to the Internet. Project wound down and they didn't clean up. I come in, look at AWS billing, notice the line item and decide to investigate. Single document on cluster read something to the effect of "we have your data hostage, pay us BTC or we publish it". Well, there goes the quarter, call InfoSec, they hurry over and there's jaws on the floor. Eventually, the GDPR fine hit, CEO and CTO walk onto the engineering floor and declare "the cowboy dev era is over, everything goes through DevOps and InfoSec".
Good luck.
You are the gatekeeper at a petting zoo basically, and the kids will always be kids. It will never change
I also come as a sysadmin role and the only memorable experience I've had from our devops tooling was that it was constantly deleting or misconfiguring my configuration files and when I spoke to the devops team they tried to explain to me that they have a policy. Their policy was ad-hoc and went both against the documentation and the empirical evidence of how things actually work for that specific system. I showed them both the documentation and the evidence, they didn't fix it. I fixed it with a workaround so it will break again on the next reboot, because I literally had no other option. I had to explain to the customer why the goddamn app randomly stops working as if it's my fault, since I'm responsible for it. And yet I don't have access to the devops tooling that directly affects it and the devops team doesn't respect my input. Devops to me is more like that healer in raids that keeps crying how bad the tank and DPS are, crying how the healer's should never DPS and then misses all the healing timings and gets you killed. I'm not saying all devops is like that, but I feel like rule #1 of big companies is that nobody knows wtf they are doing.
As others have mentioned, pls do the courtesy of linking to the article whenever you read it. It helps us viewers, as well as the author.
100% this. But maybe the sole point of this channel is to entertain and not to help.
You can solve this yourself. It’s not hard to search the title on google. Takes an extra 10 seconds of your time. Spoon-feeding is not owed.
You know what its called. Use your initiative
@@russellsanders4535 true, but also takes the reactor 10 seconds to link the article in their description. Helping boost the articles click through and adding text based credit
Yea he is getting free content, he only reacts to it lol
If copying and pasting previous work was a sign that a job was bullshit, then I have grave news…
Golang in a nuttshell lol
@tablettablete186 Hello bootcamp student.
I'm fully on board that a part of my job is bullshit.
Now imagine moving this bullshit from one type of system to another. and then expect anything to work at all.
Name a job that isn’t repeating the same thing over and over, or at least having one part that you repeat over and over.
@@danielthompson2561damn, I have not had a job like that in a decade, maybe time for something more interesting?
As a DevOps practitioner I obviously have some opinions on this article - mostly positive, tbh. I totally agree that too many businesses treat roles like SRE/so called 'DevOps teams' as an excuse to palm off responsibilities that should be shared. Also fully support the call for more platform engineering in place of a bajillion teams with a bajillion different cloud environments, all managed with whatever hand-rolled solution that team came up with. The reality is we've known about, and tried to steer away from, these problems within the DevOps community for a while now. A big part of the problem is the fact that, if you get into DevOps through a bootcamp, the only experience you get of implementing these engineering practices comes in the form of individual/small group projects, where the inherent problem in trying to coordinate these practices between tens or hundreds of development teams remains hidden. We really need to do more to expose trainee practitioners to problems of scale early on - DevOps is, after all, a philosophy for improving organisational performance, and a bootcamp simply can't do that justice.
This is also an issue in software engineering. There is no way to simulate scale-like work without actually working in an environment that has big data.
And there isn’t really anyone in CS trying to get an internship in DevOps because from what I’ve seen, it’s treated as a pivot for an engineer who doesn’t want to code as much.
So the only options are:
Hire more juniors and train them accordingly to take over once someone leaves
Continue as we are now
The word 'velocity' in the dev space drives me nuts
I hate these terms
The espeed is alot mate
we gotta synergize to provide value through elevated velocity in the agile environment to disrupt the industry maaaaaaan
ok, but what happened to 'acceleration'?
jerk?
position?
@@jonathanalonso6492 I think we need to increase the 'jerk' on the 'acceleration' of our 'velocity' otherwise we just won't reach the 'position' we aspire to 😂
Someone (boogibugs) asked a great question in the Twitch chat: "Why is the operations team not classed as an engineering team? That's one of the issues right there."
Boogibugs is correct. Just look at Star Trek. Engineering and operations always wore the same division colors. #justsayin
Who cares abou the title
@@jordixboy The title decides how much pay and respect you are given.
DevOps was never intended to be a role. It's a concept. Have your Ops integrated into the development process.. influence the product architecture from the design stage. Not a place where you show up on Friday saying "I need this whole new thing deployed on Monday." Then go to the nearest member of management and cry you didn't deliver on time because those "damn devOps are blocking you".
DevOps is a concept of enablement, not a role of blockage.
Only in the most traditional and monolithic environments I’ve worked in have I seen ops not classified as engineering.
@@jonathanalonso6492 not at all bro
Compare a java dev w/1-2 years of experience to one with 10+. Pretty drastic difference, no? (sql injection? idor? session fixation? txn boundaries?) Same thing applies to 'devops'; k8s isn't just some 'configs', there's a whole ecosystem of best practices, frameworks rising/falling, architecture and so on. That equally applies to tracing/logging, etc
The reality is you end up with devs doing stuff in application code that should really be infra, but that's not the teams expertise. They are just copy/pasting the top SO results and effectively `chmod 777`ing your cloud account to get the server to work. Generally, theres no checks/balances (that's a silo, yo!) until you're "hacked".
I don't disagree, but I work with people who got 20 years of experience and still manage to produce the most ridiculous vulnerabilities known to man.
@@freesoftwareextremist8119 humans are imperfect, it's not about writing code with 0 vulnerabilities, it expecting them to be there and to put a practice in place to catch them and to not trust the human.
@@freesoftwareextremist8119 100%. There are old fools as well. Still, there's generally a correlation with experience and specialization.
The fact that we even need DevOps is a testament to the amount of completely unnecessary tech debt 99% of tech companies have.
the question of "how do we deliver code to customers?" is not one to avoid or 'pawn off'
I think it is more a byproduct of how at many companies the backlog is owned by product. Product rarely makes time for the types of work that are often considered "devops", but the works needs doing. Eventually there is a break-off group of devs that get to operate a backlog without the oversight of product. They are going to try to get on top of all the "devops" issues that have piled up and therefore turn into a _DevOPS team_.
Companies that give engineering more influence over team's backlogs may be more likely to evolve towards platforms, because there is less of a build up of toilsome "devops" work.
Devops != Tech Debt, nor is it a result of tech debt. Devops have their own tech debt, development teams or products have their own. In some cases there is convergence.
At the end of the day, someone has to handle (build, maintain, support, scale, secure, cost optimize, observability) infrastructure, build and maintain pipelines, ensure some form of security best practices are being implemented along the way and lastly - I've rarely seen a developer answer compliance questions that didn't only apply to their product, let a lone gather and submit evidence.
Lastly, most devs in a large number of orgs are paid to code and deliver on something(s) with in their allotted time (sprint) and it's usually a feature. Rarely they're afforded the time to worry about anything outside of the features they're working on because its not in the business best interest when they want to crank out features. Think v1. V2 is typically when companies start to scale and a redesign is in in the works, well. Devs are now business on the re-arch. Not Devops! That's where we come in.
my guy, all internal politics aside:
undelivered product is UNFINISHED product. period. no way you can tell a customer "works on my machine" and call it a day.
@@davidthomson3575 I'd consider "devops" tech debt created by devs to be things like:
- our deployment process is someone manually running a script
- we are using a dev's IAM keys to give a server access to another service
And so many more. It is the brittle and insecure things that happen when corners are being cut that work until they don't.
When publishing videos like this, reacting to articles I mean, if you can share the link of the article in the description section that would be great... just saying... Great videos by the way
classic react video not crediting anything
It takes a shorter to type author, article name and google it, than it took you to write this comment, lol.
@@gercius Soy take, it takes like 5 clicks to copy the link and post it in desc
@@ThatOpinionIsWrong Yeah boy, those few seconds are too precious for your waifu pillow cuddling sigma male grindset.
You can look up what a p tag is in stack overflow but you cant google the article title? You're too virgin
“One man’s golden path is another man’s golden shower” fixed that for ya
Devops != Tech Debt, nor is it a result of tech debt. Devops have their own tech debt, development teams or products have their own. In some cases there is convergence. Granted this can be a group of product devs or a separate team, eventually the former won't scale very long.
At the end of the day, someone has to handle (build, maintain, support, scale, secure, cost optimize, observability) infrastructure, build and maintain pipelines, ensure some form of security best practices are being implemented along the way and lastly - I've rarely seen a product developer answer compliance questions that didn't only apply to their product, let a lone gather and submit evidence.
Lastly, most devs in a large number of orgs are paid to code and deliver on something(s) within their allotted time (sprint) and it's usually a feature. Rarely they're afforded the time to worry about anything outside of the features they're working on because its not in the business best interest when they want to crank out features. Think v1. V2 is typically when companies start to scale and a redesign/re-arch is in in the works, well. Devs are now busy on the re-arch, not Devops! That's where we come in.
"Devops does not work because the hired devops guy cuts corners and engineers should not work with ops. The solution? Build your own PaaS platform."
??????
Yeah, exactly. Bob, when do you think we can make that change to the frontend I was asking you about. Hey Joe, that will take me about 45 more days because I am busy building out the whole Platform. If you are a 1 man shop or a few dudes doing a startup, I understand. But when you have a scaling organization what you are referring to is not ideal.
@@davidthomson3575 Not only that. Even if you have the resources, you are still just rebuilding an inferior version of already existing AWS Kubernetes/Terraform style hosting services just to virtue signal that you are "doing deployment right".
@@adamih96 this is true, they're not in the PaaS business, are they?
Maybe if you build some quick macros to facade above those PaaS accounts and wrap common cookie cutter process in happy buttons and forms?
@@Fernando-ry5qt If easy enough to put behind dashboard buttons then there are likely already tools to automate that process within the same service or with something like github actions. The best button is no button, the best app is no app.
When you have a "DevOps" team you need to file a ticket with to get anything done, then, guess what, thats just the ops team with a buzzwordy name.
Yeah I agree, worked at a place where everything on infra was repo driven, you just sent a PR and post it on a channel to get a review, the checked if everything was fine, merge, wait a min and done
It was pretty good in contrast with the old ticket system I am used to work
I'm "doing devops" in a small company and I like it. Important part is the end-to-end-responsibility, devs learning from ops problems. But it does require management understands "OPS > DEV" and you can't have 100% schedule management. We plan for 2w sprints (~scrumban) but it's just setting priority, what tracks is everyone focusing on. Tickets are done when they are done. Most important part of "my devops" is that once I hit the same thing Nth time, I will either do a bug fix (if I think I know the code cause) or add logging or setup monitoring. In traditional Development this can't happen because the guy looking at logs doesn't understand the source line behind the log line.
This article shows me how much I have yet to learn. I understood very little of this article.
In trying to solve waterfall development's "problems" we have created a hydra of problems. Waterfall was simple and could apply to anything. Software was less brittle that way. "Real Agile" is also good, but the common way it's done and this whole DevOps stuff is just a ton of make work for very little actual work that gets done. Most of the time, I'm just left shocked what people call work. It's akin to bureaucracy and paper pushing than actually building stuff.
Based
yeah honestly the amount of filler meetings to discuss work, or the amount of "investigate blah blah blah" tickets where the scope of the ticket is just to figure out how to do the actual work - it's just too much. the team would be much more productive if we cut the filler and just built the shit. sometimes the sheer volume of meetings feels like excuses for people to avoid getting their hands dirty with code
Guys stop commenting below and open a ticket
Can we get an epic?
DevOps is a stupid buzzword name, but at least the point of taking the best practices of software development and applying to infrastructure delivery makes some sense. The challenge is mostly a people thing, nobody wants to learn how to do their job over again of let others take over their kingdom, and they resist in the most stubborn ways possible. Technology usually fails at the hands of an inflexible mind.
So the real problem is people don't value the support people...
Exactly, every DevOps person (yeah yeah or whatever title you want) has to not only shoulder the bitch work the Devs were too LAZY to do but then catch a bunch of shit about how “useless” they are at the same time, fun fun
@@itsdylac sadly, yes. Of course the ops people can be just as lazy.
@@disguysn they totally can, and incompetent too. But the competent Ops people get treated like trash, and the incompetent devs get a pass
@@itsdylac I wish I could contradict you. I moved from dev to devops at my last job and it's exactly as you stated.
the company i work for has outsourced our "devops" team, so i really felt this 1000x
My main problem with DevOps is that I don't even know what exactly are they supposed to even do.
Where do their responsibilities start and mine end?
Why are they refusing to give me a read only prod db access, when I do really need it for analytical stuff and our app isn't even close to being released yet?
"They"? As in a separate DevOps team? That's #1 in the article: DevOps doesn't mean you have a separate DevOps team.
@@JeyPeyy idk what happen to my comment.
devops is a production methodology for agile management.
devops, right now, work very well for web ressources .
Due to tech debt (or aging workforce), the average tech company is based on ITIL management.
where each step of the devops cycle is managed by a different team.
In those company the job title "devops" can mean a lot of thing (depending on HR).
@CHE6yp is talking to an infrastructure/platform admin .
Because read only DB access to a prod database can send the sucker crashing with a single query. This is not a best practice, because we learned long ago no one should do this.
You need a data pipelines or atleast a read replica.
You write code, you should worry only about writing code. If you want a shit ton of distractions that keep you from writing code, start worrying about all of the non code writing responsibilities that fall under the guise of devops.
@@davidthomson3575 Oh I am so glad you said this! Illustrates my main PITA in modern development perfectly.
In general, I would say giving any access to prod is highly dubious and not a "best practice". However, "best practice" is often just a massive distraction in itself when applied to a concrete example.
In my case, we have 2 backend devs and our project basically just agregates a bunch of data from a third party. All of that data is completely recoverable, and the amount of users is currently zero. The reason I needed a prod db is because prod is the only place that does have actual data, and my task was to basically mutate some field values to make them all confirm to the same standart. Nobody has a full list of actual values that need mutation, so basically I needed to make a bunch of SELECT DISTINCT queries just to find out what exactly is needed.
So, whats the path of least resistance and the best practice? Waiting for our DevOps/Platform Admin to make a replica, going to our prod frontend and manually cataloging all 2 million records, or, getting a prod (not even read-only necessarily) db access?
@@CHE6ypprobably asking for a data dump haha
This is unrelatable to me because our team develops, deploys end to end, writes the CDK infrastructure, handles oncall, etc. I think separate dev ops teams just encourage engineers to be lazy tbh
I used to do that. The problem with generalization of that sort is that you’ll keep tripping over amateur problems as it takes forever to develop expertise in any one area. I love the concept, but in practice it can lead to burnout.
@@samgould8567Yes. I too believe it’s better for every organization, with capacity, to engage individuals who have specialized in solving specific problems, like devops as a practice especially in the sense of ensuring reliability. You can’t have everybody doing everything, it will end in chaos. 😅
So who's managing the cloud then? Presuming you're on AWS, who's managing organization itself? Security, compliance, fin ops? Who's going to be waken up if CloudTrail logs have something suspicions - on-call from full-stack Team A or Team B? Who's gonna call you out on unused resources?
I agree. The same goes for full stack garbage. Civilization is built on specialization and the key to succeeding is the collaboration of many specialists. Imagine this in any other industry: “we’re looking for someone who can work on every single aspect of the building: electrical, heating, plumbing, concrete, etc. We will assign each full stack trades person a room, and they are responsible for the whole thing.” You’d end up with inferior, inconsistent garbage, just like today’s software!
We should all more aspire to be jacks in all trades than masters in just one.
It is very important to have overlapping skill sets and at least know what other roles are doing to enable communication and a mindset of shared responsibility and belonging.
Unfortunately big corps are usually driven by automated processes and use business units, which separate people into well defined boxes.
To say it with the words of Melvin Conway
`Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.`
Article just sounded like the guy was complaining about everything. “DevOps done by engineers is bad because the engineers hate it and suck at it.” Then he goes “DevOps done by the DevOps engineers is bad because they are not good at their jobs and just copy paste things” Then his solution is to rename the DevOps engineers to platform engineers and magically everything will be better.
The reality is that hard things are hard and random hat swapping is not gonna fix it. This is fundamentally a technology problem. When the tech changes then the situation will change.
The problem is that "dev" is creation, and "ops" is maintaining. Completely opposite perspectives (and cultures). Very few people can do both, let alone well (with the right mindset).
And the very disturbing things is that bad to mediocre DevOps earn on average MORE than good software developer. I didn't believe it until a guy I know, who doesn't even know how to write a for-loop, who worked at RedHat told me his salary and showed me their repos... He was writing yaml files and introducing very lil amount of "code", I think that three of them full time for about a year still didn't produce as much code as I alone do in just a week. Another thing is that most of them are clowns and liked to be called engineers while they can't solve the easiest equation ever.
If they can't solve easiest equation ever, why don't you do it, perhaps you don't know how to write a yaml file either
@@abdulhamid.mmousa2329 because it seems that deploy is not a developer's concern.
@FredoCorleone is right, it's common that newbies are hired for DevOps on high salaries. Managers are impressed by the DevOps term... it's all a big crap and the results are shitty.
CTO: Our software is complety secure.
Dev: Shell script goes brrr.
CTO: Suprise pikachu face.
My company writes applications in PHP with a very simple stack.
The concept of 'devops' is foreign to us because we don't need to compile, transpile, restart services, etc. The dev hits save, then refreshes the browser to test.
Deploying to production is a single line command.
DevOps sounds like a nightmare, glad we aren't needing any of that (yet). :)
"we don't need to ... restart services" - what does this mean? You clearly don't understand the stack you're running on, how can you understand an issue of a higher order paradigm?
@@cloudboogie It means our application doesn't have services to restart during development. PHP doesn't work that way.
Been writing PHP for 15 years, i think i know what i'm talking about by now.
u clearly don‘t. Give it maybe another 15 years.
Why the hate? I actually do not know why.. someone explain@@thunderstein5041
I'm MLOps. It's just software engineering good practices around ML. Introducing CI/CD for ML. Introducing model monitoring.
Just like DevOps, it's mostly about the philosophy and requires "organization buy-in".
It's just that instead of having unit tests and monitoring for (just) code, we do it for the model performance and data.
And you don't make that work with a vague MLOps / DevOps title. You do that by writing code that enables that. Therefore, it's closer to the concept of SWE imo. (provided I'm not mixing terms)
Platform engineering is just applying DevOps principles to all of the tasks that get roped into DevOps enablement (the stuff DevOps teams tend to get farmed out to do). Make a clear boundry define your interface, everything in that is your baby, craddle to grave, everything else is a customer. This is true for your k8s/serverless/vm platforms, your CI/CD pipeline infrastructure, as well as the actual user facing applications.
I remember years ago when I was a little spud starting out no software developer would touch anything related to devops or continuous integration or continuous delivery. They wouldn't even do any SCCM work if pushed. Then Google came along and created the site reliability engineer role.....Suddenly the software developers who swore against anything relating to chef, maven or ansible suddenly wanted to be knee deep in this shite. It was the best marketing ploy in tech in the last twenty years and tonnes of people fell hook line and sink for it.
The biggest problem no one thinks about is that there is a time gradient across the software you create and deploy.
DevOps only assumes that all 1000 different services are always up to date and deployed on a regular basis.
Good working services will rarely be modified and updated or deployed, so you create a rift between your latest deployed services and the oldest deployed services that may contain technology that does not even exist anymore.
When you have a shitty dev team ( which is 99% of the time) your services will be modified weekly ( if not daily)
@@houssem009 I am in currently in the 1% team. I hope it lasts long time.
I would find it very helpful if the read article was in the video description. I often find myself the need to re read this or share it and it's kind of tough right now. I also think it's fair for the writer of course.
google can help 😂 🕹
Half the problem is that DevOps is introduced as a theory (we've all read The Phoenix Project) and a responsibility (Make automation to support the new thing on the new platform). Ultimately sounds like this individual is a bit jaded because they have taken on the company's miscellaneous catch-all requests and that becomes frustrating for everyone involved. I've participated in a startup where everyone had 'DevOps' in front of their title early on (and everyone considered their automation and infrastructure needs) and slowly migrated to a dedicated DevOps team. Fortunately those teams stayed connected and DevOps would constantly assess what problems Engineers would face and provide secure solutions and a few neat little observation tools along the way. 🤷♂
I don't know man, every time I read a new title introduced as the solution - I can't help but think that it stems from communication issues that leadership failed to address.
"FAST FUCKING AUTOSCALLERRRRR" has became a meme that is said so often in my work place it's basically echoing off the walls and in Teams chats 24/7.
If there is a special team named as "DevOps team", and if other teams don't care about how to filter the online status etc. - As there is no feedback with a proper Closed-Loop, it is possible that it is already on the way to a failure.
Most important, DevOps or DevSecOps is a culture for engineering at first. Tools & process are only the method to implement that culture, to make each software engineer be a better one.
This is why heroku is amazing. No one wants to deal with DevOps. I just started at a job where they moved from heroku to AWS and no budget for DevOps. The moment I finish updating their old rails app to a newer version I'm moving everything back to heroku. This business wants new features and app bugs fixes for a super old app with 10% test coverage. I don't have enough time to also be dealing with DevOps crap from AWS.
isn't DevOps just make changes, push, deploy?
The most recent Stack Overflow Survey results would disagree with you.
@@ea_naseer /sarcasm ?
To get started! Sure, it's great. But so many have blazed that trail before you. Why do you think cloud exists because there's a market. Why don't all services run on Heroku?
It's great to get your ideas off the ground and you can run it for a longs as you want, at some point with real growth, you won't be able to.
Heroku is only good for small apps, as soon as you reach a certain threshold, I guarantee you gonna run out of memory or CPU regardless of the node scaling being used
Thanks for sharing your gripes. I finally feel like I'm not the only one who has devops issues (mine was Docker compose, which I fixed w/ Swarm) - heck, you've shared issues I didn't even know were such a big thing (Terraform and autoscaler).
DevOps internal tooling is the way to go, even though it has downsides in terms of job flexibility for devops engineers and developers in general.
Thing is, if you want a naming scheme - enforce it!
Create an internal tool that validates those inputs to ad nauseum to leave 0 flexibility in naming to devs, otherwise, they will just NAH-prod_2-eu-west-2_test-2_July_momLookAtMeNoHands all over the place...
From an Engineer to a Comedian: A journey through time
1:30 12 years of data engineering, bi-dev and analytics .. you have no idea how true this line is xD
took a sip and laughed for half a minute fighting not to spill something xD
While this author isn't doing a great job acknowledging their biases (you know, the whole "I'm selling a product in this space" thing) there are some really good points there.
I've worked as a member of the DevOps team in a company with about 70 devs and what we'd have to do was assign each devops guy to a few dev teams based on how impactful their work was on production. In the beginning we did little more than Terraform all AWS shit and build CI/CD but I don't wanna be repeating the same task over and over so I built something to let the devs self-service. The biggest limitation for me was always my deep-seated hatred for the web frontend stuff so I built them a Python CLI tool to do the whole and published a Docker container on the internal registry. That tool and the automation around quickly became my job.
Fast forward to big tech now and we are gradually moving towards platform-building; I am liking the trend in my current employer of building internal tools to solve problems by enabling self-service for the different engineers. Our problem is we're not really building a cohesive platform, instead we're building a bunch of disjointed internal self-service tools that don't talk to each other and still need to be orchestrated by hand or another layer of tooling on top of them.
What I'd suggest is to start with dedicated platform teams that can own the whole platform stack consisting of, at least, some kind of workflow engine, generic compute, a modular/pluggable architecture, network and a consistent UI. I don't care if you're building a big monolith CLI, a massive web app console thing or just some tooling that consumes our beloved YAML, just have a consistent one-stop shop for people to launch and manage what they need and then get out of their way.
From my perspective working with DC infrastructure, DevOps seems like crap that either a) breaks my configuration files or b) deletes my configuration files... automatically. or c) doesn't do anything useful anyway because it can't handle servers or highly unpredictible new releases of enterprise software.
Nobody is actually doing CI/CD right. At least technically speaking.
If I’m not mistaken, CI/CD imply pushing directly to master and then building and deploying the new version on every push.
In reality this is a ridiculous approach, so some sort of change management is used. So instead we open PRs and all that.
Please learn what CI/CD is.
@@cloudboogie Isn't it exactly what they said?
@@HominisLupis
No.
Ci -> integrate changes of everyone in team often. Multiple times a day. That it. Everyone is working with the same project state without hiding anything.
CD -> always have working build at every step. Release to world or client is separate business decision.
Finally, someone finally said it.
Devops IMHO is a discipline not a role, not a platform, not a team. Similar to agile software development
In theory, yes. But in practice note, this is why we have the role devops engineer now.
it's one of those useless terms that no one really knows what it means, like API
We had a DevOps team. After a while, they renamed themselves to SRE. After we got used to the name, they renamed themselves to Production. Their lead even gave a presentation explaining why we should call them Production, not DevOps or SRE. It felt like an identity crisis :)
If you feel like you're useless as a devops engineer, you can install some new UI, like rancher or something. Or you can just limit access to devs so they can have new blockers. 😂
This author could have saved us all some time by saying "Good engineering is hard, time consuming and expensive."
Overall he contradicts himself. He says "DevOps bad", but then says you need people with skills in both areas. 👍🤦♂️
Here's the "secret" of DevOps: it's just good Systems Engineering. We just call it DevOps for the benefit of management types who don't know DNS from a runaway chain reaction.
So much of what he complains about, is a sign that there are architectural issues or that the system has outgrown the staffing levels.
Also, his advocating for "engineers should be able to do whatever the hell they like" is about the most retarded thing I've heard in a loooong time.
Holy sweet Jesus! Have you ever had to support a heterogeneous environment? Ever had to take one over?
Yes, you need to be using the most appropriate tools for a job, but what tools you have *must* be used consistently, so that others can unpick your mess when you leave.
Does this mean that larger organisations become more stiltified? YES, OF COURSE IT DOES, because that's the fucking DEFINITION of a "big business".
Frankly, this guy comes across as "just" a programmer (not least of all because of his "hobbyist" comment!!!), who doesn't like doing serious engineering.
I'm a serious professional, fuck-you-very-much. I'm not a "hobbyist". This shit is HARD and made harder by tools like this that refuse to accept that engineering is about compromise.
Further, devs who have "no interest in the platform" are, by definition, amateurs; F1 drivers know almost as much about the cars as the people that designed and built them. This is the attitude that software engineers *need* to have about platform.
Thank you, well said and makes far better sense. I couldn't see the sense in how the entirety of software was bundled in as inherently hobbyist in nature when there's nothing about what we do that's different to... well, any other profession that takes time to learn and has specializations. Just sounds like someone who thinks that what we do is holy rather than it being a highly technically rigorous profession.
@@HirokuDev Thanks. Exactly.
I was clearly very angry when I wrote this. 🫣😂 I am passionate about my job; no apologies offered. 🤷♂️
All of these “devops is over” articles are sales ads for some deployment templating system or some new “platform”
NGL - this one flew over my head a little bit.
An example can go a long way in explaining things, but this article is only an abstract wall of text.
And I still don't know what ADP means.
Abstract Data Ptype
^ Payroll!
IDP = Internal Developer Platform
opposite of white glove is bare hands.
DevOps is fine tbh. Kubernetes is a great tool but sometimes overkill for small startups but if you are willing to create k8s manifest files and create helm charts, it makes things so much easier to deploy applications to your clusters. There is a lot to handle though and once you bring devsecops in, it immediately fucks up your CI/CD pipeline
Devops, scrum... all those things were created by middle management hoping to tame programmers
fr. I think each product should have an engineer making sure the infrastructure for that product is set in place. By using existing tools or makin new ones. I always found it weird that DevOps is a completely separated team.
Maybe have a team that is Product-agnostic that deals with the infra to deploy the pipelines and tools of each product, but at most that.
Each product has different needs, by having a team that deals with requirements for many projects just because "it's backend/infra stuff" makes their work extra hard, and in my opinion, breaks the organizational structure of the business
DevOps is a perfect example of how our industry fucks everything up. Development and Operations are *two different* teams. DevOps is the practice of these two teams working collaboratively instead of being at odds with each other.
If you've founded a "DevOps Team", you've failed to do even basic studies to learn what DevOps is and deserve every bit of pain and suffering you experience.
Don't get me started on Scrum and how utterly fucked up that is.
My market analyis shows Devops & Kubernetes are actually growing 20% per year, while sometimes an ungrateful position there's a huge positive outlook for people in DevOps & Kubernetes.
@ThePrimeTimeagen can you PLEASE post the links to the articles you talk about in the description?
I was stuck waiting for DevOps to set up a Jenkins slave for over a year. Turns out he just had to fill out a 5 minute form. Management tried escalating this repeatedly during that year.
Ops work sucks and I don’t want to do it. For that reason I’m happy they are there when I have them.
As someone who is a 'security oriented devops' person, I used to build platforms that help speed up the devs, and help automate things for the SoC, and other stuff, I can say that with where my org has gone in the last two years: the people driving us have no idea how to use us, or even understand what we do. Half my team has quit, they're not backfilling, and it feels like drowning. Where we once had a beautiful self service portal for engineering resources, today, I have to ask ops for a vmz and wait ten days. RIP
A lot of developers just cannot be bothered to learn about cloud services or containers.
As someone that comes from a different sector and now entering devops that I have quickly realized that DevOps is a organizational philosophy that the entire company needs to adhere to. You can just hire two devops engineers and hope that will fix things.
I was never fooled. I love programming. I hate DevOps. I see it all as: "broken."
"DevOps" is just another desperate label for "Quality Assurance", AkA "QA". As we all who suffered as "QA engineers" for decades before we finally wised up know, no software company currently in existence gives a flying f---- about whether their product is any good, much less even works at all. They only care that they get their customer's money. Accordingly, no software company currently in existence wants to do QA or fund it in any way at all. So, given that QA is a totally dead-end affair because of the foregoing total apathy about SW "quality", and DevOps is just another name for QA, well...... you know what they say about 2 + 2 equaling 4........
To quote Mike Judge's magnum opus, "Office Space" (1999):
"What would you say you DO here?"
devops named the prod and the QA system PROD and non-PROD. this is terror inducing when you get an alert or are e.g. rebooting QA and read PROD in your peripheral visions
"But companies pay me to FORGET TO TURN OFF ALERTS!" 😂
When you realize you're somehow doing both dev ops team by yourself, and it's shit 😂
Can you add the articles into the description of videos?
DevOps here, people would call DevOps anything but DevOps :))). Platform Engineering is still just DevOps. What dude says works well if you start project from scratch, but let see how well this applies to real world scenario where you have legacy apps, you have to do lift and shift and also have to do training for devs. At the end of the day this dude only wants to sell his platform, which is fine, but yeah DevOps is not a job is a a set of practices. Also to me seems like a business idea: why pay a 'DevOps' guy x-amount of money when you can force a developer to do the work and increase his salary with 10% so he doesnt hate you.
What works best in large companies is to have teams that are made up from devs, qa, devops etc. and each team member is embedded into the team. It's still some kind of silo but not in the traditional way.
Wasn't the idea of DevOps to make the Devs not only responsible for building new stuff, but also for continuity of the application while Ops would not only be responsible for the stability, but also to ease changing it. In our company, it sort of meant that Ops people learned how to code and developers learned responsibility (of course they already knew how to maintain a platform, just had to learn it at scale) I still like it.
The simple issue I have with most of the "Dev Ops" that I did was that the system I was forced to use that was described as a "Simple, opinionated declarative language" was that it forced you to copy pasta code that otherwise would have been simply handed with a variable. For example, for each deployment, I had to create several yaml files that had values embedded in them that had to be the same - stuff that was easily generated with a simple script.
So, so many people saying “X is bad” and they are right because their interpretation of X makes it bad.
DevOps bad? No. The idea op DevOps is that the developer is also responsible for operating it. No one in their right mind says that the developer must design and build the operations infrastructure, tools, procedures. There is no contradiction in doing DevOps AND have engineering teams/IT4IT/IT4DEV/SREs. Developers in DevOps must be given a comfortable platform to use to Operate their software on in development and in production.
The black ink-like texture at the bottom of the screen scared the shit out of me. LMAO :D
Please create a ticket and we will get to it in our monday morning intake... code for leave me alone I am doing something that was dropped on my lap this morning that was previously techdebt and now is a fire. No one gives the ops people high fives unless they are on late getting the release fixed.
Hey! Missing reference link... What article was that?
Devops isn't terrible, cloud isn't terrible, multi-cloud deployment aren't terrible.
Tools for doing these things are terrible.
Using AWS to deploy application is a nightmare, the UI/UX is a nightmare, using YAML for configuration + programming is a nightmare
Using Jenkins for more complex staff is a nightmare.
Maybe I'm to old, but Installing software from floppy disk or CD had better UX then deploying application to AWS.
Don't blame the concept, blame the implementation
The current solution to these problems is rename the concept and use the same bad tools and repeat this every 3-4 years.
This is how agile/TDD/DDD survive for 20+ years
"The current solution to these problems is rename the concept and use the same bad tools and repeat this every 3-4 years."
They are doing it I guess.
Platform Engineer is the new thing. Now Enterprise want to cut the cost and repeat the cycle by rebuilding as self hosted services
DEVOPS IS AWESOME, LOL DIDN"T WATCH AND WITH AD BLOCKER ON.
This is sort of what our DevOps team does. They build a platform and set the templates up to have proper security infra so that the dev teams can easily create what they need in a scalable and secure way.
We control our github actions based on the runners that devops provides with the appropriate iam roles. We can even spin up ephemeral environments for testing including spin up and tear down of databases in an RDS postgres cluster, all automated using strong security and vetted templates.
For me sounds it all like organizational problem, the employees were not prepared well and there was like no real plan for implementation step by step. Normally companies should start from docker and the slowly try out kubernetes, but yes some software vendors can sell Kubernetes very fast, me and my company we always ask our clients what are theirs real goal of achievement , when they have particular tech in mind.
I agree in some sense as devs have to rely on devops personnel to get access to some resources
Sorry but the fact the writer actually wrote a product in the area of Platform engineering and thinks DevOps should be thrown out of the window shows he has ZERO understanding what DevOps is and instead thinks that creating silos / titles changes the role. He has a VERY good reason to push A over B as that is his living. He chooses to wrap his ideas with "nice-words" and promotes his pink glasses.
Platform Engineering uses DevOps practices', DevOps methodologies, DevOps tools. Every single "Platform engineering" topic he mentioned is basically DevOps. The term "platform engineering" has been around for almost 2 decade, and as you mentioned PE is just a new marketing-fused new term to present same ideas we have been having for 2 decades but because of BAD implementations, our indesutry things that it needs to "reinvent" itself, get new titles, new budgets, new silos. If you really want to have a discussion about DevOps @Primeagen, feel free. Our industry really needs to start talking and have good leadership to stop this nonsense of new names every 3 years. it started with cyber, then DevOps, now PE and in 3 years well have something new as AI is shaping up the landscape.
You can talk about any subject without screaming! You woke everyone up
"I wrote some of the trashiest PHP" - "No you just wrote some PHP"
Canaries is something most companies that are really serious about DevOps and CI/Cd (for that reason) cannot do. We as people that generate energy and deliver it to the grid only have 1 nuclear power plant and 1 coal power plant and 3 gas tribune plants. We can’t phase it out. We role it out and it need to work. If we fail, we disrupt the grid, causing power outages and when we are fully inlicht a blackout.
Banking is my other customer we deliver required reports to the ECB and Dutch Bank those are the only two and it needs to be correct! And it needs to be there in time.
And for accounts you can’t have a new feature running on X customers because you can’t role back financial transactions! It’s not allowed by law! This is why you can’t book back a transaction you did. Payments are final. And even according to SOX law you need to be able to prove how any transaction worked at any time. So your audit would and reproduction or evidence will be a hell!
I'm a Devops and I think it is about to game over now. Because I don't see there''s no more openings for the DevOps jobs in my market since the last year. The fate of DevOps role is just like the EA role back in 2 decades ago. The Enterprise Architect thrived only a couple years (2005 - 2010). Then it sank forever. And I saw no more Enterprise Architect jobs available in the market in the past decade anymore. The Devops is on the same path as the EA now. So Devops will be history in a couple years later. Just a waste of time (I earned the title but no future to sustain).
Devops is a methodology, not a role. Every developer is a ‘devops engineer’.
That devOps diagram with the cow and the turd is gold
One time I looked at the dev ops team’s JIRA… never again
As Devops, we'd like to give IAM roles easily, but Security Team is worse :D
I have to work occasionally with CI/CD pipelines that take 10 hours to complete. And it’s constantly failing because of some dependencies. And all I want to do is just add a new region to the deployment config and deploy that stupid services there (don’t ask why we deploy services that aren’t ours, our processes are… weird). Oh when you’re doing this you can block devs working on their stuff, fucking beautiful.
Every time I have to do this, I really want to meet mfs who designed and developed that bullshit.
Oh, the pain, I’m getting mad even thinking about it. CI/CD my ass.
My problem with Devops is that the line responsibly is not clear at my company. I end up having to learn just enough to get the thing done every time I need anything done, but never get enough reps to feel comfortable doing it
"How to get rid of devops team" only hire people with devops experience. Isn't this like hiring a devops team with extra (more expensive) steps?
if i were to define what devops is, i would say it's making the life of contributors easier.
I was writing in php and (ohh my good) yes, it doesn't matter in the end. The main thing is that I'm able to deliver features fast, in reliable way, with only 20 lines deploy script. Im not using kubernetes terraform and so on. Why? Because product owner needs a good product, not terraform or kubernetes. If i can make things in a most simple way, I will do it that way.
Will Prime ever turn off alerts? No... Will it ever get old teasing him about it? Also No
Question: How many of you are doing devops without a college degree? I have no education beyond high school diploma, but my job history is sort of all over the place. I bounced around between C++ developer, IT Support, System Administrator, Python developer, PLC programmer, electronics manufacturing. Always used Linux and have been spinning up VPS to run my own git, mail, websites, etc.
Where do I go, what do I do? Certifications? Should I go to college? Both ideally obviously?
Are you Dr. Disrespects’ brother ? Man you both have almost the same voice, mustache, reaction and humor !!
Devops is what you get when you do OOP. Not a critique of OOP that's just the nature of things. FP makes you keep system design in mind in aess myopic way.
lol