DevOps Paradox

#362: Feature flags or canary deployments - do you need both? Viktor puts it to Alex Casalboni from Unleash, who says he argues about this with his colleagues roughly every day, and the answer lands clean. Switching a hostname, a database, an API vendor? That is infrastructure, nothing to do with who the user is, so keep your canaries and your blue-green. But a canary switches one thing at a time. Try running three A/B tests and ten behavioral changes through it and the whole approach buckles. Anything that needs to know who the user is belongs in a flag. Different layers of the stack, different tools, and most teams will end up with both whether they planned to or not.
Back up, though, because there is a new word attached to all of this. FeatureOps. There is a manifesto and everything, sitting at [featureops.io](https://featureops.io/), reading a lot like someone nailed 95 theses about feature flags to a door. Real discipline, or marketing wrapper? Alex gets about ten seconds of pleasantries before he has to answer for the word. His defense is narrower than the name suggests, and better for it: every ops discipline we have gets you to the deployment and then waves goodbye. Something breaks, you go around the whole loop again - hotfix, pipeline, 20 or 30 or 60 minutes, fingers crossed. FeatureOps is the claim that the same principles apply after the code is already running. Runtime control. Alex says enterprise customers routinely have a 12 to 24-hour round trip between finding a problem and getting the fix live. Even for a hotfix.
Viktor is not letting the seconds claim through unchallenged. If it takes you a day to notice and two seconds to flip, that is a day and two seconds - so stop measuring from the convenient starting line. Alex concedes the framing and then goes somewhere better with it: the bottleneck was never the clicking. It is the humans and the bureaucracy in between. Which is why Unleash is pushing impact metrics, where the SDK sends error rates back and the system kills the feature itself, no human in the loop. Then Darin calls BS on immutable event log, because there is no such thing as immutable data, and Alex takes the hit cleanly - fair, it is append-only with locked-down keys, not magic.
Nobody puts this part on a landing page. Flag evaluation has an input, not just a true/false output, and that input is user context - which means an external API call is not just latency, it is your PII leaving the perimeter. A compliance problem hiding inside a performance decision. And the flag graveyard is worse than you think: companies create roughly ten flags for every one they clean up, and Alex has a customer whose oldest flag dates to 2012. His fix is an MCP server that opens the cleanup PR for you when you mark a release complete. Best line of the day, on whether flags complicate your code: everything complicates your code, and the best way to not complicate your code is to not code.
 
Alex's contact information:
LinkedIn: https://www.linkedin.com/in/alexcasalboni/
X: https://x.com/alex_casalboni
 
YouTube channel:
https://youtube.com/devopsparadox
 
Review the podcast on Apple Podcasts:
https://www.devopsparadox.com/review-podcast/
 
Slack:
https://www.devopsparadox.com/slack/
 
Connect with us at:
https://www.devopsparadox.com/contact/

#361: Picture a 6,000-line pull request landing in your project from someone you've never heard of. Every test passes - yours, theirs, all of it. And you can tell it was generated. Not assisted. Generated. What do you do with it? Viktor's first move is to poke the premise: how do you even know it was generated and not assisted? You can't. Nobody can. That distinction is already gone.
So the real question isn't whether AI wrote it. It's what a maintainer is for. Viktor's answer is blunt - a maintainer's first job is to guide people in and help them contribute, and if you think your job is mainly to write code, you picked the wrong role. The moment you allow PRs, you stopped being an individual contributor and became a manager. That's the job. You don't get to complain about the job you signed up for. Don't want it? Fine - do what Ghostty did, do what curl did, turn PRs off and say so out loud. Just don't hide behind unknown contributors, because every single person on your project was an unknown contributor on their first commit, including you.
A software developer says send me anything but a PR and lists four reasons: unknown contributors are a security risk, supply chain attacks are real, style disagreements eat maintainer time across time zones, and LLMs killed code-writing as the bottleneck so the stranger's PR doesn't help with the parts that are still hard. Viktor grants the security point and then points at XZ - social engineering, a long con, zero AI required. The risk was always there. What changed is quantity, not the percentage. And the maintainer who thinks a manual, line-by-line review still works in 2026 is, in his words, terribly wrong.
Viktor guesses he can review 6,000 lines in about the time it takes to hand-review 600. CodeRabbit and Greptile clear the obvious junk so he can spend his attention on architecture and the feature itself - the stuff he never had energy for after slogging through nitpicks by hand. His read on the whole backlash: there's a new third group of maintainers now, the ones who aren't good enough with agents to fight agents, and they're the ones falling behind. Writing code is cheap. Reviewing it well is the expensive part - and the ones who refuse to use agents to review are drowning while blaming the contributors. There's a cost angle too. If you don't have tokens, you're in trouble, and not everyone can afford them. Tokens are becoming table stakes, like an internet connection. Which raises the question: what happens to open source when the reviewing tools cost money the maintainer doesn't have?
 
I don't want your PRs anymore
https://dpc.pw/posts/i-dont-want-your-prs-anymore/
 
YouTube channel:
https://youtube.com/devopsparadox
 
Review the podcast on Apple Podcasts:
https://www.devopsparadox.com/review-podcast/
 
Slack:
https://www.devopsparadox.com/slack/
 
Connect with us at:
https://www.devopsparadox.com/contact/

#360: Assume your production systems go sideways at 3:00am. Your phone doesn't ring. You wake up to a clean incident report - the system caught the problem, fixed itself, and left a Slack note saying it handled things at 3:15. Would you trust it? Viktor wouldn't, not at first. Same as a new hire, a new tool, a new anything. Trust gets earned. A few months in, if it hasn't messed up, then maybe.
So what's actually new? We had self-healing in decades past, and Kubernetes brings a crashed pod back without anyone awake to watch. Birol Yildiz, co-founder and CEO of ilert, puts the line where it belongs: the old stuff runs on predefined patterns - known problem, known fix, automate it. The new part is diagnosing a root cause nobody wrote a rule for and then knowing what to do about it. And no, ilert isn't training models on your data. The agent runs on foundational models from the frontier labs, feeds them the same context any SRE would pull - telemetry, recent changes, the health of your infra - runs the investigation loop, and hands you a root cause with the evidence attached.
Then the real argument starts, and it's about knowledge. Viktor keeps circling the blind spot he sees at every company: the wiki, the runbooks, the tribal stuff nobody keeps current. Birol's answer is blunter than you'd expect from a guy selling this - dump your Confluence and your hundred stale runbooks into the agent and you cause more damage than results. Code is the documentation that's actually true. ilert's agent doesn't even read runbooks. It builds a live service topology straight from tracing data and rebuilds it when things drift.
What happens when a bad incident month turns your $100 bill into $10,000 - Birol's answer is a credit budget with a human paged when you blow past it. Who's accountable when the agent makes it worse? Has to be a human. Agents are tools. You don't get to blame your AI coding agent for shipping garbage code, and ilert doesn't get to blame the model either. Trust is binary - five good runs and one bad one, and you strip its write access. But even in observe-only, the value is real, because the time sink in an incident was never the fix, it was finding the needle. Cut root cause from 45 minutes to a few. And the closer nobody wants to hear: if you skipped observability and fifteen years of homework, an AI SRE isn't going to save you. You can't skip the parts you skipped.
 
Birol's contact information:
LinkedIn: https://www.linkedin.com/in/birolyildiz/
X: https://www.linkedin.com/in/birolyildiz/
 
YouTube channel:
https://youtube.com/devopsparadox
 
Review the podcast on Apple Podcasts:
https://www.devopsparadox.com/review-podcast/
 
Slack:
https://www.devopsparadox.com/slack/
 
Connect with us at:
https://www.devopsparadox.com/contact/

#359: When was the last time you sat through a 30-minute product demo and walked away actually knowing anything? You would learn more from five minutes hands-on than an hour of watching someone else drive.
Now you have help. An agent can watch the 30-minute video, play in the sandbox, read every page of the docs, and come back before you finish your coffee with a verdict - tried it, does not work, next. The agent is the new tire kicker. So if you are a vendor, an open source maintainer, or the person building the internal app nobody outside the building ever sees, the demo you have been giving is aimed at a buyer who already left the room.
Your job now is to make life easier for agents. An MCP server, a CLI, skills, an AGENTS.md file, not blocking your own site with Cloudflare when someone's agent tries to read your pricing. Everything that makes a product easy for an agent would have made it easier for a human all along. We just never bothered, because we had months to burn. Now the clock runs in minutes and every corner we cut is suddenly on fire.
Three kinds of demo, three different answers. The vendor sales demo is off-putting before it starts - if a website says book a call to try it, Viktor is already gone. Open source barely needs a demo at all: a good README, a quick start, an AGENTS.md, and the agent assembles a demo tailored to your stack, your database, your questions, instead of some generic happy path. Internal is where it gets good, and it might be the one that matters most since exactly zero apps ship without customization. Viktor's bar: stop showing me plans, show me the thing running. Sit the stakeholder down and build it live while you talk. Three days to a prototype instead of 300 pages of PRD.
Sandboxes first, demos second - if you cannot spin up a sandbox, you did not build it right. And demo the failure modes, not the happy path, because resiliency is the real selling point now. Disks still fill up. No amount of AI magic empties them for you.
 
YouTube channel:
https://youtube.com/devopsparadox
 
Review the podcast on Apple Podcasts:
https://www.devopsparadox.com/review-podcast/
 
Slack:
https://www.devopsparadox.com/slack/
 
Connect with us at:
https://www.devopsparadox.com/contact/

#358: Production is on fire. You need access to one table you have never touched. So you file an access request, then phone the desk to say you filed it, then Slack them to say you phoned, then walk over to say you Slacked. Twenty-five minutes later the incident has resolved itself and the customer has already left.
That is the setup, and Ofir Stein has lived the other side of it. He is the CTO and co-founder of Apono, and before that he was an engineering leader who felt the same pain every day - not because he hated security, but because he hated being blocked. There is a difference, and the whole conversation turns on it. Put productivity on one side, security risk on the other, and access management in the middle. Tighten one and you starve the other. Nobody wants to be slower and nobody wants to be breached, so the honest answer is there is no clean answer.
Then AI agents show up and break the last assumption standing. Software used to be deterministic - your computer could not decide to do something other than what it was told. LLMs can. They can be socially engineered the way people are. Ofir's team built a full AWS environment run by AI agents, opened a Discord channel, and invited anyone to try to trick them. People could. That is the new attack surface, and it moves at machine speed - far too fast for the access reviews and approval chains built for humans.
The guardrails everyone is now scrambling to build for agents should have been there for humans all along. Access is the one thing in your stack that never went dynamic. Servers scale up and down, pipelines rebuild everything, and then access is a static policy someone set two weeks ago when security sat with your manager and guessed what you would need. That is the opposite of how the rest of DevOps works. Ofir's argument is that access should change with context - who you are, whether you are on call, whether there is an open incident - evaluated in real time. For a human that is a faster request. For an AI agent, the decision has to live inside the loop, made by silicon, because no person can approve thousands of operations a minute.
If access is per-operation and every operation is already a specific API call, what is left to scope? If the business context changes by the minute, how do you write guardrails in advance? And once the human is out of the loop, are you not just left with one AI deciding what another AI is allowed to do? Ofir does not pretend that part is solved. What he is sure of is the direction: the doors at the mall open when you walk up and close when you leave, and you never think about them. That is where access is headed - and there is a lot of road between here and there.
 
Ofir's contact information:
LinkedIn: https://www.linkedin.com/in/ofir-stein/
 
YouTube channel:
https://youtube.com/devopsparadox
 
Review the podcast on Apple Podcasts:
https://www.devopsparadox.com/review-podcast/
 
Slack:
https://www.devopsparadox.com/slack/
 
Connect with us at:
https://www.devopsparadox.com/contact/