After outages, Amazon to make senior engineers sign off on AI-assisted changes (arstechnica.com)

The article claims:

>He asked staff to attend the meeting, which is normally optional.

Is that false? It also discusses a new policy:

>Junior and mid-level engineers will now require more senior engineers to sign off any AI-assisted changes, Treadwell added.

Is that inaccurate? It is good context that this is a regularly scheduled meeting. But, regularly scheduled meetings can have newsworthy things happen at them.

It didn't seem to make the news but at least in NYC the entire Amazon storefront was broken all afternoon on Friday.

Items weren't displaying prices and it was impossible to add anything to your cart. It lasted from about 2pm to 5pm.

It's especially strange because if a computer glitch brought down a large retail competitor like Walmart I probably would have seen something even though their sales volume is lower.

I am not in that specific meeting but it made me chuckle that a weekly ops meeting will somehow get media attention. It's been an Amazon thing forever. Wait until the public learns about CoEs!

It’s always sobering to see a news story about something you have insider perspective on.

> Feels like the media is making a mountain out of a mole hill here.

That's been their job ever since cable news was invented.

> This meeting happens literally every week, and has for years. Feels like the media is making a mountain out of a mole hill here.

Are you completely missing the point of the submission? It's not about "Amazon has a mandatory weekly meeting" but about the contents of that specific meeting, about AI-assisted tooling leading to "trends of incidents", having a "large blast radius" and "best practices and safeguards are not yet fully established".

No one cares how often the meeting in general is held, or if it's mandatory or not.

This reply chain is confusing but I'm guessing got merged from another thread that had a different title?

Must have as the comments are hours older than OP.

The core message of the article is that Amazon has been having issues with AI slop causing operational reliability concerns, and that seems to be 100% accurate.

This is correct. We ran them on Wednesday’s in Alexa. Jessy actually used to come and sit in ours once a quarter or so when he was running AWS.

What has really happened is that those employees were made into "reverse centaurs":

https://www.theguardian.com/us-news/ng-interactive/2026/jan/...

Who is the media you're accusing here? This is a twitter post. As far as I can tell they do not work a media company.

What is worth being pointed out is how quickly people blame "The Media" for how people use, consume and spread information on social networks.

I believe it is by group - AWS started the weekly operations meeting, effectively every service's oncall from the last week had to attend. Then it grew massive, so they made it optional. Alexa had a similar meeting that tried to replicate what AWS did. A lot of time spent reviewing load tests getting ready for holiday season, prime day, and the superbowl (super bowl ads used to cause crazy TPS spikes for Alexa). And a lot of finger pointing if there was an outage from one team. While it probably did help raise the operational bar, so much time wasted by engineers on busywork/paperwork documenting an error or fix vs improving the actual service.

When I was really early in my career, a mentor told me that code review is not about catching bugs but spreading context (i.e. increasing bus factor.) Catching bugs is a side effect, but unless you have a lot of people review each pull request, it's basically just gambling.

The more expensive and less sexy option is to actually make testing easier (both programmatically and manually), write more tests and more levels of tests, and spend time reducing code complexity. The problem, I think, is people don't get promoted for preventing issues.

Expert reviews are just about the only thing that makes AI generated code viable, though doing them after the fact is a bit sketchy, to be efficient you kinda need to keep an eye on what the model is doing as its working.

Unchecked, AI models output code that is as buggy as it is inefficient. In smaller green field contexts, it's not so bad, but in a large code base, it's performs much worse as it will not have access to the bigger picture.

In my experience, you should be spending something like 5-15X the time the model takes to implement a feature on reviewing and making it fix its errors and inefficiencies. If you do that (with an expert's eye), the changes will usually have a high quality and will be correct and good.

If you do not do that due dilligence, the model will produce a staggering amount of low quality code, at a rate that is probably something like 100x what a human could output in a similar timespan. Unchecked, it's like having a small army of the most eager junior devs you can find going completely fucking ape in the codebase.

> requires an amount of time approaching the time spent if they had just done it themselves

It's actually often harder to fix something sloppy than to write it from scratch. To fix it, you need to hold in your head both the original, the new solution, and calculate the difference, which can be very confusing. The original solution can also anchor your thinking to some approach to the problem, which you wouldn't have if you solve it from scratch.

> For a person (senior or otherwise) to examine code or configuration with the granularity required to verify that it even approximates the result of their own level of experience, even only in terms of security/stability/correctness, requires an amount of time approaching the time spent if they had just done it themselves.

Hell, often it feels slower/worse. Foreign code is easily confusing at first, which slows you down - and bad code quickly gets bewildering and sends you down paths of clarifications that waste time.

Right, code reviews should already have been happening with human written junior code.

If AI is a productivity boost and juniors are going to generate 10x the PRs, do you need 10x the seniors (expensive) or 1/10th the juniors (cost save).

A reminder that in many situations, pure code velocity was never the limiting factor.

Re: idiot prooofing I think this is a natural evolution as companies get larger they try to limit their downside & manage for the median rather than having a growth mindset in hiring/firing/performance.

Seniors are going to need to hold Juniors to a high bar for understanding and explaining what they are committing. Otherwise it will become totally soul destroying to have a bunch of juniors submitting piles of nonsense and claiming they are blocked on you all the time.

It could create the right sort of incentives though. If I'm a junior and I suddenly have to take my work to a senior every time I use AI, I'm going to be much more selective about how I use it and much more careful when I do use it. AI is dangerous because it is so frictionless and this is a way to add friction.

Maybe I don't have the correct mental model for how the typical junior engineer thinks though. I never wanted to bug senior people and make demands on their time if I could help it.

I.e. senior review is valuable, but it does not make bad code good.

I suspect that isn't the goal.

Review by more senior people shifts accountability from the Junior to a Senior, and reframes the problem from "Oh dear, the junior broke everything because they didn't know any better" to "Ah, that Senior is underperforming because they approved code that broke everything."

This is also why I think we will enter a world without Jr's. The time it takes for a Senior to review the Jr's AI code is more expensive than if the Sr produced their own AI code from scratch. Factor in the lack of meetings from a Sr only team, and the productivity gains will appear to be massive.

Whether or not these productivity gains are realized is another question, but spreadsheet based decision makers are going to try.

> Review by a senior is one of the biggest "silver bullet" illusions managers suffer from

Especially in a big co like Amazon, most senior engineers are box drawers, meeting goers, gatekeepers, vision setters, org lubricants, VP's trustees, glorified product managers, and etc. They don't necessarily know more context than the more junior engineers, and they most likely will review slowly while uncovering fewer issues.

Why only AI generated code? I wouldn’t let a junior or mid level developer’s code go into production without at least verifying the known hotspots - concurrency, security, database schema, and various other non functional requirements that only bite you in production.

I’m probably not going to review a random website built by someone except for usability, requirements and security.

The unwritten thing is that if you need seniors to review every single change from junior and mid-level engineers, and those engineers are mostly using Kiro to write their CRs, then what stops the senior from just writing the CRs with Kiro themselves?

I seriously doubt that they think senior reviewers will meticulously hunt down and fix all the AI bugs. Even if they could, they surely don't have the time. But it offers other benefits here:

1. They can assess whether the use of AI is appropriate without looking in detail. E.g. if the AI changed 1000 lines of code to fix a minor bug, or changed code that is essential for security.

2. To discourage AI use, because of the added friction.

> Review by a senior is one of the biggest "silver bullet" illusions managers suffer from.

My manager has been urging us to truly vibe code, just yesterday saying that "language is irrelevant because we've reached the point where it works - so you don't need to see it." This article is a godsend; I'll take this flawed silver bullet any day of the week.

Senior review can definitely help, regardless if the code comes from a junior or an LLM. We've done this since the dawn of time. However, it doesn't scale and since LLM volume far exceeds what juniors can do, you end up overwhelming the seniors, who are normally overbooked anyway.

The other problem is that the type of errors LLMs make are different than juniors. There are huge sections of genuinely good code. So the senior gets "review fatigue" because so much looks good they just start rubber stamping.

I use an automated pipeline to generate code (including terraform, risking infrastructure nukes), and I am the senior reviewer. But I have gates that do a whole range of checks, both deterministic and stochastic, before it ever gets to me. Easy things are pushed back to the LLM for it to autofix. I only see things where my eyes can actually make a difference.

Amazon's instinct is right (add a gate), but the implementation is wrong (make it human). Automated checks first, humans for what's left.

Senior reviews are useful, but as I understand it, Amazon has a fairly high turnover rate, so I wonder just how many seniors with deep knowledge of the codebase they could possibly have.

What a statement at the end. You are absolutely right.

I hear “x tool doesn’t really work well” and then I immediately ask: “does someone know how to use it well?” The answer “yes” is infrequent. Even a yes is often a maybe.

The problem is pervasive in my world (insurance). Number-producing features need to work in a UX and product sense but also produce the right numbers, and within range of expectations. Just checking the UX does what it’s supposed to do is one job, and checking the numbers an entirely separate task.

I don’t many folks that do both well.

>requires an amount of time approaching the time spent if they had just done it themselves.

I would actually say having at least 2 people on any given work item should probably be the norm at Amazon's size if you also want to churn through people as Amazon does and also want quality.

Doing code reviews are not as highly valued in terms of incentives to the employees and it blocks them working on things they would get more compensation for.

I would argue that the amount of time needed for a proper review exceeds the amount of time needed to just do it yourself.

When reviewing, you need to go through every step of implementing it yourself (understand the problem, solve the problem, etc.), but you additionally need to 1) understand someone else’s solution and 2) diff your solution against theirs to provide meaningful feedback.

Review could take roughly equivalent time, but only if I am allowed to reject/approve in a binary way (“my solution would not be the same, therefore denied”) which is not considered appropriate in most places.

This is why I am not a fan of being the reviewer.

The goal of Sr code review is not to make the code better, it's to make the author better.

Deming's point 3 (of 14): Cease dependence on inspection to achieve quality. Eliminate the need for massive inspection by building quality into the product in the first place.

What stops the senior from using AI to review the AI generated code the junior published?

> [...] requires an amount of time approaching the time spent if they had just done it themselves.

Yes, but with the caveat that the junior learns and eventually can become the senior.

Eventually they'll get rid of juniors and mid level devs because realistically it's easier to review when you're the one doing the prompting.

the outcome of the review isn't just that the code gets shipped, it's knowledge transfer from the senior engineer to the junior engineers that then creates more senior engineers

Going to systemically turn off your senior staff over time also. Most Senior Engineers aren't that interested in doing even more code review.

Other than “don’t hire idiots”, what is the solution to this problem? I agree with you, and this particular systems management issue is not constrained to software.

Don’t forget that this auto generated code will have subtle bugs and feels complete at the outset

Reviewing code changes (generally) takes more time than writing code changes for a pretty significant chunk of engineers. If we're optimizing slop code writing at the expense of more senior's time being funneled into detailed reviews then we're _doing it wrong_.

LGTM

Who said PR reviews need to solve all the things and result in proof against idiots?

So you're saying that peer reviews are a waste of time and only idiots would use/propose them?

The thing is, this management philosophy worked when AWS knew what they needed to build and just needed to execute with top notch operations.

But now with AI, they are getting disrupted. Most AWS services might become obsolete, why does an ai need these janky higher levels abstractions AWS piles on.

So now they need innovation, but the company isn’t set up for it. They are forcing short deadlines for product launches that don’t matter

Not only is having too many comments on your PRs bad for you, but so is not leaving comments on other people's PRs. Both are metrics used

> 2. Having Less comments on their PRs: for some drastically dumb reason, having a PR thoroughly reviewed

I'm very far away from liking Amazon's engineering culture and general work culture, but having PRs with countless of discussions and feedback on it does signal that you've done a lot of work without collaborating with others before doing the work. Generally in teams that work well together and build great software, the PRs tend to have very little on them, as most of the issues were resolved while designing together with others.

4. Don't work in the corporate equivalent of The Hunger Games.

if someone responded with a lot of PR comments, I would set up a meeting directly and avoid unnecessary discussion on PR.

Indeed. My view as a CEO is, if you are still reviewing the code yourself then what use is it that you can produce a bunch of text at a faster rate?

I'd prefer people wrote good quality code and checked it as they went along... whilst allowing room for other stuff they didn't think of to come to the front. The production process of using LLMs is entirely different, in its current state I don't see the net benefit.

E.g. if you have a very crystalised vision of what you want, why would I want an engineer to use an LLM to write it, when the LLM can't do both raw production and review? Could this change? Sure. But there's no benefit for me personally to shift toward working that way now - I'd rather it came into existence first before I expose myself to incremental risk that affects business operations. I want a comprehensive solution.

This is what I don't understand about this policy. There's no way a senior has enough spare capacity to be the gate keeper on every PR made by AI below them. So now we are just making it so the senior people use more AI to keep up but now they're to blame for letting it happen.

It sounds like a piss poor deal for seniors unless senior engineer now means professional code reviewer.

Most AI advocates I know believe this period, reviewing every line of code, will come to an end when models improve. So there will be no bottleneck. We will simply test and ship, with AI doing all the code and review.

Surely they know all this. They're worried about AI code degrading codebase quality, so they're putting on the brakes.

> Senior leadership doesn’t know this, though.

Well, you'd think senior leadership should know how their business and their people work.

Yes, underthinking is rampant. Glancing at "AI" output is not reviewing code: you have to grok it (in the Heinlein sense) in order to treat it as your own.

Heck, doing a self review when you wrote the code catches stuff like forgetting debug prints.

We have it in a checklist in PR template. I can’t imagine a fiat class feature that would be much more meaningful. It surprised me to learn there are developers who have to be reminded to review their own code and test it, but does seem to help.

I've been lucky to discover git relatively late and sublime merge relatively soon. It seems like separating the concern of editing and reviewing code is making me consider each more as separate thing.

It also makes me more comfortable figuring out how a project's pull acceptance are like (maybe due to how fast local ui is compared to web-based git). On the other hand, I can only run some basic git cli commands and can't quickly comprehend raw text-based diff, especially when encountering some linux patches from time to time.

If someone was confident enough to push through an AI change without even reading/reviewing it themselves adding more buttons to the UI isn't going to change anything.

TBH, I do PRs on repos with no other devs just do do self-review, and I did that before AI.

the tooling doesnt make it easy currently.

working at amazon, when I wanted to review code myself through the CR tool, Id still end up publishing it to the whole team and have to add some title shenanigans saying it was a self review or WIP and for others to not look at it yet

Self review is #1

> The continuing mass departures of long tenured folks

You mean senior programmers that have been there for ages don't want to spend their time reviewing AI slop? Who'd a thunk it!

So will it turn out that actually writing code was never the time sink in the first place?

That has always been my feeling. Once I really understand what I need to implement, the code is the easy part. Sure it takes some time, but it's not the majority. And for me, actually writing the code will often trigger some additional insight or awareness of edge cases that I hadn't considered.

My prediction is a concorde-like incident is going to shatter trust and make people re-think their expectations of the capabilities of LLMs and their abilities of the present.

Essentially something big has to happen that affects the revenue/trust of a large provider of goods, stemming from LLM-use.

They wont go away entirely. But this idea that they can displace engineers at a high-rate will.

> Mentoring Juniors is an important part of the job and crucial service to the industry

Not really.

As Deming once said in regard to manufacturing inspections: "Inspection does not improve the quality, nor guarantee quality. Inspection is too late. The quality, good or bad, is already in the product."

The fact that software is "soft" makes it seem like this doesn't apply, but it does, not least because of the fact that once you have gone down the wrong path with software design, it is very difficult to pull back and realize you need to go down an entirely different one.

The next thing these geniuses will think of will be to have AI review the diffs.

Check back in a year or two. Just the other day we had Claude finding significant bugs in Firefox. https://news.ycombinator.com/item?id=47273854

It's the same as the offshoring episode of the early 2000's. There is such a massive financial incentive to somehow make the low quality code work. And they will try to resist the reality that it's a huge net negative for as long as they can.

Juniors could just code things the old fashioned way. It isn't hard. And if they do find it too hard, they aren't cut out for this job.

Accelerate a person speed toward being burned out..

But Amazon is something you tolerate for a year or two early in the career, before moving somewhere better (which is anywhere else)?

I'm sorry what? Junior engineers can't learn anything without using AI assistants (or is the implication that having seniors review their code makes them incapable of learning?) and senior engineer would hate their jobs reviewing more code from their teammates? What reality do people live in now?

Its useful but it makes wrong assumptions, not checking the code is essentially gambling.

there's nothing else left to chase apparently

You shouldn't be surprised.

How else would they train the LLM PR reviewers to their standards?

I've never personally been in the position, because my entire career has been in startups, but I've had many friends be in the unenviable position of training their replacements. Here's the thing though, at least they knew they were training their replacements. We could be looking at a potential future where an employee or contractor doesn't realize s/he is actually just hired to generate training data for an LLM to replace them, and then be cut.

Interesting. How would it be an early step towards an apprenticeship system?

It will be written much the same as now, but using AI to improve quality through code inspection etc.

Just because nearly all software is going to be written by AI, does not mean critical infrastructure will be.

Cabin in the woods.

>> I wonder how this will work in practice. Say I'm a senior engineer and I produce myself thousands of lines of code per day with the help of LLMs as mandated by the company.

LOL, it's the age old "responsibility without authority". The pressure to use AI will increase and basically you'll be fired for not using it. Simultaneously with the pressure to take the blame when AI fucks up and you can't keep up with the bullshit, leading you to get fired. One way or the other, get some training on how to stack shelves at the supermarket because that's how our future looks, one way or the other.

Sign-off requirements like this quickly become performative when LLMs generate code faster than anyone can review it in detail. Relying on human oversight at scale is unrealistic unless the volume of changes drops or the review process itself becomes more automated.

This is an important bottleneck. You can have LLM-based reviewers help you. But unless you yourself understood your thousands of lines, it's "somebody else's" code and that somebody else cannot be fired or taken to court.

The presumably human mid-level or junior engineer has their own issues with this, but the point of the LLM is that you don't need that engineer. For productivity purposes, the dev org only needs the seniors to wrangle all the LLMs they can. That doesn't sustain, so a couple of more-junior engineers can do similar work to mature.

Because he doesn’t.

The impression I get from SWEs I’ve met throughout my life is that most of them don’t actually care about their job. They got in because it paid well and demand was plentiful.

Yes, more time on up front spec and plan building. Bite sized specifically to fit within the context window of a single implementation session. Each step should have a verification process that includes new tests.

Prior to each step, I prompt the AI to review the step and ask clarifying questions to fill any missing details. Then implement. Then prompt the AI after to review the changes for any fixes before moving on to the next step. Rinse, repeat.

The specs and plans are actually better for sharing context with the rest of the team than a traditional review process.

I find the code generated by this process to be better in general than the code I've generated over my previous 35+ years of coding. More robust, more complete, better tested. I used to "rush" through this process before, with less upfront planning, and more of a focus on getting a working scaffold up and running as fast as possible, with each step along the way implemented a bit quicker and less robustly, with the assumption I'd return to fix up the corner cases later.

Id maybe consider spinning out a different job role for like "review engineer" who's not busy making strategic decisions and near term planning, just making sure the code is actually good

still within the engineering IC role, but on a different track

makes me want to vomit. I am not spending more time reviewing code than the "author" spent creating it. Ill just leave the industry if that happens.

Code review is mandatory at Amazon

AI will fix it, same way AI wrote it. At the behest of a human.

When they fire everyone, juniors will fix it with AI.

This is in general. I wouldn’t recommend this at critical services like AWS.

[dead]

but thats what it was like when i started at amazon in 2016?

i think the team i was on was a bit of an outlier in terms of owning 40 dumptser fires at once, and the first time reading any one of them was at 2AM because it was down.

having an LLM give early passes on reading the godawful c++ code that you can tell at a glance that its not gonna work as expected, but you cant tell why, or what expected actually is would have been phenomenal, and gotten me back to sleep at 3 on those codebases rather than 5.

You vastly underestimate the complexity of systems in a company like Amazon.

COEs and Operation Readiness Reviews are already the documents that you mention, but they are largely useless in preventing incidents.

code review is too late to give some of that feedback, and design/requirements documents dont have nearly the standardization of presentation and feedback tools for that to be useable.

Amazon does have those things, and has fine tuning on models based on those postmortems.

Noisy reviews are also a problem causer. the PR doesnt know what scale a chunk of code is running at, without having access to 20 more packages and other details.

Why can't they?

Create the problem and then create the solution.

"Why don't they just make the plane out of the black box?"

This is incredibly circular lol...

Emphasis on shit.

You mean like what Anthropic announced yesterday ? Code Review can review your code for $15 - $25 per review.

/s

So now, you can speed up using Claude Code and use Code Review to keep it in check.

a) what happens if there is change that hasn't been encountered yet so it's not in .agentnotallowed? b) is there a guarantee that something described in these files won't be touched? I've seen examples when agents directly violate these rules, profusely apologising after they get caught on it.

LLMs have been garbage for real work until very recently. Doesn't this show they were adopted too soon at amazon?

that isnt computer science at all though? if it is, that would be phd research topics, rather than than basics.

maybe as software engineering topics, but thats a different discipline

“Haha, sure man. CSI, great tv series. Look, just review Jeff’s PR and move that button down, okay? Have a tennis lesson, kbye”

Right? If they're using some sort of tool, there's always another tool to fool the tool.

I used Kiro IDE and really liked it. The all you can eat model of LLM usage is very tempting compared to say Cursor. The features in the editor are basically the same.

Haven't tried Kiro CLI.

The technique of creating specs before implementation that Kiro embodies was used widely internally before Kiro's release, but as a (now former) employee I gained access to the Kiro tool at the same time as the public. Others may have had internal access earlier but I'm not aware of them.

I use Kiro IDE (≠ Kiro CLI) primarily as a spec generator.

In my experience, it's high-quality for creating and iterating on specs. Tools like Cursor are optimized for human-driven vibing -- they have great autocomplete, etc. Kiro, by contrast, is optimized around spec, which ironically has been the most effective approach I've found for driving agents.

I'd argue that Cursor, Antigravity, and similar tools are optimized for human steering, which explains their popularity, while Kiro is optimized for agent harnesses. That's also why it’s underused: it's quite opinionated, but very effective. Vibe-coding culture isn't sold on spec driven development (they think it's waterfall and summarily dismiss it -- even Yegge has this bias), so people tend to underrate it.

Kiro writes specs using structured formats like EARS and INCOSE. It performs automated reasoning to check for consistency, then generates a design document and task list from the spec -- similar to what Beads does. I usually spend a significant amount of time pressure-testing the spec before implementing (often hours to days), and it pays off. Writing a good, consistent spec is essentially the computer equivalent of "writing as a tool of thought" in practice.

Once the spec is tight, implementation tends to follow it closely. Kiro also generates property-based tests (PBTs) using Hypothesis in Python, inspired by Haskell's QuickCheck. These tests sweep the input domain and, when combined with traditional scenario-based unit tests, tend to produce code that adheres closely to the spec. I also add a small instruction "do red/green TDD" (I learned this from Simon Willison) and that one line alone improved the quality of all my tests.

Kiro can technically implement the task list itself, but this is where agents come in. With the spec in hand, I use multiple headless CLI agents in tmux (e.g., Kiro CLI, Claude Code) for implementation. The results have been very good. With a solid Kiro spec and task list, agents usually implement everything end-to-end without stopping -- I haven’t found a need for Ralph loops. (agents sometimes tend to stop mid way on Claude plans, but I've never had that happen with Kiro, not sure why, maybe it's the checklist, which includes PBT tests as gates).

Kiro didn't have the strongest start, but the Kiro IDE is one of the best spec generators I've used, and it integrates extremely well with agent-driven workflows.

To make the AI companies money and prop up the American economy, obviously.

This was always the case in the before times with humans. AI just pulls back the curtain on the delusion that code can and is being meaningful reviewed.

A lot of juniors only graduated using these tools. Good luck taking it away from them.

hardly.

as an alternative, a bunch of people got into their one-person trucks and drove to the store to buy whatever thing would have been efficiently delivered