cross-posted from: https://lemmy.ca/post/61948688
Excerpt:
“Even within the coding, it’s not working well,” said Smiley. “I’ll give you an example. Code can look right and pass the unit tests and still be wrong. The way you measure that is typically in benchmark tests. So a lot of these companies haven’t engaged in a proper feedback loop to see what the impact of AI coding is on the outcomes they care about. Lines of code, number of [pull requests], these are liabilities. These are not measures of engineering excellence.”
Measures of engineering excellence, said Smiley, include metrics like deployment frequency, lead time to production, change failure rate, mean time to restore, and incident severity. And we need a new set of metrics, he insists, to measure how AI affects engineering performance.
“We don’t know what those are yet,” he said.
One metric that might be helpful, he said, is measuring tokens burned to get to an approved pull request – a formally accepted change in software. That’s the kind of thing that needs to be assessed to determine whether AI helps an organization’s engineering practice.
To underscore the consequences of not having that kind of data, Smiley pointed to a recent attempt to rewrite SQLite in Rust using AI.
“It passed all the unit tests, the shape of the code looks right,” he said. It’s 3.7x more lines of code that performs 2,000 times worse than the actual SQLite. Two thousand times worse for a database is a non-viable product. It’s a dumpster fire. Throw it away. All that money you spent on it is worthless."
All the optimism about using AI for coding, Smiley argues, comes from measuring the wrong things.
“Coding works if you measure lines of code and pull requests,” he said. “Coding does not work if you measure quality and team performance. There’s no evidence to suggest that that’s moving in a positive direction.”
Too many people are willingly paying anti-democratic billionaires to outsource their thinking and agency.
Too many people know their job is only going to last 6 months before the next round of layoffs, and that talent and hard work has never been the way to keep a job in the tech industry… so why try?
Not really a valid excuse in this case as we aren’t really experiencing layoffs here. Au contraire, our company is hiring. I’m not in the U.S.
Still think that letting language models controlled by billionaire paedophiles and wannabe dictators is a poor idea, regardless of how fed up one is with one’s job.
Where is “here?”
And, if you want to bring pedophiles into it, most of what you touch on a daily basis involved a billionaire pedophile at some point. You just sound lazy at this point.
Did you just call them lazy after making the argument that building talent and checks notes hard work is not worth doing in the modern tech landscape?
The argument is lazy to blanket everything in “pedophile” instead of actually talking about the issue.
Oh absolutely, and we can do our best to swear off of that but thanks to them worming their way in like a cancer in every part of society, shaping it to benefit them, that’s just the nature of taking part in society. The ones in power have always, and will always continue to exploit us for as long as we let them.
All the more reason to not outsource our thinking to their machines. Governments are already doing it, getting caught red-handed acting on reports that never existed. Why rely on that when the option not to is so readily available?
Ehhh… this sounds more like blanket ai-hate and less about you actually caring. You’re already in their cloud. I doubt you run bare metal. You probably use GitHub. Etc. caring on one hand and not on the other means nothing.
I’ll continue farming out bullshit tasks to AI while I play with my cat and prepare for the next round of layoffs, rather than giving my soul to a company who doesn’t actually care about me.