Thats not really how open source works. If you use an open source tool like say, nano. It has been looked at and improved for many years by many people who have worked up an understanding of the code.
I realize that this can only be natively understood by a programmer.
What we (I) do when we work at open source projects is reading through the code for so long until we “get it”. It means we start to understand what does what. If you want so change something, you must locate it, finding out what it is not. The chance that someone stumbles across something that then sparks a full blown investigation isnt that low. Of course you can hide something in extremely long and boring code but its alas automatically tested by most software shops.
In short: we dont do this since yesterday and opeb source is so many universes better than closed source is a truth that only a fool would disregard.
So you’re a programmer yourself. That helps me understand where you are coming from. Thanks for clarifying.
As a programmer, you know that you need to depend on the work of others. Otherwise you cant use libraries at all. Of course the libraries are only as good as their own people. But the important part here is that the library doesnt have a makefile for example, which renders your former argument moot. They are often included in huge projects which themselves both have automated and manual reviews.
Somehow I dont believe you have experience in foss programming, at least not in larger projects. Tons of stuff is being done which ensures tons of eyes go over every bit of code, over time. for example in kodi, I have to depend on the upstream people doing their work. they have upstream themselves, etc. All of this is reviewed over and over and over again.
Also, leftpad is a prime example of how you are completely unable to do your thing in a cooperative. you will always get shut down. maybe not immediately but eventually.
My former argument? You might be confusing who you are talking to, since you answered to my first post in this thread.
You also seem to remember leftPad wrong. What happened there was that someone made a tiny library that did nothing but to pad a string. Something so trivial that any programmer should be able to do that within a minute. But still tens of thousands of projects, even large and important libraries, would rather add a whole dependency just to save writing a line of code. In fact, in most dependency management systems it requires more characters to add that dependency than to write that oneliner yourself.
The issue with leftpad was that the maintainer of that “library” was angry for unrelated reasons and pulled all his libraries, which then broke thousands of projects and libraries because leftpad wasn’t available any more.
My point was that everyone just relies on upstream doing their stuff and hardly anyone bothers to check that the code they include is actually doing what it should. And everyone just hopes that someone else already did their job of reviewing upstream, because they can’t be bothered to do it themselves.
A better example though would be Heartbleed. OpenSSL is used in everything. It’s one of the core libraries for modern online communication. Everyone and their grandma used it, most distros, all the cloud providers and so on. Everyone has been making money using the security that OpenSSL provides. Yet OpenSSL was massively underfunded with only one permanent developer who was also underpaid for what he was doing. And apparently nobody thoroughly reviewed the OpenSSL code. Somehow in version 1.0.1 someone made a mistake and added the Heartbleed bug. Stuff like that happens, nobody’s perfect, and if there’s only one person working on this, mistakes are bound to happen.
And then this massive security vulnerability just stayed in there for over two years, allowing anyone to read out whatever’s in the memory of any server using OpenSSL. Because nobody of the billions of people using OpenSSL daily actually reviewed and analysed their code. Because “so many people use OpenSSL, someone surely already reviewed it”.
Or take Log4Shell. That’s a bug that was so trivial it was even documented behaviour. To find this, someone wouldn’t even have had to review the code, just reviewing the documentation of Log4J would have been enough. And still this one was in production code for 8 years. For a library that’s used in almost every Java program.
Nobody reviews upstream.
If upstream makes a mistake, that mistake is in the code. And then everyone just happily consumes what they get.
And upstream is often just a random library thanklessly maintained by some dude in their spare time.
Edit: Just to prove my point: Think of your last big FOSS project that you worked on. Can you list every single dependency and every single transient dependency that your project uses? For each of these dependencies, do you know who maintains it and how many people work on each of these dependencies? Do you know if everyone of these people is qualified and trustworthy enough to put reliable and secure code in your project? Or do you, like everyone else, just hope that someone else made sure it’s all good?
You talk as though closed-source developers reviewed all the upstream code. The exact same problem exists with closed-source, except there isn’t even the possibility of reviewing all the code if you want to. At worst, the lack of review in FOSS projects is on par with closed-source projects. At best, it’s a much smaller problem .
That’s definitely a problem with every bit of code, that everyone relies on stuff they don’t or can’t review.
My point is that FOSS provides a false sense of security (“Millions of people use this library. Someone will already have reviewed it.”).
But the bigger issue is that FOSS is massively underfunded. If OpenSSL was for-profit, it would be a corporate project with dozens if not hundreds of developers. Nobody would buy a piece of core security infrastructure from a self-employed dude working away in his basement. That would be ridiculous to even think about that. And if this standard component was for-profit, even very low license fees would generate huge amounts of revenue (because it’s used in so many places) and this would allow for more developers to be employed.
And since it would be an actual thing that companies would actually buy, they’d demand that third-party security audits of the software would be done, like on any paid-for software that companies use. They’d also demand some SLA contracts that would hold this fictional for-profit OpenSSL accountable for vulnerabilities.
But since it’s FOSS, nobody cares. Companies just use it, nobody donates. It’s for free, so the decision to use it usually doesn’t even go through procurement and anything related to it. I tried to get my old company to donate to OpenSSL in the wake of Heartbleed, and the company said they don’t have a process to donate to something, so can’t be done.
So everyone just uses this little project created by one solitary hero and nobody pays for it. And so that dude works alone in his basement, with literally the online security of the whole world resting on his shoulders.
Luckily after Heartbleed a lot of large corporations started to donate to OpenSSL, but there are hundreds of other equally important projects that still nobody cares about. As seen e.g. with the .xz near miss.
If OpenSSL was for-profit, it would be a corporate project with dozens if not hundreds of developers
It seems like you don’t have a very broad exposure to closed source development. Corporations frequently have a skeleton crew working on a component or entire project. You might notice if you get escalated to development enough that it’s always like the same guy or two. It’s because they might only have a couple of guys working on it. Some companies will spend more on measures to obfuscate that reality than they would spend on actually developing. Certainly some corp closed source projects are that big, but so too are many open source projects.
Hell I’ve dealt with financial institutions using proprietary software that was abandoned by their vendor 15 years prior (came up because the software no longer worked with new stuff, and the institutions demanded wrapper software for new stuff to imitate the old stuff enough to keep using the unmaintained, unpatched, zero developer project).
I also don’t think companies are holding the proprietary vendors to quite the standard you imagine, certainly not automatically. By the same logic you propose for open source “someone else must have done it”, you also have that for big companies, if not more so. “Surely they have good security practices” or “it’s so popular someone must have done that”.
It seems like you don’t have a very broad exposure to closed source development.
Probably not. 15 years is not that long, what do I know, I’m just on senior expert level.
Companies run skeleton crews on crap products that don’t make money. Stuff they give away for free or that’s only used by legacy customers. Stuff they can’t shutdown because of contracts or because it still making a bit of money.
You might notice if you get escalated to development enough that it’s always like the same guy or two. It’s because they might only have a couple of guys working on it.
This is where your lack of knowledge about products like that shines through. It’s common to only get the same guy or two, because that’s the people designated (or willing) to talk to customers.
In real life, OpenSSL was run by a single person. That’s not a skeletton crew, that’s abandonment.
From what you are writing you aren’t a programmer and you haven’t worked in a software corporation before, but instead just extrapolate from your experiences with customer support.
Probably not. 15 years is not that long, what do I know, I’m just on senior expert level.
Longevity is not a guarantee of broad exposure. It may mean you have deep exposure, but making the rounds around the industry I can’t imagine maintaining such a universally optimistic picture of commercial management of software development.
Companies run skeleton crews on crap products that don’t make money.
Companies run skeleton crews on products when they think they can get away with it. Very high profile commercial projects with a lot of analyst attention may not be able to get away with it, but some surprisingly high profile projects without quite as much scrutiny get away with more than you would guess.
This is where your lack of knowledge about products like that shines through.
I’m speaking from familiarity with the provider side of things, wondering when a customer will catch on that they can’t seem to get that awesome support unless it’s the same guy as their peers get, and suspiciously unable to get decent support for a random week in June or something.
From what you are writing you aren’t a programmer and you haven’t worked in a software corporation before
Incorrect assumption on both counts. A few companies across a couple of decades and two of those companies extensively engaging with other companies on projects to get me some exposure to closed source development organizations even at some other companies.
I saw the code. It was pretty obvious once you look at that particular piece. You have to adapt the makefile pretty often so you also would see gibberish. If you’re a programmer and you encounter what YOU think is gibberish, all alarms go off.
i dont know your experience in coding but I dont see how a huge number (a given with old and popular code) of experienced people could overlook something like this.
But this is the crucial thing. It wasn’t in the repository. It was in the tarball. It’s a very careful distinction because, people generally reviewed the repository and made the assumption that what’s there, is all that matters.
The changes to the make process only being present in the tarball was actually quite an ingenius move. Because they knew that the process many distro maintainers use is to pull the tarball and work from that (likely with some automated scripting to make the package for their distro).
This particular path will probably be harder to reproduce in the future. Larger projects I would expect have some verification process in place to ensure they match (and the backup of people independently doing the same).
But it’s not to say there won’t in the future be some other method of attack the happens out of sight of the main repository and is missed by the existing processes.
Thats not really how open source works. If you use an open source tool like say, nano. It has been looked at and improved for many years by many people who have worked up an understanding of the code.
I realize that this can only be natively understood by a programmer.
What we (I) do when we work at open source projects is reading through the code for so long until we “get it”. It means we start to understand what does what. If you want so change something, you must locate it, finding out what it is not. The chance that someone stumbles across something that then sparks a full blown investigation isnt that low. Of course you can hide something in extremely long and boring code but its alas automatically tested by most software shops.
In short: we dont do this since yesterday and opeb source is so many universes better than closed source is a truth that only a fool would disregard.
Are you sure?
All I’m saying is leftPad, if you still remember.
As a programmer I do not believe you when you claim that you read through all the code of all the libraries you include.
Especially with more hardcore dependencies (like OpenSSL), hardly anyone reads through that.
So you’re a programmer yourself. That helps me understand where you are coming from. Thanks for clarifying.
As a programmer, you know that you need to depend on the work of others. Otherwise you cant use libraries at all. Of course the libraries are only as good as their own people. But the important part here is that the library doesnt have a makefile for example, which renders your former argument moot. They are often included in huge projects which themselves both have automated and manual reviews.
Somehow I dont believe you have experience in foss programming, at least not in larger projects. Tons of stuff is being done which ensures tons of eyes go over every bit of code, over time. for example in kodi, I have to depend on the upstream people doing their work. they have upstream themselves, etc. All of this is reviewed over and over and over again.
Also, leftpad is a prime example of how you are completely unable to do your thing in a cooperative. you will always get shut down. maybe not immediately but eventually.
Thats why foss is the ultimately better system.
My former argument? You might be confusing who you are talking to, since you answered to my first post in this thread.
You also seem to remember leftPad wrong. What happened there was that someone made a tiny library that did nothing but to pad a string. Something so trivial that any programmer should be able to do that within a minute. But still tens of thousands of projects, even large and important libraries, would rather add a whole dependency just to save writing a line of code. In fact, in most dependency management systems it requires more characters to add that dependency than to write that oneliner yourself.
The issue with leftpad was that the maintainer of that “library” was angry for unrelated reasons and pulled all his libraries, which then broke thousands of projects and libraries because leftpad wasn’t available any more.
My point was that everyone just relies on upstream doing their stuff and hardly anyone bothers to check that the code they include is actually doing what it should. And everyone just hopes that someone else already did their job of reviewing upstream, because they can’t be bothered to do it themselves.
A better example though would be Heartbleed. OpenSSL is used in everything. It’s one of the core libraries for modern online communication. Everyone and their grandma used it, most distros, all the cloud providers and so on. Everyone has been making money using the security that OpenSSL provides. Yet OpenSSL was massively underfunded with only one permanent developer who was also underpaid for what he was doing. And apparently nobody thoroughly reviewed the OpenSSL code. Somehow in version 1.0.1 someone made a mistake and added the Heartbleed bug. Stuff like that happens, nobody’s perfect, and if there’s only one person working on this, mistakes are bound to happen.
And then this massive security vulnerability just stayed in there for over two years, allowing anyone to read out whatever’s in the memory of any server using OpenSSL. Because nobody of the billions of people using OpenSSL daily actually reviewed and analysed their code. Because “so many people use OpenSSL, someone surely already reviewed it”.
Or take Log4Shell. That’s a bug that was so trivial it was even documented behaviour. To find this, someone wouldn’t even have had to review the code, just reviewing the documentation of Log4J would have been enough. And still this one was in production code for 8 years. For a library that’s used in almost every Java program.
Nobody reviews upstream.
If upstream makes a mistake, that mistake is in the code. And then everyone just happily consumes what they get.
And upstream is often just a random library thanklessly maintained by some dude in their spare time.
Edit: Just to prove my point: Think of your last big FOSS project that you worked on. Can you list every single dependency and every single transient dependency that your project uses? For each of these dependencies, do you know who maintains it and how many people work on each of these dependencies? Do you know if everyone of these people is qualified and trustworthy enough to put reliable and secure code in your project? Or do you, like everyone else, just hope that someone else made sure it’s all good?
You talk as though closed-source developers reviewed all the upstream code. The exact same problem exists with closed-source, except there isn’t even the possibility of reviewing all the code if you want to. At worst, the lack of review in FOSS projects is on par with closed-source projects. At best, it’s a much smaller problem .
That’s definitely a problem with every bit of code, that everyone relies on stuff they don’t or can’t review.
My point is that FOSS provides a false sense of security (“Millions of people use this library. Someone will already have reviewed it.”).
But the bigger issue is that FOSS is massively underfunded. If OpenSSL was for-profit, it would be a corporate project with dozens if not hundreds of developers. Nobody would buy a piece of core security infrastructure from a self-employed dude working away in his basement. That would be ridiculous to even think about that. And if this standard component was for-profit, even very low license fees would generate huge amounts of revenue (because it’s used in so many places) and this would allow for more developers to be employed.
And since it would be an actual thing that companies would actually buy, they’d demand that third-party security audits of the software would be done, like on any paid-for software that companies use. They’d also demand some SLA contracts that would hold this fictional for-profit OpenSSL accountable for vulnerabilities.
But since it’s FOSS, nobody cares. Companies just use it, nobody donates. It’s for free, so the decision to use it usually doesn’t even go through procurement and anything related to it. I tried to get my old company to donate to OpenSSL in the wake of Heartbleed, and the company said they don’t have a process to donate to something, so can’t be done.
So everyone just uses this little project created by one solitary hero and nobody pays for it. And so that dude works alone in his basement, with literally the online security of the whole world resting on his shoulders.
Luckily after Heartbleed a lot of large corporations started to donate to OpenSSL, but there are hundreds of other equally important projects that still nobody cares about. As seen e.g. with the .xz near miss.
It seems like you don’t have a very broad exposure to closed source development. Corporations frequently have a skeleton crew working on a component or entire project. You might notice if you get escalated to development enough that it’s always like the same guy or two. It’s because they might only have a couple of guys working on it. Some companies will spend more on measures to obfuscate that reality than they would spend on actually developing. Certainly some corp closed source projects are that big, but so too are many open source projects.
Hell I’ve dealt with financial institutions using proprietary software that was abandoned by their vendor 15 years prior (came up because the software no longer worked with new stuff, and the institutions demanded wrapper software for new stuff to imitate the old stuff enough to keep using the unmaintained, unpatched, zero developer project).
I also don’t think companies are holding the proprietary vendors to quite the standard you imagine, certainly not automatically. By the same logic you propose for open source “someone else must have done it”, you also have that for big companies, if not more so. “Surely they have good security practices” or “it’s so popular someone must have done that”.
Probably not. 15 years is not that long, what do I know, I’m just on senior expert level.
Companies run skeleton crews on crap products that don’t make money. Stuff they give away for free or that’s only used by legacy customers. Stuff they can’t shutdown because of contracts or because it still making a bit of money.
This is where your lack of knowledge about products like that shines through. It’s common to only get the same guy or two, because that’s the people designated (or willing) to talk to customers.
In real life, OpenSSL was run by a single person. That’s not a skeletton crew, that’s abandonment.
From what you are writing you aren’t a programmer and you haven’t worked in a software corporation before, but instead just extrapolate from your experiences with customer support.
Longevity is not a guarantee of broad exposure. It may mean you have deep exposure, but making the rounds around the industry I can’t imagine maintaining such a universally optimistic picture of commercial management of software development.
Companies run skeleton crews on products when they think they can get away with it. Very high profile commercial projects with a lot of analyst attention may not be able to get away with it, but some surprisingly high profile projects without quite as much scrutiny get away with more than you would guess.
I’m speaking from familiarity with the provider side of things, wondering when a customer will catch on that they can’t seem to get that awesome support unless it’s the same guy as their peers get, and suspiciously unable to get decent support for a random week in June or something.
Incorrect assumption on both counts. A few companies across a couple of decades and two of those companies extensively engaging with other companies on projects to get me some exposure to closed source development organizations even at some other companies.
That was feature creep galore. Ffs…
OpenSSL does have makefiles (or perl scripts if you will).
Even as a library? That sounds horrific.
That’s assuming the attacker is stupid enough to put the exploit in the source code where it can be easily discovered.
The Xz exploit was not present in the source code.
It was hidden in the makefile as an obfuscated string and injected into the object file during the build process.
I saw the code. It was pretty obvious once you look at that particular piece. You have to adapt the makefile pretty often so you also would see gibberish. If you’re a programmer and you encounter what YOU think is gibberish, all alarms go off.
i dont know your experience in coding but I dont see how a huge number (a given with old and popular code) of experienced people could overlook something like this.
But this is the crucial thing. It wasn’t in the repository. It was in the tarball. It’s a very careful distinction because, people generally reviewed the repository and made the assumption that what’s there, is all that matters.
The changes to the make process only being present in the tarball was actually quite an ingenius move. Because they knew that the process many distro maintainers use is to pull the tarball and work from that (likely with some automated scripting to make the package for their distro).
This particular path will probably be harder to reproduce in the future. Larger projects I would expect have some verification process in place to ensure they match (and the backup of people independently doing the same).
But it’s not to say there won’t in the future be some other method of attack the happens out of sight of the main repository and is missed by the existing processes.
Absolutely understand the point. They had a good idea. They failed. Done. my point stands. Foss is superior.
Really?
Yes. Afaik.