>>I answered the email very briefly and said I will be happy to answer with details as soon as we have a support contract signed.
This made my day. If a wealthy individual takes your tools and then calls for help while fixing-up their shed with said tools, do not move a muscle until you agree on the fee.
I expect they'll gladly sign a support contract with Daniel.
As a commercial SaaS vendor we received these same emails from all of the major banks / insurance companies. It's interesting to see that we got some on the Monday after the issue was discovered and some a few weeks later, with some showing a clear understanding of the risk in the context of our product and some looking like a standard copy/paste. Gives you a rare behind-the-scenes view of the information security practices of these companies.
I think it is pretty easy to see how this sort of thing happens:
1. Someone decides that we need inventory of all the libraries used (iirc requirement for some certifications and generally not a bad practice)
2. A system (/excel sheet) is enrolled where you have fields like $our_product, $library_used, $vendor_email
3. A dev, not quite understanding the point, dutifully fills in the data for the project they are working on
4. No-one reviews the data
5. Crisis strikes, so mass-send email to all vendors how they are handling it
Problem here is around point 4.; for the process to work, someone should have reviewed the data to check that the used libraries are from vendors with some sort of support arrangement.
I think the reply they provided is pretty promising, it makes it sound like they wanted to be a customer but are not only due an oversight.
For everyone boggling at the tone of the email, stop for a moment and have a guess at how many different sources of software they think the average large corp has on their books let alone on their infra. It can literally be hundreds or thousands of different sources. And each of those will have their own topology.
This is clearly a scatter-gun survey because they're realised they really have no idea of their exposure. (And before you re-boggle at that, there's a whole business ecosystem in just being able to answer that question let alone do anything about security issues.)
Also, for any FOSS author who gets one or more of these inquiries don't laugh it off or write blog posts mocking the sender. Take it as the business opportunity it is and send a professional response indicating your willingness to help them navigate through this, at least as it relates to your bit of code, for customers with paid support plans. You want money, they have money and you can trivially provide them something at least some of them are willing to pay up for with a potential opportunity for a non-trivial longer term relationship.
This is the best kind of sales call: they are coming to you.
Generally this is an accurate take. I'd add two things:
> ...because they're realised they really have no idea of their exposure.
This is partially because it is often non-engineers being asked to figure this out. The "information security analysts" at F500s are asked to do a lot of unfair work, such as analyze risks related to decades-old software they didn't build.
> ...there's a whole business ecosystem in just being able to answer that question let alone do anything about security issues.
The first part (answering "what dependencies does my software have") isn't inherently bad. I'd emphasize the underinvestment in the second part more.
>> The "information security analysts" at F500s are asked to do a lot of unfair work, such as analyze risks related to decades-old software they didn't build.
I think that's putting it mildly. When it comes to responding, they'll look around and find that they only have a small number of full-time employees with the skills to partake in a response. Most of the IT organization will be dependent on vendors who struggle during the best times while their leadership has the ear of the CIO because IT is only viewed as cost.
The full-time employees will frequently be the real heroes, but when the incident passes this won't be recognized. Things will repeat themselves with the next major vulnerability discovered, but the organization may find that they have even fewer employees at that point to lead a response.
"...The level of ignorance and incompetence shown in this single email is mind-boggling...no code I’ve ever been involved with or have my copyright use log4j and any rookie or better engineer could easily verify that..."
Yeah, well, I've been quite shocked how rookie some F500 devs can be and how dysfunctional large corporations can also be. Probably what happened here is someone wrote a script that compiled the dependencies of all projects they have and they sent this same email to all of them (!) regardless of any actual or potential use of log4j.
When I worked at a large, but not F500, company I had to once every 6-12 month or so fill in a spreadsheet with all third-party dependencies, with their licenses and some other info, the project I was working on used. I then emailed this to a mystery person and never heard anything back ever. I can easily see someone pulling out these spreadsheets and just emailing away without any developer, rookie or otherwise, being aware of what was happening.
Yeah, but that's still a dumb thing to do. They're basically delegating their IT infrastructure's security status to some low-level help in the legal department. What could possibly go wrong?
Your story is all too common. Have you ever seen that old TV show Lost? I think these kinds of stories are the reason why pointlessly pushing the button in that show was such a popular and memorable trope. Things that people "have to do" but no one knows why, and they just keep doing it over and over...because what if? I feel your pain
We used to joke about doing this at my last company. We knew for a fact that our accounts payable folks frequently paid invoices without doing any verification that they were valid.
I'm willing to guess this happens a lot more than people realize. I doubt we were the only people joking about it. People joke, other people hear, some of those follow up with action. The smart ones keep quiet and stop well before getting to $122M.
I don't want to defend this company, but my company (a dev tool used by many other companies) receives a handful of these a day. It's almost the exact same email, and they're just mass-sending them. It's not personal, and it's pretty standard.
The tone feels off if you assume a human wrote it. But that's only because it's a form letter their legal department wrote for them to send off. They probably collected "dependencies" from the entire company (and someone wrote "curl"), and sent a mass email.
If you just reply with a simple "We're unaffected!" (or ignore them), you'll never hear from them again.
If there's no expired support contract, that would be making a false statement of fact in order to get someone to sign a contract and pay me money. It's plausible that that would be fraud.
Of course it's also plausible that that's not fraud at all. But I have no way to know for sure unless I ask a lawyer, which needless to say I wouldn't do. And if it turns out that it is fraud, well, the legal department of Fortune 500 companies tends to be pretty humorless.
Opensource license is a form of contract. I provide free 5 minute support to new users. And good luck suing me if I am not even US/EU based.
Departments (small managers) are authorized to spend small money without approval, lets say up to 200 euro/month. If they send this type of emails, someone ass is on fire. They will DO spend it just to get legal green light.
Anyway, I do not see reason to hold back, just because I am open source developer.
Why would you lie about a contract being expired when you could just say, "this software is provided without warranty (see license) - I offer support services starting at $X/day" and likely see the same result?
A already have enough work on $X/day. If they need to be compliant and treat me like their corporate drone, I am happy to comply. I can charge X*5 and spend one week working on my opensource project.
This is basic marketing.
Airbnb, Facebook, Amazon etc are allowed to do shady stuff, but single contractor should be clean as lilium?
Many organizations document their 3rd party vendors and libraries and it doesn't surprise me that an automated email reached Daniel. Most likely someone mis-documented using one of Daniel's projects in a spreadsheet.
I am personally a bit surprised about the responses here. It is completely reasonable for this email to reach Daniel and is most likely an artifact of bad documentation by engineers in the company. At the scale this company is running the person/team sending out these emails do not have time to dig in and understand each dependency they are sending emails on.
The response is as simple as "What library/product does this email pertain to?", "Please see the licenses for the libraries or products in question.", and what Daniel responded with as well: "I would be willing the dig in further for specific questions with a support contract.".
> At the scale this company is running the person/team sending out these emails do not have time to dig in and understand each dependency they are sending emails on.
That alone is extremely disrespectful, it means they couldn't care less about the time of open source software maintainers. To say nothing of their "request" for review.
It's not about open source maintainers. This isn't an "open source" problem further than the fact that Daniel's software is used in a product they are using. Daniel could take a couple of seconds to ignore this email and there was very little time wasted.
The real "disrespect" should be whatever engineer put Daniel's name into the spreadsheet that blasted out these emails. Someone didn't do their job and is checking a box. How is the (possibly non-technical) person that is required for managing 100s of vendors and thousands of open source libraries supposed to verify all of that information?
I'm personally happy to hear that this company is trying to do SOMETHING to make sure that Log4j is patched even if it's a bit incompetent in it's implementation. There is not malice here.
> The real "disrespect" should be whatever engineer put Daniel's name into the spreadsheet that blasted out these emails.
Agree 100%. Any engineer that got far enough to put this email address into a spreadsheet knew damn well it was inappropriate to do so. They should have put their own email address, as they made the choice to use downloaded software in their project and become responsible for that decision.
Not in this context. This wasn't spam, this was a person with the information they had trying to close the loop on some information an engineer put into a a spreadsheet. This message was not "irrelevant" in the context that the person sending it had. This was not a robocall trying to extort money.
As I noted in another thread: A simple response pointing to the license should be the default response to requests like this which can be automated.
Yep, it can/will happen. My assumption is that this is related to Curl, which has a pretty well documented license. Responding to emails like this with an automated email pointing to the license at https://github.com/curl/curl/blob/master/COPYING seems like an obvious thing to have setup.
Namely: THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. IN
NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
OR OTHER DEALINGS IN THE SOFTWARE.
As far as I learned, a couple of big companies are sending this kind of mail to every provider, partner or copyright owner of code that they could find.
I assume some developer/supplier used curl and provided a list of third party code and licenses they use.
In the aftermath of the log4j incident, companies now target everyone about this issue partly to learn about potential exposure that they are not aware yet, eg exploited infrastructure of depending services like newsletter or analytics services.
Yes, it's annoying and pointless to spam this mails to open source projects. But at least someone is now behind auditing the supply chain.
It's a really dumb approach to vulnerability management for CYA. Spray and pray that the regulators are assuaged. It might even work as far as that goes.
But obviously, it's not a sound approach to actual vulnerability management.
The first email looks like someone who had zero idea of what they were doing, just did some dependency scanning and got your name/email there, probably these emails were sent to everything that they could find.
Quite well handled, not arrogant, not bending over and doing whatever they say, but being honest.
If curl is impacted or not, may not really matters for them, usually these companies go after compliance and someone who they can blame when things go wrong.
That would be fraud. No, start grep on the source code and a few things like that, then provide the results: "a detailed audit found no reference to log4js, so another audit was started which found no reference to any java code in the C source; it was repeated 5 times to confirm these promising results. Another audit followed the Boltzman brain hypothesis to check if the affected log4js binary code could not be spontaneously generated during compilation, by following a Monte Carlo simulation to check for various length of binary data that would match the log4j binary code. (...)
Finally, to avoid this extremely remote risk, the code changed to switch to reproducible builts, which can guarantee this will not happen"
There's no need to have actual interns read it, that would be unnecessarily cruel. Service fees don't need to be based on actual billable hours. You can charge 400% of the time it would take interns to read it without actually doing that, as long as your grep one-liner delivers the same value.
There was a HN post about selling to the Enterprise market. Doing it the way that was described there would be. Also, to not perform a scam as other posts here would be.
1. Insist that you need to talk to upper management until you get to the CEO.
2. Once there you need to sell them on a Fixed fee contract for five engineers so let’s say $1MM or more
3. Actually create a few scripts that run the log4j scanner from Google.
4. Have an extended support contract by doing this yearly at $1MM.
well said, and these companies have way too many lawyers with free time to keep suing you, even if you are right and the judge solves the case in your favour, not always it is required to cover legal expenses and the amount of legal fees burned on it won't worth.
Fixed fee or monthly "support contract", with minimum of 1year.
>Just because F500 companies are big, stupid, slow and greedy, doesn't exactly make stealing right.
That is precisely why it's right. These capitalists have stolen our labour, and corrupted our politics for centuries. `Stealing` it BACK is the ONLY way history has shown us works.
You'd probably need all of the 10 days to fight through all of their supplier management forms, answer pointless questions about security certifications, people involved and if you do business with iran.
In a Fortune 500 company, I'd imagine it could be quite difficult to definitively prove that they are not a customer of any one organization.
The company I work for is not Fortune 500, but we have several Fortune 500 customers. The amount of inane bullshit we have to deal with as a result is mind-boggling.
I recall an incident of large company paying whatever bill they receive and only to find out that they never had a contract with some of the companies and never receiving any service.
If it makes you feel better, it goes both ways; I once signed up for a utility, got service, and discovered months later that they somehow completed the paperwork to give me service but not to actually bill me.
I would bet this was sent out to a list that was put together that contained all of their "partners" which was in turn compiled from various other spreadsheets. Including one that had a 'support contact' column, or something like that and they assumed any that had that value was a partner. Over the course of the 6 years that the sheet has been cut and paste in various formats, they completely lost where any of it came from in the first place.
OK, a large corporation legal team doesn't understand the nuance of ownership of open-source software.
Do we mock every single open source guy who displays the same amount of cluelessness about the inner workings of a business because I see plenty of that displayed here and everywhere else.
Nuance is, you don't even know what the end goal of the communication is.
It's as dumb as mocking a scam email/phone call telling "You are so wrong about me". The end goal of the scammer is to make money for the total time he put.
Sure, the scammer can go in great detail about your life and tailor the scamming for you, but that's not his best ROI. His best ROI is a generic message sent to everyone.
Yes, the idea that this company paid a lawyer to go through its tech stack and figure out who owns the code relied upon by the company is fairly ridiculous. Either a lawyer drafted a template email for the company to send to its suppliers, or a lawyer (read: intern) was given a list of suppliers to email on behalf of the company, or a lawyer wasn't involved at all.
But in software communities (and particularly in FOSS communities) tech people can do no wrong; every time a tech company does an aggressive or foolish or otherwise objectionable thing, there must be a dastardly lawyer somewhere pulling the strings.
Exactly, everything is a Search problem. Most organizations (recruiters, sales) have their own efficient Search algorithms. Unless you know the intricacies of that, you deserve to be mocked for displaying ignorance.
It's as dumb as mocking scammers for their methods. They are effective in their own way. Just because it didn't apply for you, doesn't mean they aren't making money out of this -- which is their ultimate goal. Their goal is not to satisfy your ego and custom tailor a message to you
Welcome to Corporate Life. Somebody at the top says "Make sure we find out from all vendors what their log4j impact is", and that trickles down until some poor sap in InfoSec is told to do it. And of course "all vendors" includes "open source vendors", aka some dude named Carl in Uzbekistan who wrote a Node.js module. Since InfoSec sap shouldn't even have been tasked with this ridiculous ask, and he's got 10,000 of them to send, he sends a form letter.
Not every open source project is run by the little guy. I want to see a a security vulnerability in something like AES. Then the complaint emails demanding answers in 24 hours would be going to nsa.gov addresses.
Anyone leading a shareholder action would love to see these emails. They are basic admissions that the company doesn't know how or from where it gets essential software.
I find it a bit sad that a tech literate group is bashing a non-literate group fo people. The entire reason your salary is much larger than many other career paths is because of your ability to deal with technology. The premise that when the less educated and informed try to question something they don't understand only to be left with pandering and jabs is disingenuous. The questions although perhaps better phrased by someone with a more tech focused background are fine questions for a business to ask. Stop being douchebags and grow up.
> I find it a bit sad that a tech literate group is bashing a non-literate group fo people.
Creating a software bill of materials is a technical task. Managing software security risk is a technical task. These need to be performed by a technically literate person.
A Fortune-500 company has the resources to pay for such technical competence. They are not a mom-and-pop shop.
No Fortune-500 CEO would get their teeth done by a fly-by-night "dentist", nor would they hire "builders" who can't nail two planks together. They would pay for the expertise. If they don't know how to find the expert they would pay for the expertise of finding the expert first and then they would pay for the expertise.
But this is not what they did. They found someone who is both lacking the necessary technical common sense and is terribly arrogant. That is worthy of ridicule. And I'm not ridiculing the individual employee but the whole company.
> The premise that when the less educated and informed try to question something they don't understand only to be left with pandering and jabs is disingenuous.
That idea flies when a student is lost in the woods. When an economic juggernaut combines technical illiteracy with a lack of tack they can get the sharp ends of our tongues.
> The entire reason your salary is much larger than many other career paths is because of your ability to deal with technology.
Won't be for long if we silently support huge companies to employ muppets. Which is why asking for a support contract is the right answer here.
I find it sad that the security department of a Fortune 500 company is sending out emails demanding OSS maintainers respond within 24 hours or else.
You can feel sorry for the poor sap that was forced to embarrass himself, but it doesn't change the fact that everyone here feels like that company can get bent.
Why should the company get bent? Because some executive caught wind of a critical zero day and decided to have their company mitigate damage the same as any other company.
Do you really think the security department in this specific company would not find this email dumb? In many cases, when things are reacted to hastily and in parallel its easy to take one action and generalize it to the whole company and not realize this is one of many actions the company took. No need to get bent out of shape over this and say this entire fortune 500 company is equally incompetent. If you think that you are not living in reality.
I had the same thought - the e-mail really wasn't that unreasonable, coming from the perspective of somebody who didn't realize there was no support contract in place (and maybe didn't even understand how that could happen). Haxx's response seems similarly reasonable - we don't have a support contract, let's get one in place and then move forward from there. This really seems to be an object lesson that if you're depending on somebody for business-critical infrastructure, make sure they have a reason to support your business.
I agree, the response was reasonable. My frustration is with this hackernews thread and the constant judgement and snarky attitude we give to less tech literate folks. If everyone understood tech, we wouldn't be paid nearly the salaries we are for what we do.
It reads like this was a form letter that was to be sent out to all their actual paid partners, and after the massive game of telephone that is corporate hierarchy, it somehow became all dependencies they had contacts for. And somewhere someone filled out a form, probably years ago, with his email on it as the maintainer of a dependency they use (because leaving it blank isn't allowed). And he got caught up in that mass email, totally dumb, but also could easily see how it can happen.
I think daniel's reaction is appropriate and well thought. One can suppose thousands of these emails asking for free work have been sent. There is close to zero chance his demand for a support contract would get past the first filter. Whereas making a blog post about it makes for a good story and also has more chances to get the attention of the right people at this company. Even if it's slightly aggressive.
The document uses a monospace font, and the redacted name can be seen to be 10 characters long.
Based on the 2019 Fortune 500 list, that gives these possible candidates: Activision, Alaska Air, Albertsons, Altice USA, Amazon.com, Ameriprise, AutoNation, BB&T Corp., Bed Bath &, Blackstone, Booz Allen, BorgWarner, Burlington, CBRE Group, Chesapeake, CMS Energy, CVS Health, Dean Foods, DTE Energy, Enterprise, Eversource, Expeditors, Fannie Mae, First Data, Ford Motor, Home Depot, Huntington, JM Smucker, Jones Lang, Laboratory, Mastercard, McDonald's, Murphy USA, Nationwide, News Corp., NGL Energy, NRG Energy, Occidental, PBF Energy, Prudential, PulteGroup, S&P Global, State Farm, Unum Group, US Bancorp, WEC Energy, Windstream, World Fuel, WR Berkley, Yum Brands
From the article: "The email comes from a fortune-500 multi-billion dollar company that apparently might be using a product that contains my code, or maybe they have customers who do. Who knows?"
The "or maybe they have customers who do" makes me think that this company must provide services to other companies, so probably not a Mcdonald's or Albertson's or something like that.
Wasn't Java's SecurityManager stuff supposed to prevent these kinds of exploits?
I haven't used log4j for ages, so I didn't know offhand. Somewhat curious, I gleened that none of the enterprisey stacks use SecurityManager. I guess I kinda understand; SecurityManager was fashioned and pitched for an ecosystem of applets, agents, and sandboxes.
Further, I then gleened there's a JSR to outright remove SecurityManager. With no apparent replacement, just some vague advice to roll your own capabilities based system.
So, however we got here, what's then plan? Run JVMs on top of something like OpenBSD's pledge?
More accurately "a clueless IT lackey at a Fortune 500 company" sent the mail. I doubt the chairman was pounding the board table and barking "We demand answers from Haxx!"
It's signed off "Information Security". The template email might have been drafted by legal/compliance, but it is surely for the IT guys to figure out what code they use in their tech stack and lead those discussions.
It's a reply to David/Daniels email to the F500 org. The dev didn't post a screenshot of their reply, but they mentioned this - "I answered the email very briefly and said I will be happy to answer with details as soon as we have a support contract signed."
Is there a product out there that makes it easy for open source maintainers to offer enterprise support services?
I think support is probably the best way of making money from open source, but a lot of maintainers are unlikely to have everything set up to do so (business entities, contracts, ways to receive payment, probably a dozen other things that you'd never think of, etc.).
Versus asking for a support contract because I don't really want to support anyone like this long term, I would have sent an invoice... If it gets paid, I answer the questions, if it doesn't everyone knows where everyone stands. I also think it's easier to get an invoice paid versus trying to negotiate a support contract.
> In my tweet and here in my blog post I redact the name of the company. I most probably have the right to tell you who they are, but I still prefer to not. (Especially if I manage to land a profitable business contract with them.)
If he wasn't trying to land a contract, then he would have posted the company name.
A tangential point, and granted it's been around and rightly ridiculed since probably the 90s, but how 'bout that classic signature about CONFIDENTIALity? It's like the icing on the don't-know-how-computers-work cake!
You can learn a lot from this. This is how efficient companies operate. No one who knows the difference between C and Java was involved in the sending of this letter. If they were, that would be a waste of resources.
If you want a company to change their behavior, give them a reason to do so. Daniel has quite a platform being a well known maintainer, but instead of using that platform to shame the company in question, he politely emails back to the person sitting with an outdated excel sheet of 500 suppliers. That person didn't decide that the "email everyone on this list demanding info" strategy was a good idea.
To actually make a difference when you have a platform, use it. Tweet-shame them so that the fallout actually reaches the manager in question. This is just complaining about a behahavior while at the same time more or less doing everything possible to encourage that behavior.
I have a feeling that some security automaton at a major corporation is about to have their mind blown when they discover the world of Open Source Software. They had absolutely no idea that non-commercial software was even a thing.
Why black out the company name? Confidentiality notices at the bottom of emails aren't legally binding, especially when it's an unsolicited email from a company you have no relationship with.
This was addressed directly within the linked blog post:
> In my tweet and here in my blog post I redact the name of the company. I most probably have the right to tell you who they are, but I still prefer to not. (Especially if I manage to land a profitable business contract with them.)
>Why black out the company name? Confidentiality notices at the bottom of emails aren't legally binding, especially when it's an unsolicited email from a company you have no relationship with.
This made my day. If a wealthy individual takes your tools and then calls for help while fixing-up their shed with said tools, do not move a muscle until you agree on the fee.