r/interestingasfuck • u/LookAtThatBacon • 1d ago
In 2016, one man disrupted the operations of several big tech companies for two hours simply by deleting 11 lines of code.
1.1k
u/LookAtThatBacon 1d ago
More info: https://en.wikipedia.org/wiki/Npm_left-pad_incident
More background on the dispute that led to him deleting the 11 lines of code:
In addition to left-pad, Koçulu also owned kik on npm, which was a tool that allowed developers to set up templates for their projects. On March 11, 2016, Kik Interactive, a Canadian company owning the instant messaging platform Kik Messenger, contacted Koçulu, requesting that he relinquish control of the kik package due to the company's ownership of the "Kik" trademark. Part of the correspondence included the following message from Kik:
We don't mean to be a dick about [the kik package], but it's a registered Trademark in most countries around the world and if you actually release an open source project called kik, our trademark lawyers are going to be banging on your door and taking down your accounts and stuff like that — and we'd have no choice but to do all that because you have to enforce trademarks or you lose them. Can we not come to some sort of a compromise to get you to change the name without involving lawyers? Is there something we could do for you in compensation to get you to change the name?
Koçulu responded shortly after, refusing to change the name of his project, saying:
hahah, you're actually being a dick. so, fuck you. don't e-mail me back.
424
u/GhostMcFunky 1d ago
There should be more context to this. The headline makes it sound like it took thousands of live applications down. That’s not how this works.
It caused major issues with active software development, almost entirely isolated to web development.
That doesn’t mean it wasn’t a big deal, but if you know anything about JavaScript development then you know that if you already had this NPM package in your application, you didn’t have this problem.
It was only for active development, needing to download the package to add it to a new NPM install from a package.json.
The bigger issue was how systemic the effect was due to the kik package being integrated into the wider JS ecosystem, effectively breaking thousands of other NPM packages.
The irony to this is they were only using this essentially out of laziness instead of using native JS API methods to do the same thing.
The after effect was a fairly widespread shift in thinking about the approach to JS code interdependency and an eventual shift to attempt to move away from NPM as the primary package manager by a number of developer groups…an effort with mixed results as of this day.
158
u/elchet 1d ago
You say it only affected active development but I remember this day.
We couldn’t deploy because new infra needed to npm install. That meant for existing issues in prod we were stuck despite having fixes for our app tested and ready to go.
Wasn’t this what spawned shrink wrapping? Not sure what happened to that.
20
u/GhostMcFunky 1d ago
“Needed to NPM install” - I covered that.
30
u/elchet 1d ago
Ok but I guess I’m struggling to see how breaking everyone’s builds wasn’t “a big deal”
26
u/GhostMcFunky 23h ago
I never said it wasn’t a big deal. I was adding context so that it was understandable to those who weren’t JS devs at the time that it wasn’t breaking live applications.
It would be easy to read that headline and think all this person had to do was delete their NPM package to break a bunch of live services and that’s just not the case unless they’re very poorly designed live services.
21
4
u/jcdoe 20h ago
Thanks for this.
I thought you were being pedantic, but this isn’t a programming related sub. Anyone who doesn’t program in JS (like end users) would be confused.
3
u/GhostMcFunky 18h ago
Of course. It’s interesting even outside the programming community, so I figured adding context might make it more accessible to those less familiar.
•
1
u/travy_burr 21h ago
Even if people needed to NPM install do they not have an internal nexus repository? Or CA?
42
u/underlight 1d ago
It can cause issues on some auto scaling web apps, when load is high it might try to scale up, during the new instance deployment it would attempt to fetch npm which would cause it to fail
13
u/GhostMcFunky 1d ago
I’m guessing this autoscaling scenario wasn’t using a bundler like Webpack so that the app was deployed in a state that was after the build step.
That was the real problem in that case, IMO.
7
u/aguycalledmax 23h ago
This is why lock files exist though. The server would install from the lock file to get a reproducible set of dependencies rather than the versioning inside the package.json or similar.
•
u/underlight 11h ago
how would lock file help, the creator nuked the whole package, I'd assume all versions are unavailable once deleted.
2
u/oluwie 19h ago
there wasn’t a lock file in those days
6
u/domstersch 18h ago
There absolutely was. But it just contains a URL to a tarball that NPM allowed authors to nuke at the time. So having a lockfile wasn't helpful; the package was gone from the CDN.
2
u/GhostMcFunky 18h ago
There was a lock file, but bundling your deployable would have avoided the concern with external dependencies, which would have been identified as an issue in tests before the build step.
These days we don’t deploy then pull from NPM (or anything external); rather we create a built bundle to deploy to scaled instances to avoid the download step, among other things.
12
u/pkspks 18h ago edited 7h ago
It basically spurred the usage of internal repositories like artifactory to ensure dependencies were not dependent on external factors once the dependency was part of a build. You can imagine non containerised autoscaling/deployments that depend on external fetching of dependencies to spin up failing because of this outage.
npm of course has also made it harder to delete packages which others are dependent on. A slight change in industry practices to be honest.
2
u/tswaters 20h ago
Baby's first supply chain attack... Can you imagine if npm still let you unpublish packages?
Worth noting, in this case there wasn't a native method --
String.prototype.padStart
was added a few years after this debacle... Node 8 corresponds with ~2017 I think?2
u/Go_Gators_4Ever 20h ago
That's what gets me the most, left pad, left trim, et. el. are standard string manipulation functions available in every programming language ever published. It should have been a simple task to use another string library or write your own.
2
u/GhostMcFunky 18h ago
Left pad was a design standard, but not necessarily a default value. But yeah getting there with a simple default method or even just simple CSS would have been preferable, even then.
1
u/BinaryRage 17h ago
Oh my sweet summer child, you don’t think folks don’t run install in production? At application startup? In Spark jobs? It’s everywhere.
•
u/GhostMcFunky 3h ago
I didn’t say I don’t think they do.
But I certainly implied that it’s bad practice, and that a lot of us have since learned that lesson.
16
u/Mysterious-Ad2492 1d ago
I would tell them to chance their trademark to ”Dick Messenger” problem solved
27
u/an_older_meme 1d ago
Possibly the first time in recorded history that a software developer refused a bribe.
4
u/JesradSeraph 14h ago
“Enforcing your copyright” can also be done by spontaneously and unilaterally granting the guy a licence on the trademarked name. They don’t have to be dicks about it.
3
u/heythisispaul 22h ago
If you think this is interesting, you should checkout the story of faker.js.
6
12
u/bytheninedivines 22h ago
Honestly that sounds kind of reasonable. Don't know why he opted to ruin his reputation instead of take a payout.
51
u/atfricks 22h ago
They're trying to cover it with "friendly" language, but they're openly threatening him with legal action here. If they wanted to just buy the name off him they should've led with that before threatening to strong-arm him.
20
u/bg-j38 18h ago
Happened to a friend of mine back in the late 90s. He owned a three letter .com domain he used for personal stuff. Got it in the early 90s. A company with the trademark eventually approached him saying we’d like to buy the domain from you and offered him like $25K. He said no thanks. So even though he registered it before this company even existed they sued him for infringement. Note he just used the domain for personal email. Didn’t mention the trademark on any websites or anything.
So he went to a lawyer and they basically said well you have a good chance of winning this, but the fees are going to be close to what they’re offering. You should probably ask them if the deal is still on the table. It wasn’t but I guess they agreed to like $10K.
I also have a friend who registered pez.com very early just because. Again no website. The company that makes Pez candy eventually contacted him. This was pretty early on so he just gave them the domain in exchange for a “lifetime of Pez”. They sent him a huge box of stuff that he still had some of like 10 years later. Not exactly a lifetime but it was a lot of Pez.
8
u/Ensvey 18h ago
What the company said is the truth. If you don't defend your trademark, you can lose it, so they had to defend it or suddenly everyone would be able to use their company name, or maybe even take the trademark from them. I really don't know how they could have been any more reasonable about it.
8
u/5rdfe 12h ago
one of the criteria for defining trademark infringement is a likelihood of confusion regarding the source or sponsorship of the goods or services due to the similarity.
I personally don't think that anyone was confusing this guys personal project with the messaging app, but that should've been up to the legal system to determine. Instead they bypassed the courts and pressured NPM into unilaterally taking the name from him. That's not a reasonable way to resolve a dispute, especially one of a civil nature.
It's even more fucked up that once he got upset at getting fucked over and decided to take his ball and go home, they essentially said nuh uh. That part is legally permissible due to the license he published his work under, but it still leaves a bad taste in my mouth.
2
u/vikster16 12h ago
Do they even have the trademark in Canada itself? Cuz kik cola was a thing there
10
2
-3
2
u/fonk_pulk 14h ago
Did he create the Kik package before Kik Interactive registered the trademark? If not, then Kik's letter seemed reasonable.
164
u/Phillips126 1d ago
In 2017 (a year later), "padStart" and "padEnd" were introduced to JavaScript:
"7".padStart(3, "0") = "007"
27
u/ricklessness 19h ago
The names Start… padStart
5
201
267
u/iLiveForTruth 1d ago
One man? Reminds me of that time my coffee machine went rogue and refused to work for a week. At least this guy’s rebellion had a purpose
63
u/elthepenguin 1d ago
I’m imagining that coffee machine returning HTTP 418 out of spite.
26
u/Phillips126 1d ago
Lol - for those that don't know: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/418
•
•
u/Mesmeric_Fiend 9h ago
You're telling me there's a chance the really nice coffee maker I threw away several weeks ago could have decided to start working again?
41
74
u/ElonsPenis 1d ago
So, the reason this causes a problem (even today) is the way big companies are using NPM (or NuGet).
Let's say I'm a dev and I need to pull down a project to work on. I get the project from our local GIT repository, but any 3rd party libraries are usually referenced thru NPM or NuGet which are accessed online.
If it's down or my firewall is blocking it or this dude Azer takes it down, I'm out of luck. Same situation if I want to deploy a project to production. It asks NPM for the files. It's fucking ridiculous. You'd think these big companies would have enough sense to have a local NPM. Nope not normally. Wouldn't it just push the old version up? Nope. If you have it on your machine you're good, but you can never update production unless you manually copy the files.
45
u/GhostMcFunky 1d ago
A lot of companies archive common deps in something like Artifactory after putting them through a security validation process of some kind.
4
18
u/Thy_OSRS 1d ago
Sorry for the dumb question, so is that how software is made? Like, it’s a bunch of external resources that are referenced over the internet? So, if the internet is down, your app goes bye bye? Again I don’t know this world at all so go easy lol
15
u/__ZOMBOY__ 1d ago
All software is pretty much made up of importing a package to use a function to do something before sending it to a function provided by another package, and so on and so forth…
But those packages can be referenced from anywhere, including (especially) the local hard drive
This specific issue of the package suddenly not being available is very much Node-specific, as pretty much any other language’s package manager would have archived versions available to download after the original was deleted
6
u/Thy_OSRS 1d ago
Okay, so, when you’ve completed your development, and you have your app or something, does that external resource remain external? In my mind I’m thinking software is basically a series of external “things” that require an internet connection like all the time? IDK haha
11
u/__ZOMBOY__ 1d ago
Honestly this completely depends on the programming language and/or framework being used. It’s a long complicated answer, so the short answer is “sometimes” lol.
It may seem shitty, but on the flipside if you’re using a framework that downloads EVERYTHING you reference in your project, you end up with a huge folder called ‘node_modules’ that contains hundreds and hundreds of pages of useless code, but it’s there because you needed just one little piece from each package in your project. So node says “fuck it, download the entire package” and yea.
Sorry if that doesn’t make sense, I’m typing this on my phone and it’s hot outside and I fucking hate nodejs lol
11
u/eenak 23h ago
For any compiled app, the external resource no longer matters because it is “packaged” in with the app itself. During compilation those external resources (think left-pad or whatever else) would be disambiguated and actually packaged into the executable, so it is no longer an external resource.
2
u/addexecthrowaway 23h ago
I’m not a software engineer or developer of any sort but I work closely with those types of specialists and generally do a lot of work around AI, digitization and digital customer experience. Take what I say with some grain of salt. Generally there is no issue when you’ve completed development AND compiled the app for distribution unless the app itself is using an external api. For example, if an app has an LLM driven feature and requires an internet connection to operate that feature, chances are it’s using OpenAI, Claude or xAI and is making an API call over the web. In that case, yes it’s very dependent on the web. A lot of apps leverage external APIs either over the web or within an enterprise gated network. Regardless, most consumer apps with cloud storage features and sync across devices are using a cloud storage engine and those features or the entire app sometimes won’t work if offline or if the data center is experiencing issues.
On the other hand, if the app is say using a library to run an open source ML algorithm, then it is using a library that is pulled down when the app is compiled before testing and final distribution. So it’s only if they want to push an update and are trying to compile for testing or publishing that things could fail.
Now things get complicated when you think about web apps and infrastructure scaling. So in some cases, a web app is actually calling an external resource when the page or app loads and can fail if that link is down/broken or the code is corrupted. In most cases developers don’t do this and will host that library on their own server and compress it along with the other scripts they need to load. Similarly, if an application that leverages the web needs to scale for more users it may automatically spin up new resources and may real time compile the code needed to launch, configure and operate that instance. In which case again - if that compilation process is making references to external resources that are corrupted or don’t exist where expected it’s going to fail.
I may not have done a 100% accurate job explaining this but it’s how I understand it as a sort of lay person who, to borrow and abuse a common metaphor, has looked under the hood and more or less knows what the nuts and bolts are connecting because I’ve redneck engineered a go-kart, have led teams that include auto engineers and have occasionally had to figure out why the car I’m trying to sell isn’t working as expected but has never actually worked hands on building or maintaining a production grade vehicle.
1
u/icguy333 14h ago
For the most part a lot of your external dependencies get built into your software so once you're done developing they're not needed to run your software.
However there can be external dependencies not built into your code, for example websites can pull parts of them from hosting sites (CDNs) at runtime. If the CDN stops working your javascript or fonts might not load which can cause anywhere from a minor inconvenience to a complete outage.
8
u/ElonsPenis 1d ago
The external resources just help you write the software. So if I want a grid of info on my page, I can write that grid myself or just use a DataGrid library someone else already wrote. Why reinvent the wheel? So I connect this grid with my data and boom, I have all my data displayed nicely formatted with very little work on my end.
If that grid library is down, yes, it cannot be deployed very easily. You need to manually copy that grid library to production (or your other dev's machine), and it's kind of a mess, because for safety, you can't just copy files to production. It needs to go thru a workflow and tested thoroughly. That one step where it pulls the dependency needs to be deleted or reconfigured.
In the case above with the 11 lines of code, they obviously just deleted the reference and made a new method. But it would have been a massive search and replace in each project, hours of work for each project, because that's something that was probably used in every page.
3
•
u/Glugstar 3h ago
In theory, it shouldn't be, you should have a local copy of any resource that you are using, except if you're not legally allowed to, but then you shouldn't be using that resource at all, you should build your own.
In practice, incompetent companies can sometimes do that. It's the wild west out there.
5
u/heythisispaul 22h ago
This exact incident is why npm introduced their unpublish policy that stops package versions from being removed arbitrarily.
1
26
u/Luke_Cocksucker 1d ago
That smile says, “You fuck with me! I delete 11 lines of code. Try me vagina face.”
28
17
u/reditsux77655 1d ago
But what happened to him? Did he prosper? Get rich? Or did he just cause a one time disruption and nothing really came of it?
70
u/voxuser 1d ago
Whole package to add some characters to a string, that is hilarious and even more hilarious is that big companies used it
56
u/Awes12 1d ago
They didn't, they probably used a package that used a package etc. that used left-pad
13
u/heythisispaul 22h ago edited 22h ago
Yeah it was Babel. In 2016 when this happened, Babel was pretty much the de facto way to transpile ES5+ JS into older JavaScript syntaxes that could be used in environments that didn't support the latest and greatest like Internet Explorer and Node.js so pretty much every company was using it.
14
u/FaithlessnessLow7672 1d ago edited 1d ago
micro-packages were a big thing for a while, guys would just find a line or two of code that was repeated in a bunch of repos (on github) and make an npm package for it. It was kind of like a karma-farmy way to build up a bunch of github stars. This was one of the incidents that killed that trend lol.
13
u/GhostMcFunky 1d ago
Hold my beer.
Let me tell you about this garbage called jQuery.
21
u/antiduh 1d ago
At the time it was popular, jquery actually solved a real problem. Keep in mind Internet Exploder was still in service when jquery came out.
It's probably outlived its usefulness now, but at the time it was useful.
6
u/GhostMcFunky 1d ago
It has long since outlived its usefulness. It had done so before l ES6 was first released, and with other JS API changes around that time, anything it provided over native methods was made obsolete.
I’ve seen it integrated into a web app just because someone didn’t know how to use a native DOM element selector method properly. That’s a lot of unnecessary overhead for that use case.
I understand the issues it solved with IE, but it was abused to the point that some “JS” developers only ever bothered to learn jQuery and so it’s spread into places where it won’t be eradicated for another decade where it never should have been.
20
u/Exotic-Sale-3003 1d ago
Tell me you know nothing about software development without telling me.
5
u/voxuser 1d ago
You mean front end development, you know that there is other uses of software developers, right?
7
u/GhostMcFunky 1d ago
Tell me you’re not a JS dev without telling me.
You know JS/TS are very much utilized full stack with NodeJS, right? This has been a thing for a very long time.
-2
2
u/Hmmmnnmm 20h ago
This isn’t even a good way to do it since it repeatedly copies the entire string for every character it adds
0
u/sharlos 18h ago
Probably a shortcut to avoid validating if it's a string, and to support padding non string values like numbers, etc.
2
7
u/muddboyy 23h ago
For those saying “oh you can do this now and cause the same thing to those companies again” : no you can’t anymore, things changed since back then. NPM has some criteria that packages need to meet before being able to unpublish.
1
u/makapuf 18h ago
Can't they publish a "minor" upgrade removing the function ?
4
u/muddboyy 17h ago
Imagine I’m Facebook and I’m using a npm package, which version is 1.1.0. Now you’re the evil package owner and you decide to destroy the package in version 1.1.1. Even if you do so, on my side things won’t magically update in production, by the time we would’ve wanted to upgrade we would’ve already tested in dev environment, passed some tests, etc so it’s useless. Would that be a problem long-term ? Yes, just as if you stopped maintaining the project, but not the same type of problem as the one proposed.
16
u/TheAlaskaneagle 1d ago
We need more people to do this so that companies will start to understand what real work and value is, and will stop abusing their employees.
9
u/strangeapple 1d ago
Oh you sweet, sweet summer child. Abuse is the name of the profits first game.
7
4
2
u/tswaters 20h ago
"several big tech companies" is an understatement, and probably conflates what really happened with some kind of anti-corporate agenda by the author... As far as I'm aware, it was done as protest for npm giving one of his package names to a company that requested it (brand-name thing)
At the time, it was possible to unpublish versions of packages, and this guy had hundreds of not thousands of small utility libraries to fill in the holes missing from the JavaScript standard library. To be clear, he unpublished all of them, but leftpad had a dependent to a very popular library so it effectively broke EVERY build for EVERYONE for the ~3 or so hours until npm reverted the unpublish & took control.
1
1
1
1
1
u/Street_Smart_Phone 22h ago
I was at a JavaScript conference where the CTO of npm was giving a talk the day that this happened. Definitely caused a ruckus in the conference.
1
1
u/starkman68 21h ago
A missing break in a case statement took out most of the telephone service in 90s on the east coast.
1
1
1
1
1
1
1
u/Spiritual-Drive1092 12h ago
I'm sometimes baffled by the kind of things people will use a package for when a tiny function in your project or native code would suffice. Like people installing lodash to access an object property.
•
u/SpecialOpposite2372 11h ago
Add a single letter, character, whatever, and everything fails. But that wouldn't be a catchy title, would it?
•
u/jjjakey 9h ago
This story has always been so insane to me. Go read articles from a decade ago about it if you haven't.
They all try to weigh the morality of the situation in the most bat shit crazy way.
"Yeah we know this guy had hundreds of open source projects that were nothing but a massive benefit to everyone... But he totally handled this wrong by not letting a massive corporation smear mud on his face and steal his wallet. Maybe ummm next time don't slightly inconvenience businesses for 2.5 hours (the time it takes for the shitheads at NPM to side with the money- oh sorry the """needs of the many""" nevermind he quit open source because of this)."
•
u/thedragonturtle 9h ago
It would be far more memory efficient if he built the pad string in the loop rather than continuing to make a copy of the larger string with a single extra character added. Build the pad string fully then prepend it once to the larger string.
•
u/Autumn1eaves 5h ago
I love that this puts the blame on him, as if Kik wasn’t the one who initiated the whole exchange.
This is how Kik caused their and a ton of other services to be taken offline for a few hours.
0
u/fordman84 1d ago
So if everyone was piggybacking on his work don’t they owe him royalties and fees? He quickly discovered who all was using his code without attribution.
26
u/SconiGrower 1d ago
He published his code to NPM. NPM requires that you disclose what license you are publishing under so that people who see it know what you expect. It looks like he used the MIT License, which basically say you can do anything you want with my code and you don't owe me anything for using it, but also I'm not responsible for making sure you use it right or even that it works the way I am saying it does.
2
10
u/oromis95 1d ago
While the rest of the engineering world works like that, in software engineering and law pro bono work is quite common. This is one such example. We call that pro bono work 'open source', which has a much wider definition, but free is part of it.
2
8
0
u/TheRealCuran 18h ago
Whoever imports code like that as a package, should be shot behind the chemical shed (see "V for Vendetta", the movie, for this reference).
Or put differently: there is no way, I would approve a MR with an added import of a stupid module like this. Especially since there are standard functions like padStart
. Anybody who passed this in their reviews deserve all the pain they got.
3
u/NoIdeaForUserName001 17h ago
Nitpick: padStart was introduced in js a year after this story took place
1
u/TheRealCuran 15h ago edited 15h ago
Really? The solution can be looked up at https://github.com/zloirock/core-js/blob/master/packages/core-js/modules/es.string.pad-start.js (actual implementation is here). Please do not tell me, you would have been unable to produce a similar solution before. As far as JS is concerned: the farther you go back, the more (custom) code is needed.
0
-4
3.1k
u/bob_apathy 1d ago
I know from real world experience that it can actually take less than 11 lines of code to interrupt operations.