Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Big Kahuna: Underscore + Lodash Merge Thread #2182

Closed
jashkenas opened this issue May 21, 2015 · 204 comments
Closed

The Big Kahuna: Underscore + Lodash Merge Thread #2182

jashkenas opened this issue May 21, 2015 · 204 comments

Comments

@jashkenas
Copy link
Owner

Dear Everyone,

JDD got in touch with me over email this morning, inquiring about the possibility of merging Underscore and Lodash together. I think it's high time to talk about the possibility again.

Some initial points for discussion:

  • The Name: "Lodash" is very cute, but keeping "Underscore" makes the most sense, both for historical and _ variable reasons. Because npm doesn't allow name redirects (AFAIK), the merged project would be available as a module under both names for the foreseeable future.
  • Lodash is currently > 12k lines of code, and Underscore is 1.5k. I understand that part of that is JSDoc noise, but it still seems potentially problematic and worth talking about.
  • Documentation style. Ideally I'd like the Underscore docs to be written even more conversationally than they are at the moment.
  • We should keep nicely annotated source code as a priority.
  • And then the actual main work — I'm not familiar with the Lodash codebase. What parts of it do folks have objections to merging? What might not fit in well within a merged Underscore umbrella? Anything? Maybe not?

Edits in response to comments below:

  • Strict(er) SemVer. Totally fine with me. I'm still of the opinion that true believer SemVer isn't something that actually exists in the real world (most changes are subtly backwards-incompatible in minor ways), but I have no problem praying to this golden cow to keep the philistines happy ;)
  • "Governance Model": I don't think we need anything fancy here. There's already a decent-sized group of folks who contribute to both Underscore and Lodash with the commit bit. They can keep it. I do think we'd need a clear mission statement about the spirit of what the library should be. Utility libraries have a tendency to grow to include the kitchen sink, and the more we can successfully keep things focused, the better.
  • _(obj) wrapper chains lazily by default. (Breaking change). Sounds fine to me.
  • How does Underscore-Contrib fit in to all this?
  • Folks. Please STOP +1-ing, 👍 -ing and meme-ing GitHub tickets. We get it, but it makes tickets hard to have conversations on. Contribute with something substantive instead.

Although it would be nice to keep the bulk of the discussion here ... there's also a Contributors-only cross-thread here: #2184

@jdalton
Copy link
Contributor

jdalton commented May 21, 2015

Before digging into the discussion I just wanted to say 👍 and thank you to Jeremy for hosting and being open to discussions.

Sidenote: For those interested there's also a fair amount of chatter on twitter on the merge.

@tyscorp
Copy link

tyscorp commented May 21, 2015

Does that mean lodash will no longer follow semver?

@jamesplease
Copy link

For future commenters: many of you will 👍 to this. If that's all you've got to say, it might be best to not add noise to the conversation. There's quite a bit to be discussed before this could ever happen, and it seems like this issue is attempting to be the forum for that discussion. If you want to stay involved, click the "Subscribe" button to the right to see notifications.

@phpnode
Copy link

phpnode commented May 21, 2015

Does that mean lodash will no longer follow semver?

@tyscorp I'm going to go ahead and say no, that won't happen

@lennym
Copy link
Contributor

lennym commented May 21, 2015

Firstly 👍

Now in actual response to a few of the points raised (disclaimer - I am a lot more familiar with underscore than lodash):

Call it underscore or something completely new. Lodash is obviously a take on underscore, and doesn't make a lot of sense on its own.

Things I would like to see from lodash in underscore:

Code split by category/method like var chunk = require('lodash/array/chunk'); in the lodash readme. I think this could go some way to mitigating the fact that lodash is much larger code-wise.

_.get - I find this absurdly useful, and it's basically the reason I started installing lodash instead of underscore.

Probably more to come as I think of things...

@Ky6uk
Copy link

Ky6uk commented May 21, 2015

Lodash is currently > 12k lines of code, and Underscore is 1.5k.

How about comparing only existing functions in both of __? Ofc we can make a custom build of lodash if 12k is our problem.

@JasonGhent
Copy link

"Lodash is currently > 12k lines of code, and Underscore is 1.5k. I understand that part of that is JSDoc noise, but it still seems potentially problematic and worth talking about."

Can we categorize what else contributes to the size differential?

@tdd
Copy link

tdd commented May 21, 2015

Hey all,

I obviously 👍 this to death. Thanks @jashkenas for re-opening this, let's hope it's not a can of worms anymore.

A couple notes off the top of my head.

One thing from Lo-Dash I'd love to see happen in Underscore is the code split (that you find as multiple modules in the amp.js ecosystem) and general build/tooling that lets us not bundle the whole code when browerify'ing / webpacking / brunching / etc. a small lib of ours that only uses a couple _.xxx methods.

I wasn't aware of LoDash's _.get, which is admittedly kinda sweet.

Underscore would also need to be way more open about the iteratee thing, and use both situations (callback vs. prop name) in its code/usage examples, to drive the point home. It's great that it took this page from LoDash's book, but perusing the docs, it's almost completely hidden. It only shows up once and then is never illustrated again. This makes for way shorter code, though, and ties neatly with _.result, too.

Let's move this forward!

@zacechola
Copy link

The Name: "Lodash" is cute, but keeping "Underscore" makes the most sense, both for historical and _ variable reasons.

I hear iojs may be back on the market. 😜 But seriously, either name is fine by me.

Lodash is currently > 12k lines of code, and Underscore is 1.5k. I understand that part of that is JSDoc noise, but it still seems potentially problematic and worth talking about.

As has been pointed out already in this thread by @Ky6uk, there needs to be an apples to apples function comparison here.

Documentation style. Ideally I'd like the Underscore docs to be written even more conversationally than they are at the moment.

👍

We should keep nicely annotated source code as a priority.

👍

And then the actual main work — I'm not familiar with the Lodash codebase. What parts of it do folks have objections to merging? What might not fit in well within a merged Underscore umbrella? Anything? Maybe not?

Versioning needs to be clarified for sure and I also would personally appreciate the modularization that @tdd brought up as a nice to have.

@wraithgar
Copy link

The only pain point we've noticed using lodash is that in some packages that have several modules, each with quite a few lodash dependencies under them and in their dependencies, npm dedupe has a lot of trouble and often silently exits while trying to run. This is of course not currently a problem with things using underscore because it's one single module, and is something worth at least mentioning if you're talking about a transition to the lodash individual module approach.

@aredridel
Copy link

npm 3 will help the dedupe process immensely.

@glesperance
Copy link

Would be great if the merge could keep using semantic versioning as in lodash

@kud
Copy link

kud commented May 21, 2015

To be honest, I was already afraid with nodejs / iojs dilemma but glad they'll merge too. So I couldn't be agree more to see your projects merge too.

Congratulations anyway to you, you make a really good work.

(warning: I mostly use lodash for the moment on all my projects)

+1 for "underscore" because history + this is what I say when I read it.

+1 for the underscore's doc, clearer/cleaner in my opinion.

@xcoderzach
Copy link

I'm a huge fan of how lodash does lazy evaluation chains by default, rather than explicitly needing to call .chain(). Ideally that would be the standard going forward.

@mathiasbynens
Copy link

👍 Let’s make lodash ∶ Underscore ∷ io.js ∶ Node.js and merge already!

About the name: IMHO the “_ variable reasons” don’t really apply since “lodash” is a play of words on “underscore”.

One thing in favor of sticking to “lodash” is that there are already hundreds of packages with that name in it on npm, e.g. https://www.npmjs.com/package/lodash.foreach. Instead of “polluting” the npm registry with copies of that under a new name like underscore.foreach (i.e. deprecating hundreds of packages and creating new copies instead, only for the new name), it might make more sense to just deprecate the one “underscore” package in favor of “lodash” and the related packages.

@bjmiller
Copy link

Both names are highly depended on. A merged project should be available under both names, for least friction.

@rafde
Copy link

rafde commented May 21, 2015

So would lodash function behaviors be a super set of underscore? Or the other way around?
I tried to replace underscore with lodash 3.x and a function like _.extend differ. Underscore will copy all props (even prototypes) while lodash only copies hasOwnPropertys.
But I suppose that's what underscore 2.x will be, the breaking change.

@alexindigo
Copy link

What the governance model would be?

@alexindigo
Copy link

Re: @mathiasbynens

And as for the naming, how about keeping underscore for the lightweight version of lodash, like basic / core module, but the whole ecosystem will be lodash thing.

@mathiasbynens
Copy link

@bjmiller

Both names are highly depended on.

Agreed.

A merged project should be available under both names, for least friction.

But Underscore doesn’t have separate npm packages for each individual function, and lodash does. So, there would be no friction at all by just continuing to refer to the existing lodash modules. Creating copies of those on the other hand, would.

@spadgos
Copy link
Contributor

spadgos commented May 21, 2015

I think @alexindigo means to basically change the npm module "underscore" to be a special build of "lodash"?

@neberej
Copy link

neberej commented May 21, 2015

+1

  • Lodash custom build
  • annonated source
  • use of .tap instead of temporary variables
  • chaining

etc. Oh this is exciting.

@buddy-sandidge
Copy link

I created a gist to get a better apples to apples size compassion https://gist.github.com/xbudex/a03aaf17ab8f984c18f0

To try to get a more apples to apples compairison, this is what I did.

  • Downloaded underscore 1.8.3 and lodash-compat 3.9.0, unminified development builds of both
  • Used the very simple jsmin to remove comments in both underscore and lodash
  • Use $ jscs --fix to create a consistant style in both

The results:

  • Underscore is 587 lines at about 32K
  • Lodash is 2702 lines at about 155K

Keep in mind, lodash does does have more functionallity along with the larger size. There are probably flaws with this method, but it gets us closer to an apples to apples compairison.

@calvinmetcalf
Copy link

based on browserifing a file with module.exports = require('lodash|underscore');

regular minified minified + gzipped
underscore 53515 16690 5895
lodash 406335 51729 18942

commands were

browserify script.js | wc -c
browserify script.js | uglifyjs -mc | wc -c
browserify script.js | uglifyjs -mc | gzip | wc -c

edit correct size for lodash

@samiconductor
Copy link

  • If you stick with Underscore, consider moving to an organization like lodash
  • Underscore's docs are easier on the eyes while lodash's extensive examples provide a more complete picture

Thanks for creating the most used modules!

@basicallydan
Copy link

The Name: "Lodash" is very cute, but keeping "Underscore" makes the most sense, both for historical and _ variable reasons. Because npm doesn't allow name redirects (AFAIK), the merged project would be available as a module under both names for the foreseeable future.

Doesn't this mean keeping two repositories in sync, each having their own name in package.json? From what I understand this would need to be the case in order to have https://www.npmjs.com/package/lodash and https://www.npmjs.com/package/underscore continue to work. Am I wrong?

This is great, by the way!

@rtorr
Copy link

rtorr commented May 21, 2015

Could underscore (on npm) just become an official build of lodash (like lodash-underscore build used to be)? Merge docs as well.

@wheaties
Copy link

Have to agree on the comment "lazy by default" of Lodash. That right there helps immensely with many normal use cases. And _.get.

@jdalton
Copy link
Contributor

jdalton commented Jun 4, 2015

@schmod Getting waaay ahead of the cart here. Let's kick stuff around a bit.

@jashkenas
Copy link
Owner Author

Alright, I've quickly fleshed out a draft of what I think a consensus API might look like in the Underdash repo:

https://github.com/underdash/underdash

The idea being — require('underdash') gives you all of the core modules — the super-minimal 90% functions that are all you need a lot of the time. require('underdash/more') gives you everything.

Of course, you can require any or as few of the individual modules as you like, if that floats your boat.

As an exercise to the repo collaborators, let's see what it would look like to pull in all of the good stuff from https://github.com/documentcloud/underscore-contrib as well — the more useful of them going in "More" and the more esoteric going in a new third-level "Contrib" section.

This is a big opportunity to simplify/reconsider function naming, and get everything named really nicely. We should take advantage.

I may have missed and or fudged things ... feel free to edit as you like.

Have at it.

@tyscorp
Copy link

tyscorp commented Jun 4, 2015

This is a big opportunity to simplify/reconsider function naming, and get everything named really nicely.

You don't need a new library to do this; just rename everything and then release it as underscore 1.8.4.

@jdalton
Copy link
Contributor

jdalton commented Jun 4, 2015

Rock. Let's move discussion over to underdash then.

Time for gif parade?

@tiandavis
Copy link

@jdalton

Ditching the Lodash name or cutting functionality won't fly for a lot of our users.

Ditching the Underscore name won’t fly for a lot of users either.

Lodash is the dominate library in node/io.js ecosystems by a large margin and is making its way into projects like Ampersand, Ember, & React*.

Lodash is the dominate library...by a large margin? Do you actually have any numbers to back this up? Or is this just a cheap shot at Underscore?

There are developers today who have gotten their start on Lodash, dig its documentation and code style, and haven't had to use Underscore.

So let me get this straight...You thought you could make a better Underscore. Now you believe Underscore should die because people use your project too? And please do explain what you mean by "...and haven't had to use Underscore." Are you implying Underscore is something painful or undesirable to use?

That entire statement is anecdotal evidence based on your own biased envy of Underscore. You claim Lodash is the “dominate library in node/io.js”. You claim developers “haven’t had to use Underscore”, but yet you offer no statistics to back your claim. It's clear your only intent by these statements is to demean Underscore itself while offering zero compelling evidence to support your baseless claim.

Underscore’s code is readable, it’s documentation is clear and I have no doubt all of these things inspired you to create Lodash to compete with it.

We already know your position. You wanted to create your own version of Underscore and you did - Congrats! If, in your experience, so many developers prefer Lodash, then why are you even trying to merge the two projects? Clearly, you believe your project is so much better right? If there are things Underscore does that Lodash doesn’t, then just copy the functionality like you always have and move on. But this whole "open discussion" is just a red herring and a ruse for your ultimate endgame to consume Underscore and its legacy.

@jamiebuilds
Copy link
Contributor

Do you actually have any numbers to back this up? Or is this just a cheap shot at Underscore?

Data is king.

Don't troll.

@EvanCarroll
Copy link

Do you actually have any numbers to back this up? Or is this just a cheap shot at Underscore?

It's actually /way/ more dominant than that. Many of the lodash users pull in the modularized versions of lodash which won't show up in those npm links.

@tiandavis
Copy link

@thejameskyle

Data is king.

So what? All I see is two popular libraries. One of them inspired by the other.

Don't troll.

Ad hominem.

@jdalton
Copy link
Contributor

jdalton commented Jun 19, 2015

Hey @tiandavis!

Ditching the Underscore name won’t fly for a lot of users either.

Yap, I agree. I wasn't suggesting Underscore change its name either.
I think we can at least share a common core though.

Lodash is the dominate library...by a large margin? Do you actually have any numbers to back this up? Or is this just a cheap shot at Underscore?

Sure thing. Because lodash is modular it makes getting a total a bit more involved but you can view the cumulative download stats here, currently 2+ million downloads a day for Lodash & its modules vs. ~300k for Underscore. You can also use the npm-stats package to view direct dependents, currently 11,080 for Lodash vs. 9,391 for Underscore. Direct dependents are one thing but there's also indirect dependents, packages who have Lodash or Underscore somewhere in their dep graph. Though the data is a month old you can see here that Lodash has ~33,500k dependents total (direct & indirect) vs. ~14,761 for Underscore. Then there's each package's trending growth. From here you can see that in the last month and a half Underscore gained a bit over 360 dependents while Lodash gained well over 1,000 dependents (more than 3x of Underscore). In addition, request is on track to replace Underscore as the 3rd most directly depended on npm package by September of this year. Meaning within the span of a year Lodash has risen to the top spot while Underscore will have fallen from 1st to 4th with no sign of recovery.

So let me get this straight...You thought you could make a better Underscore.

Our goal has been getting high quality utility methods to devs regardless of environment, module or package preference, style, or composition.

Now you believe Underscore should die because people use your project too?

I don't think I've said that. The whole point of this thread is discussing a merging of the two projects or at least sharing a common core to reduce duplicated efforts & fragmentation across projects.

And please do explain what you mean by "...and haven't had to use Underscore." Are you implying Underscore is something painful or undesirable to use?

Not at all. I was merely stating that Lodash is established enough that there are some devs today who have gotten their start with Lodash instead of Underscore.

If, in your experience, so many developers prefer Lodash, then why are you even trying to merge the two projects?

I encourage you to take some time to reread this thread and its related thread.

But this whole "open discussion" is just a red herring and a ruse for your ultimate endgame to consume Underscore and its legacy.

Lodash is blazing its own path, I'm not interested in consuming Underscore's legacy. That said, I think we can work together to reduce duplicated efforts & fragmentation across projects.

@megawac
Copy link
Collaborator

megawac commented Jun 19, 2015

Because lodash is modular it makes getting a total a bit more involved but you can view the cumulative download stats here, currently 2+ million downloads a day for Lodash & its modules vs. ~300k for Underscore.

I don't think that's a fair statement as installing one module may install n others (because they depend on one another); i.e. its hard to judge how many real downloads lodash's modules have.

Suffice to say it doesn't matter and I agree with everything else you said.

@jdalton
Copy link
Contributor

jdalton commented Jun 19, 2015

@megawac

I don't think that's a fair statement as installing one module may install n others (because they depend on one another); i.e. its hard to judge how many real downloads lodash's modules have.

That's cool. You can judge each by their monolithic build numbers then. Either way Lodash comes out on top. I think the other data points I've provided have more than covered it.

Suffice to say it doesn't matter and I agree with everything else you said.

Rock!

@tiandavis
Copy link

@jdalton For the record, I've followed both threads from the beginning. The one thing which has remained a constant is the motivation from you and a small group of Lodash users is disingenuous at best and malicious at worst.

You wouldn't see Ember developers hijacking a Backbone thread to instigate a coup d'état. So why are you pulling this PR stunt here?

Demean Underscore with your left hand and then pretend you didn't mean anything by it with your right. It's all a "misunderstanding" - I get it. One position I'm not changing is I don't believe a word you say.

@jamiebuilds
Copy link
Contributor

Let's try not to derail this issue please. No one has malicious intent here, so let's not devolve into bait trolling. Attacking @jdalton accomplishes nothing... cough cough Ad hominem.

@jdalton
Copy link
Contributor

jdalton commented Jun 19, 2015

@tiandavis

For the record, I've followed both threads from the beginning. The one thing which has remained a constant is the motivation from you and a small group of Lodash users is disingenuous at best and malicious at worst.

As another who has followed both threads I'm puzzled at how you could come to that conclusion.

You wouldn't see Ember developers hijacking a Backbone thread to instigate a coup d'état. So why are you pulling this PR stunt here?

There is no hijacking or coup d'état. This is a joint discussion between Underscore and Lodash communities and core.

Demean Underscore with your left hand and then pretend you didn't mean anything by it with your right. It's all a "misunderstanding" - I get it. One position I'm not changing is I don't believe a word you say.

I'm sorry you feel that way. Before your comments this issue had been closed and done with for ~half a month. Further discussion can be had over on underdash.

@CrossEye
Copy link

@tiandavis:

While I might also wonder a bit at the motivation, please note that Jeremy thought enough of this idea to bring it forth to the community. This is not @jdalton trying to break in and steal the silver. This really is both teams trying to work together.

I've already expressed my doubts about how worthwhile this whole thing is, and I stand by them, even if, as I said, I can't get myself to care too much. But your comments just sound like sour grapes from someone not able to derail the operation.

Repository owner locked and limited conversation to collaborators Jun 19, 2015
@megawac
Copy link
Collaborator

megawac commented Jun 19, 2015

Locking this issue as it's no longer constructive.

Repository owner unlocked this conversation Jun 19, 2015
@akre54
Copy link
Collaborator

akre54 commented Jun 19, 2015

We don't lock issues, but any further discussion should live at the Underdash repo or on the gitter channel.

@tiandavis
Copy link

@jdalton

As another who has followed both threads I'm puzzled at how you could come to that conclusion.

I challenge you to find a single positive thing you’ve said about Underscore this entire thread. Can't...?

Ok, then I challenge you to point to a place here where you didn't encourage - with your silence - your followers to trash Underscore. Can't...?

Let's try 3/3...

Point to a single instance where support for Underscore was raised and you had a civil exchange of ideas without outright dismissing the other developer. Can't...?

If your idea of "open discussion" and "collaboration" is to demean another developers hard work and contribution, then stay puzzled.

@CrossEye

"...derail the operation"

Freudian slip or nah? Quick, better hit that edit button before you reveal any more collusion. Just kidding, I already saw how @jdalton "collaborated" with you above:

RE @jdalton: @CrossEye to help the discussion and make it easier to digest can you try to keep your points on track and tangents to a minimum.

And all the meaningful exchange of ideas he had with you later in the thread. Oh…there wasn't any?

@thejameskyle

Attacking @jdalton accomplishes nothing... cough cough Ad hominem.

Tu Quoque.

@megawac

Locking this issue as it's no longer constructive.

You wanted an exchange of ideas, well here's one...

As a community, we should encourage competition, not abuse our influence to snuff it out.

Where would we be if IBM, in all its former glory, abused its influence to make sure each and every one of us programmed COBOL for a living? Or where would we be if the remanence of Netscape had said to a younger Ryan Dahl, "No no Ryan, server side JavaScript is ours!!" Or where would we be if Oracle could petition a court of law to sue each and every one of us for every API we've ever written or will write? …Too soon?

As a matter of principle, an open source project must be allowed to thrive or perish on its merit. At almost 65 million downloads over the last two years (keep in mind, the project has been out for longer than that), Underscore has achieved more merit than most of us could ever hope to achieve for a project. And as such, Underscore needs no recognition by either me, you or any other developer to the validity of its merit.

This blatant, clearly orchestrated, campaign to bring about its demise is an affront to the very nature of open source software - an ecosystem that every single one of us here derive our livelihood. In your mind, it’s just taking out the "competition", I understand. Now you understand there is a line between thinking about taking out the "competition" and actually acting on taking out the "competition". That line was crossed and it is not ok.

@jamesplease
Copy link

ahahaha who is this guy

@jridgewell
Copy link
Collaborator

@tiandavis: Stop. Your comments are not helpful. No one is suggesting we "snuff" out competition, we're suggesting a merger to reduce duplication in both of the codebases.

@jamesplease
Copy link

@tiandavis just fyi i saw @jridgewell and @jdalton out at brunch last week...I think he's a part of the conspiracy, too!!!

@akre54
Copy link
Collaborator

akre54 commented Jun 20, 2015

Chill out, all of you. Too much negativity in my inbox.

@jdalton
Copy link
Contributor

jdalton commented Jun 20, 2015

@tiandavis

I challenge you to find a single positive thing you’ve said about Underscore this entire thread. Can't...?

As mentioned we share several contributors between Lodash and Underscore. The fact that we're discussing, collaborating, and working together is a positive thing.

Ok, then I challenge you to point to a place here where you didn't encourage - with your silence - your followers to trash Underscore. Can't...?

The thread is a bit long but I think most have been respectful. Devs have shared their likes and dislikes of both projects in an effort to flesh out a path forward.

Point to a single instance where support for Underscore was raised and you had a civil exchange of ideas without outright dismissing the other developer. Can't...?

I don't think anyone is being dismissive. Libs are hard work.

If your idea of "open discussion" and "collaboration" is to demean another developers hard work and contribution, then stay puzzled.

I don't think I've demeaned another's hard work or contributions. Lodash isn't developed in a vacuum. Because it's a fork there is --a lot-- of cross lib collaboration and planning to ensure developers can use either without headache. That collaboration wouldn't work well if we were hostile towards each other. We help and push each other.

@michaelficarra
Copy link
Collaborator

Sorry @akre54, but there is nothing of value going on in this thread. It should have remained locked.

Repository owner locked and limited conversation to collaborators Jun 21, 2015
@jdalton
Copy link
Contributor

jdalton commented Jul 15, 2015

Hey all. I'm prepping lodash v4 which tackles many of the goals/items of the merge to proof things out and I'd dig some feedback over on the Underdash issue. Thanks! 📢

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests