Though these two software solutions carry mundane and short names, they have come a long way in the last few years to solve tough problems like deployment time management or module distribution across servers. To fully appreciate their stature in the open source community, it is worthwhile to understand the knotty problems they solve.
npm Solves ‘Dependency Hell’
Using proven software packages speeds development when building reliable web applications. This approach becomes even more compelling when the packages are open source. Open source means the code is available for inspection by anyone with an internet connection. Thus, open-source software packages have run the gauntlet of the world’s best software engineers.
The Problems: Dependencies and Versions
The software packages that comprise an application quite naturally depend on one another. More packages multiply dependency management. By 2010, a rat’s nest of barely manageable dependencies was all too frequent for developers. Managing the dependencies with a spreadsheet or other rudimentary solutions consumed enormous hours.
Versions were another headache. Developers often make use of a software package in several places of an application. New versions of a package have the potential to break an existing application. Therefore, developers like to selectively phase in a new package version in one section of an app and continue to use an older version elsewhere. Before package managers, attempting to install a newer package version frequently overwrote an older version. Along with other more profane terms, this aggravation won the name “dependency hell.”
The Back-End Solution: npm
January 2010 brought the Network Package Manager, and with it, a software revolution. npm was an integral component of node.js, a framework that enabled JavaScript programming for a website’s back-end operations. With each developer-selected package, npm combed the appropriate software registry for all dependent packages and organized the collection into a reliable and readable hierarchy tree.
Along with automating the dependency tree, npm also solved “dependency hell.” The nascent program manager ingeniously enclosed package versions in individual folders and assigned them to their proper place on the hierarchy tree. npm quickly spurred the creation of a massive package registry, enabling developers to access both free open-source software and premium packages with a license fee.
A Partial Front-End Answer: Bower
npm functioned beautifully for an application’s back end, the tasks involving servers and databases. At the front end, where users interact with the application, npm struggled. Installing unnecessary packages chokes web browsers and impairs user experience. Bower, a package manager from Twitter’s open-source arm, partially addressed this issue and enabled compact, efficient dependency trees for a web app’s client-side operations.
With Bower streamlining front-end performance and npm handling back-end duties, the combo flourished for the next three years. Nonetheless, the growing shortcomings of Bower began to cramp the style of client-side developers. Among the frustrated users was the world’s largest social media corporation, and its developers created a solution.
Yarn Fills the Need for Speed
In November 2016, Facebook introduced Yarn, an open-source front-end package manager. Able to take full advantage of the npm package registry, Yarn won strong support from Google and acclaim across the developer community. Among Yarn’s admirers were the founders of Bower, who moved their creation into deprecation and urged users to migrate to the new kid on the JavaScript block. While Yarn’s command-line syntax varied slightly from npm, software engineers mastered it with ease.
With Yarn, front-end developers finally gained a tool that matched npm’s back-end elan. Compared with Bower, Yarn built equally compact hierarchy trees and did so in a fraction of the time. Beyond streamlining front-end dependencies, Yarn delivered raw build speed. The Facebook-developed parallel algorithm simultaneously installed multiple packages, slashing build times by as much as 80%.
Caching in on Security
Compounding this advantage was a clever use of memory. Once a package was plucked from the registry, it was stored — cached in programming lingo — in memory and committed to the developer’s local disk. With subsequent installs, packages were available in an instant, with no further network interactions needed.
The caching scheme brought more than just impressive speed. The prime motive behind Facebook’s move into package management was security. Reducing network access shrank the target open to hackers. Facebook’s security push included two other innovations:
- Lock files: With the initial installation of each package, Yarn creates a lock file to memorize the package’s version and hierarchy position. The lock file guarantees identical builds on different systems and lessens the risk of malicious code leaking in.
- Checksums: Yarn’s hashed checksums effectively create a fingerprint for each package in a project, flagging the component as malicious with the change of even a single character.
While enjoying the warm reception for Yarn, the product’s development team was hard at work on a thorough overhaul. In January 2020, Yarn introduced version 2.0 and promptly careened into the sophomore jinx.
Yarn Meets Version Revulsion
Yarn 2 threw radical changes at developers, provided a complex migration path, and topped its troubles with inadequate documentation. This version also did not play well with React, a software tool that streamlines the development of iOS and Android applications. Adding the ultimate humiliation, Facebook’s in-house engineers refused to use the initial version. After creating better migration tools and building up the documentation, the Yarn team apologized to developers and pledged incremental changes for the future. With a cooling-off period, some developers have warmed to the changes.
Yarn 2 added substantial project management functions, particularly for mono-repositories. Mono-repos are giant collections of code for simultaneous use in multiple applications. Unsurprisingly, mono-repos are the preferred approach of Facebook and Google, Yarn’s prime backers. Mono-repos are controversial in the industry, so this effort is not a strong selling point for all developers.
Yarn’s new Plug’n’Play strategy is a radical attempt to leapfrog package installation inefficiencies. With PnP, Yarn seeks to shoulder the burden of locating packages instead of delegating the chore to the node.js architecture. Yarn’s team argues that this approach brings sizable benefits:
- Elimination of duplicate packages
- Even faster install times
- Cleaner hierarchy trees
- Snappier startup times for applications
Quick startup times are essential for smartphone apps, and delivering on this capability would be a coup for Yarn. Nonetheless, achieving these goals now requires developers to climb a substantial learning curve. Yarn won wide adoption by hewing close to npm’s syntax conventions, so it remains an open question of how many developers are willing to make the full commitment.
The npm Strikes Back
While the Yarn team rode the roller coaster of developer expectations, npm’s engineers quietly addressed their product’s shortcomings. npm5 introduced lock files, and in the view of many developers, this implementation was even more secure than Yarn’s solution. The same version introduced a revamped caching scheme that narrowed the speed gap with Yarn. The new cache system also implemented checks on each install to enhance security. With successive improvements, npm’s installation of cached and file-locked packages has now pulled ahead of Yarn on some benchmark sites.
In npm7, the venerable program manager added workspaces. A workspace is an essential software component for interacting with mono-repos. Rather than match Yarn’s ambitious workspace functionality, npm’s developers aimed to build compatibility with Lerna, a full-featured mono-repo manager with wide respect in the industry. Perhaps learning by Yarn’s example, npm rolled out features incrementally, taking care not to add too many code-breaking changes with each new version.
The Contenders Move Toward Coexistence
In 2018, Yarn made it possible to import lock files written in npm’s well-received format. In the same year, npm quietly simplified importing files written for Yarn projects. The respective development teams had twin motives for these moves. The first reason was to tangibly demonstrate good citizenship to the vast open-source community. Second, by making it easier to dip a toe in their product, the teams hoped to win long-term developer loyalty.
In 2020, Microsoft’s GitHub subsidiary acquired npm, inc. With both package managers now backed by corporate 800-pound gorillas, the competition is likely to remain robust for years. Developers with close ties to Google or Facebook will likely gravitate to Yarn. Software vendors who rely on the Microsoft ecosystem now have a strong incentive to adopt npm. For all other developers, the decision will fall to the preferences of the in-house engineers.
The Case for SCA Grows Stronger
With more and more packages used in modern web applications, the need for Software Composition Analysis (SCA) grows daily. Security is not an option, so whether you build your app’s dependency tree with Yarn or npm, SOOS will highlight your critical vulnerabilities. That same ease of use also holds for the languages and frameworks in your projects.
Along with security, maintaining compliance is equally vital. SOOS keeps you on top of your projects’ license status, so compliance issues will never derail your business. Keep your focus on deploying great applications–while selecting a package manager may be difficult, your choice of an SCA tool could not be easier: SOOS.
