Guidance on linking TypeScript packages together in a repo

[gregjacobs]

Hey there. I think the docs need some guidance on linking packages together in a repo.

Just for quick context: we have a monorepo of about 200 packages that we publish to npm. Some of these packages rely on each other, and need to refer to their dependencies by package name:

// Inside package-2:
import { Thing1, Thing2 } from 'package-1';

We can't import using relative paths like in some of the examples because then the imports wouldn't work correctly when the packages are npm installed by our users.

Some attempts at linking packages in our Bazel repo so far:

1. pnpm workspaces: This is recommended on the rules_js readme but doesn't seem to work for typescript packages because it links to the source folders rather than Bazel's output folders. So when we want to build package-2, we really need it to build against the output of package-1 (i.e. bazel-bin/package-1) rather than the source of package-1.
   
   
2. npm_link_package: I thought that this would solve my issue but observed some weird effects. Package-2 didn't actually need to declare //:node_modules/package-1 as a dependency for it to be available at build time (worker), and thus package-1's output could be stale when building package-2. Unfortunately, it's too easy to miss declaring all first party deps like this in a large repo when there is no build error for it. Also it isn't exactly desirable to put 200 npm_link_package entries into the root BUILD file (although I would if the stale package issue didn't exist!).
   
   
3. tsconfig.json paths: I added:

    paths: {
        "*": ["bazel-out/*"]
    }
   

   This got me building between typescript packages by having package-2's ts_project use package-1's ts_project rule in its deps, but sadly fails at runtime of package-2 because Node.js can't find the runtime files for package-1 (since 'package-1' doesn't exist in the node_modules tree for package-2).
   
   
4. NODE_PATH: Total hack to get case 3 working, I set NODE_PATH=path/to/bazel-out so that Node.js can find the "node_modules" (i.e. Node treats these as "global node modules" in this case). However, the problem I'm having with this now is that our directory structure doesn't match our package names, so NODE_PATH won't work for the package at components/alert/react because its package name is @uitoolkit-react/alert. (Just as an fyi, our directory structure optimizes the developer experience and code reviews because we have 'common' code that relates to the angular/react packages, and we can place those folders physically close together under a common directory)

As for one more point: if I'm recalling correctly, it's best for two ts_project rules to rely directly on each other to optimize the build graph between them since package-2 shouldn't need to rebuild if the .d.ts files for package-1 haven't changed. If so, I guess option 2 wouldn't be good since it would have the extra npm_package and npm_link_package rules in its graph such as the following?

flowchart RL
    tsproject1("ts_project (package-2)") --> npm_link_package
    npm_link_package --> npm_package
    npm_package --> tsproject2("ts_project (package-1)")

Loading

Anyway, looking for recommendations. Thanks!

[jbedard]

I think pnpm workspaces is still the best option and should work with ts. It's not merged yet but here's an example: aspect-build/bazel-examples#114

[gregjacobs]

@jbedard Interesting - I'll take a look at your PR for inspiration here, thanks for that.

One other disadvantage I was thinking about with pnpm workspaces though is keeping all of the 3rd party dependency versions in sync. Upgrading a commonly-used dependency across the repo will result in hundreds of package.json file changes showing up in a pull request. Having a single root-level package.json solves this issue. Any thoughts on that?

[jbedard]

You can avoid repeating the version everywhere doing "dep": "workspace:*" to declare the dependency (so it will get linked in the runfiles) but avoid specifying the version everywhere: https://pnpm.io/workspaces#referencing-workspace-packages-through-aliases

[gregjacobs]

@jbedard I'm familiar with doing "workspace:*" for 1st party dependencies inside the repo itself, but what about 3rd party dependencies from npm?

[jbedard]

Right... for third-party deps I think link: ../.... works? It's a bit uglier then "*" but avoids requiring changes to every package.json...

[jbedard]

^ nope, I don't think link: works either...?

[BeyondEvil]

@jbedard Interesting - I'll take a look at your PR for inspiration here, thanks for that.

One other disadvantage I was thinking about with pnpm workspaces though is keeping all of the 3rd party dependency versions in sync. Upgrading a commonly-used dependency across the repo will result in hundreds of package.json file changes showing up in a pull request. Having a single root-level package.json solves this issue. Any thoughts on that?

The example @jbedard referenced solves just that. The inspirational-quotes dependency is declared in the root package.json, and then used in the alpha and beta projects.

[BeyondEvil]

Let me know if you encounter a specific use-case wrt. internal or external dependencies that doesn't work you.

The approach when publishing packages to NPM requires a bit more "stuff" than what the aforementioned example contains.

[gregjacobs]

@BeyondEvil Ah, yes, we will be publishing these packages to npm so we can't simply inherit the dependencies from the repo's directory tree.

My current setup actually involves not having any package.json files at all in any of the packages (other than the one at the root level). For publishing, I'm dynamically generating a package.json for each package by scanning the .js/.ts source files in that package to find their import/require statements. I then compile a list of the module specifiers found and look up their version numbers in the root package.json. Using this information, I can generate a package.json with the package name, version, and the dependencies needed for that particular package. I'm liking that this setup doesn't involve duplicating the package's node_modules dependencies between the BUILD file and a manually-maintained package.json.

That being said, if pnpm workspaces are "the way", then I might do this a little differently

[jbedard]

@gregjacobs I think you might have to just put those into the root package.json and not specify them in each workspace package. Each workspace package will still specify the fine grained deps in the BUILD but no longer in each package.json.

This is what @BeyondEvil mentioned and is used in aspect-build/bazel-examples#114

[thesayyn]

@jbedard @gregjacobs can this be closed?

[jbedard]

I think it can be closed with pnpm workspaces as the common solution. aspect-build/rules_js#818 will also help when depending on 3rd-party packages (without having to repeat them in every package.json in a workspace).

[gregmagolan]

#313 landed which is a better alternative to aspect-build/rules_js#818. non-dev npm packages in deps will now automatically be propagated down to the direct deps of a linked npm_package that a ts_project feeds into.

I'll tag this issue as documentation since I think we needed better docs on how this should work.