Local web development workflow broken

[dabreegster]

trunk serve can't be used for local development anymore. I'm getting CORS errors: Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://unpkg.com/[email protected]/dist/leaflet.js. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 301.

If I understand modern web dev best practices, the "right" answer is to instead grab local copies of our JS dependencies like leaflet and serve them. In osm2streets we're manually vendoring the dependencies here, but it's an unpleasant process. I manually grabbed files from unpkg, and watched browser network requests to find all of the files.

Of the plethora of dependency management + bundler tools out there, what should we use? What'll play nice with trunk? Can https://trunkrs.dev/assets/#script-assets help?

[dabreegster]

trunk-rs/trunk#273
trunk-rs/trunk#32 (comment)

Not sure of the easiest path forward today. We have just 2 dependencies (leaflet and a gesture plugin).

[dabreegster]

I'd also vote for moving away from trunk and towards the straightforward wasm-pack + local python HTTP server. It adds layers of confusing indirection and not much value that I see.

[BudgieInWA]

Developing the app itself in rust (trunk) vs vanilla JS vs TS is a matter of preference. If simplicity of architecture is most important, then vanilla JS seems sensible to be - especially for prototyping and dogfooding the JS build of the library.

If we're trying to make a fully featured and polished web app, I think we're going to end up wanting a good templating solution, some third party components, etc. Buying into some level of framework is certainly a time-efficient way of solving a whole bunch of boring problems, but of course they all have their own drawbacks.

I would personally reach for React, because that is the UI paradigm I am familiar with, and I would probably reach for create-react-app so I don't have to think about how to set up the JS tooling.

[BudgieInWA]

(from the other thread)

Is there a simple tool out there to grab all the final distribution files for an NPM package?

That sounds like just npm to me :D (or alternative yarn). It can be used to get the packages, then the whole node_modules directory can be added to source control (along with the package.json and yarn.lock).

ben@ben-xps:~/code/blah$ yarn add leaflet
yarn add v1.22.19
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 1 new dependency.
info Direct dependencies
└─ [email protected]
info All dependencies
└─ [email protected]
Done in 0.90s.
ben@ben-xps:~/code/blah$ ls node_modules/leaflet/dist
images  leaflet.css  leaflet.js  leaflet.js.map  leaflet-src.esm.js  leaflet-src.esm.js.map  leaflet-src.js  leaflet-src.js.map

[dabreegster]

I'm attempting this in https://github.com/a-b-street/osm2streets/tree/yarn but I suspect something is already going wrong...

1. When I did yarn init, it didn't recognize the existing git repo, and so I had to manuallyish bring over the files it made
2. It's generating a few ~2MB files like .pnp.loader.mjs, seemingly part of https://yarnpkg.com/features/zero-installs
3. yarn add leaflet-osm times out
4. Nothing actually produces node_modules or similar, like yarn install

yarn --version says 3.2.4, but half of the Google results are for "classic" yarn. So... I'm pretty lost. Am I doing something majorly wrong? Or maybe I'll try npm

[BudgieInWA]

Hmm, turns out I am using "yarn classic" 1.22. I guess I had heard about yarn modern back when it was coming out and chosen not to upgrade because it does things differently, before forgetting altogether!

I'd recommend starting fresh and skipping the yarn init step: we don't need anything other than dependencies in package.json. Try to yarn add leaflet-osm without using pnp. If that doesn't work easily, try yarn 1.x, and if that's tricky, just use npm instead!

[dabreegster]

I got files downloaded locally by using npm: https://github.com/a-b-street/osm2streets/tree/npm/street-explorer/www

But now I guess the bundler problem is next. I changed the .html to just reference node_modules directly. We wouldn't need to commit all 19MB to git necessarily; npm ci should reproduce it from the lockfile on CI or any developer's machine. But just looking for the equivalent files in each directory to include doesn't quite work -- I hit require is not defined <anonymous> http://0.0.0.0:8000/node_modules/graphviz/lib/graphviz.js:4 and a few other errors. From googling around, serving the entire node_modules directory isn't recommended anyway. So as I understand it, we should also use a bundler and call it in serve_locally.py and the github actions workflow. Or even better -- that bundler ought to be able to replace serve_locally.py entirely by also being a tiny HTTP server.

I guess I'm still struggling to detangle the ecosystem of different options. There's a spectrum between full frameworks with scripts that auto-create all the base config, and the totally vanilla JS option using CDNs that we've done so far. We moved onto manually copying dependencies from the CDN. Now we're using a little automation to grab those packages, but they bring in way too much, and not in an immediately servable form.

If you weren't using React or another full framework in one of your projects, but you grew past the point of manually copying files from a CDN, what would your vanilla JS stack be?

[BudgieInWA]

I'll admit that I've only got a rudimentary understanding of the full npm and packaging ecosystem.

I haven't worked on any vanilla JS projects of any size recently. One of the benefits of going the vanilla JS route is avoiding any build tooling at all. As we've just discovered npm is not suitable for a tooling-free approach. There are a bunch of different systems that have evolved that do "javascript modules" (a quick google shows CommonJS, UMD, AMD, Require.js, and Es6 module
as options) and each has their own dependency resolution/loading mechanism. I guess each package on npm chooses which systems to support (and tends to support whatever Node expects). I have only looked into this stuff to the extend needed to fix various problems-at-hand in existing build systems that I have been working in.

The way that vanilla JS has worked, is the CDN approach, where each dependency publishes their JS (usually minified) on their website for you to download and add to your www dir. On reflection, your question of "how do we get the files in an npm package" has the false assumption that those files would be useful without a bundler! The example that we looked at - leaflet - has a dist directory in its npm package to serve our workflow, but that is the exception, not the rule! The workflow is to find the "download" section of the website offering the dependency that we're looking at and following any advice given there. Leaflet has https://leafletjs.com/download.html#using-a-downloaded-version-of-leaflet which lets us know we should be serving the images folder that's part of the download offered at the the top of the page.

In the modern ecosystem that approach might limit the dependencies that we have access to more than we like, and maybe we should be looking to set up a bundler and pay the cost of requiring tooling for local dev. If so, I would recommend starting choosing one of the bundler options out there and using whatever module system that expects (treating those details as implementation details that we hopefully don't have to dive into). I'm not across the current bundler landscape, but things to consider would include install/config/cli ergonomics, build features (like tree-shaking, TypeScript, custom build steps, etc.), build speed, community buy-in, ... I don't know how the choice of bundler interacts with the choice of npm vs yarn, but I prefer the ergonomics of Yarn Classic over npm. I'd be interested to try Yarn Modern with a bundler that plays nicely.