Reliable JavaScript/SourceMap processing via `DebugId` #81

Swatinem · 2023-03-21T12:57:34Z

We want to make processing / SourceMap-ing of JavaScript stack traces more reliable.
To achieve this, we want to uniquely identify a (minified / deployed) JavaScript file using a DebugId.
The same DebugId also uniquely identifies the corresponding SourceMap.
That way it should be possible to reliably look up the SourceMap corresponding to
a JavaScript file.

Rendered RFC

mitsuhiko · 2023-03-22T16:06:22Z

Ad unresolved questions: I think the debug_id reference in the sourcemap is something that makes quite a bit of sense to change.

iambriccardo · 2023-03-24T14:47:52Z

Good work @Swatinem on the RFC, I am a bit confused by the cons of the Add the DebugId to a global at load time. Is it a big con the parsing of the Error.stack? To me, considering the other options, it is more of a pro, unless it is computationally expensive but on this I don't know.

lforst · 2023-03-26T01:34:14Z

text/0081-sourcemap-debugid.md

+
+**cons**
+
+- Might incur some async fetching / IO when capturing an Error. Though any `abs_path` in the stack trace should be cached already.


This doesn't sound like a good idea to me. Mostly gut feeling but some other things come to mind:

Resource could be large

There might be delay in fetching the resource causing racing situations with navigations and so on

(I'll add additional reasons here when they come to mind)

text/0081-sourcemap-debugid.md

lforst · 2023-03-26T01:42:18Z

text/0081-sourcemap-debugid.md

+- It does however require parsing of the `Error.stack` at time of capturing the `Error`.
+
+An alternative implementation might use the [`import.meta.url`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta)
+property. This would avoid capturing and post-processing an `Error.stack`, but does require usage of ECMAScript Modules.


but does require usage of ECMAScript Modules

IMO this is a deal breaker. At least while IE is still somewhat relevant.

I'd rather build for the future, as when this spec become widespread the IE will be no more.

This is an implementation detail anyhow so instead of building for the future I'd rather build for compatibility right now.

Once this is being implemented as bundler plugins, its possible to also make this configurable, with appropriate defaults. For example, if a bundler is configured to output ESM, it might inject the more compact ESM snippet, falling back to the other one in case of CommonJS/UMD output.

text/0081-sourcemap-debugid.md

marandaneto · 2023-03-27T07:37:14Z

Have we considered languages/frameworks that are transpiled to JS? Such as Flutter Web (Dart -> JS), you don't have much control over the transpilation and minification steps, nor you do have build hooks, not sure how the inject would work here.

Swatinem · 2023-03-27T09:40:05Z

@marandaneto
I have not considered Dart / Flutter at all so far, and I do not know anything about it either. Can you give more details here? How does the compilation pipeline look like? Does Dart use SourceMaps? If so, how? Do developers usually combine that pipeline with other JS-native build tools/bundlers?

marandaneto · 2023-03-27T09:48:53Z

@marandaneto I have not considered Dart / Flutter at all so far, and I do not know anything about it either. Can you give more details here? How does the compilation pipeline look like? Does Dart use SourceMaps? If so, how? Do developers usually combine that pipeline with other JS-native build tools/bundlers?

Yeah Flutter web spits out source maps, and our symbolication works except for a few optimizations on the Dart side (as of now not a problem at all).
I guess it's possible to combine with other build tools/bundlers but it's not a common case at all, nor I have seen that, nor is documented, but maybe technically possible, I guess we don't need to support this at all at least as of now.
People just do flutter build web --source-maps and the magic happens.
Not sure which tooling flutter uses for source maps generation, would need to dig into the source code, here and here.

the-spyke · 2023-03-29T18:38:15Z

Could be also hash(minifiedContent + sourceMap)

We want to make processing / SourceMap-ing of JavaScript stack traces more reliable. To achieve this, we want to uniquely identify a (minified / deployed) JavaScript file using a DebugId. The same DebugId also uniquely identifies the corresponding SourceMap. That way it should be possible to _reliably_ look up the SourceMap corresponding to a JavaScript file, which is necessary to have reliable SourceMap processing.

lforst · 2023-03-31T09:04:33Z

Could be also hash(minifiedContent + sourceMap)

Don't know about this, since in some cases we don't have either minifiedContent or sourceMap available. (e.g. when injecting during bundler builds)

There are ways to make the debug id deterministic, but I don't think we can or should enforce the "origin of determinism".

iambriccardo · 2023-04-03T06:50:23Z

Considering that our goal is to capture change as defined by any change in the non-minified source, the hash of the source map is our best bet. The only caveat I am thinking of is if there is the possibility that hash(sm1) != hash(sm2) if sc1 == sc2 where sm is the source map and sc is the non-minified source. An example of this is if the bundler omits/adds some information in the source map for some reason even if the original source stays the same.

mitsuhiko · 2023-04-06T07:38:27Z

Auxiliary options with source hashes: Original Spec from Edge

Swatinem · 2023-04-06T09:34:15Z

Do you know if the SourceHash proposal still takes suggestions? I would recommend to be able to explicitly state the hash algorithm, similar to https://github.com/dotnet/runtime/blob/main/docs/design/specs/PE-COFF.md#pdb-checksum-debug-directory-entry-type-19 to make it forward compatible to changing the hash algorithm.
In your example SourceBundle manifest, you have prefixed the digest with the algorithm as well. And that example snippet is missing the corresponding hash for the SourceMap entry.

mitsuhiko · 2023-04-07T10:44:04Z

@Swatinem as the source hash proposal is already implemented in Chrome I'm not sure how many changes will happen there.

I moved the source hash proposal from my comment here: #85

mitsuhiko · 2023-04-07T11:00:26Z

I think we generally want to merge this RFC with some minor edits, but we need to also document somewhere, what the changes to the artifact bundle are. I'm okay if this is not in the RFC as we already have it somewhere, but as it stands it's undocumented.

Swatinem changed the title ~~Be able to look up SourceMaps by DebugId~~ Reliable JavaScript/SourceMap lookup via DebugId Mar 22, 2023

Swatinem changed the title ~~Reliable JavaScript/SourceMap lookup via DebugId~~ Reliable JavaScript/SourceMap processing via DebugId Mar 22, 2023

Swatinem force-pushed the sourcemap-debugid branch from 32e2939 to 0a6d0ca Compare March 23, 2023 10:42

Swatinem marked this pull request as ready for review March 23, 2023 13:17

Swatinem mentioned this pull request Mar 23, 2023

Broken source file after uploading sourcemaps getsentry/sentry-cli#1532

Closed

lforst reviewed Mar 26, 2023

View reviewed changes

marandaneto reviewed Mar 27, 2023

View reviewed changes

text/0081-sourcemap-debugid.md Outdated Show resolved Hide resolved

marandaneto reviewed Mar 27, 2023

View reviewed changes

text/0081-sourcemap-debugid.md Show resolved Hide resolved

Swatinem force-pushed the sourcemap-debugid branch from e88ea47 to f9682c5 Compare March 31, 2023 08:46

Swatinem added 2 commits April 3, 2023 15:09

clarify deterministic IDs

c944e7a

inject at the top

4949f3d

mitsuhiko mentioned this pull request Apr 7, 2023

Source Hash Extension for Source Maps #85

Open

Larger update to the RFC

b09d7b4

Merge branch 'main' into sourcemap-debugid

33a6644

mitsuhiko merged commit ab0f755 into main Apr 13, 2023

mitsuhiko deleted the sourcemap-debugid branch April 13, 2023 08:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reliable JavaScript/SourceMap processing via `DebugId` #81

Reliable JavaScript/SourceMap processing via `DebugId` #81

Swatinem commented Mar 21, 2023 •

edited

Loading

mitsuhiko commented Mar 22, 2023

iambriccardo commented Mar 24, 2023

lforst Mar 26, 2023

lforst Mar 26, 2023

the-spyke Mar 29, 2023

lforst Mar 30, 2023

Swatinem Mar 31, 2023

marandaneto commented Mar 27, 2023 •

edited

Loading

Swatinem commented Mar 27, 2023

marandaneto commented Mar 27, 2023 •

edited

Loading

the-spyke commented Mar 29, 2023

lforst commented Mar 31, 2023 •

edited

Loading

iambriccardo commented Apr 3, 2023 •

edited

Loading

mitsuhiko commented Apr 6, 2023 •

edited

Loading

Swatinem commented Apr 6, 2023

mitsuhiko commented Apr 7, 2023

mitsuhiko commented Apr 7, 2023


		cons

		- Might incur some async fetching / IO when capturing an Error. Though any `abs_path` in the stack trace should be cached already.

Reliable JavaScript/SourceMap processing via DebugId #81

Reliable JavaScript/SourceMap processing via DebugId #81

Conversation

Swatinem commented Mar 21, 2023 • edited Loading

mitsuhiko commented Mar 22, 2023

iambriccardo commented Mar 24, 2023

lforst Mar 26, 2023

Choose a reason for hiding this comment

lforst Mar 26, 2023

Choose a reason for hiding this comment

the-spyke Mar 29, 2023

Choose a reason for hiding this comment

lforst Mar 30, 2023

Choose a reason for hiding this comment

Swatinem Mar 31, 2023

Choose a reason for hiding this comment

marandaneto commented Mar 27, 2023 • edited Loading

Swatinem commented Mar 27, 2023

marandaneto commented Mar 27, 2023 • edited Loading

the-spyke commented Mar 29, 2023

lforst commented Mar 31, 2023 • edited Loading

iambriccardo commented Apr 3, 2023 • edited Loading

mitsuhiko commented Apr 6, 2023 • edited Loading

Swatinem commented Apr 6, 2023

mitsuhiko commented Apr 7, 2023

mitsuhiko commented Apr 7, 2023

Reliable JavaScript/SourceMap processing via `DebugId` #81

Reliable JavaScript/SourceMap processing via `DebugId` #81

Swatinem commented Mar 21, 2023 •

edited

Loading

marandaneto commented Mar 27, 2023 •

edited

Loading

marandaneto commented Mar 27, 2023 •

edited

Loading

lforst commented Mar 31, 2023 •

edited

Loading

iambriccardo commented Apr 3, 2023 •

edited

Loading

mitsuhiko commented Apr 6, 2023 •

edited

Loading