feat(gnovm): packages loader #2932

n0izn0iz · 2024-10-10T11:30:05Z

WIP

Continuation of #2201
Taking #2904 into account

This implements a Load function with a similar interface to the packages.Load go function

Contributors' checklist...

Added new tests, or not needed, or not feasible
Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory
Updated the official documentation or not needed
No breaking changes were made, or a BREAKING CHANGE: xxx message was included in the description
Added references to related issues and PRs
Provided any useful hints for running manual tests
Added new benchmarks to generated graphs, if any. More info here.

…esolver

Signed-off-by: Norman Meier <[email protected]>

…no-resolver

codecov · 2024-10-10T11:32:47Z

Codecov Report

Attention: Patch coverage is 74.02707% with 307 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
gnovm/pkg/packages/expand_patterns.go	60.71%	47 Missing and 8 partials ⚠️
gnovm/pkg/packages/load.go	69.44%	38 Missing and 6 partials ⚠️
gnovm/pkg/gnofiles/match.go	81.97%	32 Missing and 10 partials ⚠️
gnovm/pkg/packages/analyze_packages.go	71.96%	26 Missing and 11 partials ⚠️
gnovm/cmd/gno/list.go	37.50%	34 Missing and 1 partial ⚠️
gnovm/pkg/gnofiles/gnofiles.go	10.71%	24 Missing and 1 partial ⚠️
gnovm/cmd/gno/lint.go	76.66%	11 Missing and 3 partials ⚠️
gnovm/cmd/gno/transpile.go	77.77%	8 Missing and 4 partials ⚠️
gnovm/cmd/gno/mod.go	68.00%	6 Missing and 2 partials ⚠️
gnovm/pkg/packages/pkglist.go	89.04%	6 Missing and 2 partials ⚠️
... and 9 more

📢 Thoughts on this report? Let us know!

Signed-off-by: Norman Meier <[email protected]>

…ve download to own file Signed-off-by: Norman Meier <[email protected]>

Signed-off-by: Norman Meier <[email protected]>

moul · 2024-10-11T15:30:03Z

examples/gno.land/gno.mod

@@ -0,0 +1 @@
+module gno.land


do we need this file?

if we need it, please explain in a comment.

please see #2904 (comment) and tell me what you think

I think the best would be to have a gno.work file at repo root to support multiple modules in a monorepo
But in that case we need to implement the gno work commands

moul · 2024-10-11T15:32:03Z

gnovm/tests/machine_test.go

@@ -40,7 +40,7 @@ func TestMachineTestMemPackage(t *testing.T) {
 					Name: tt.name,
 					F: func(t2 *testing.T) { //nolint:thelper
 						rootDir := filepath.Join("..", "..")
-						store := TestStore(rootDir, "test", os.Stdin, os.Stdout, os.Stderr, ImportModeStdlibsOnly)
+						store := TestStore(rootDir, "test", nil, os.Stdin, os.Stdout, os.Stderr, ImportModeStdlibsOnly)


create a variable that you set to nil, so we now what is this parameter.

done in 0123512

gnovm/pkg/packages/download.go

moul · 2024-10-11T15:35:17Z

gnovm/pkg/packages/download.go

+func DownloadModule(pkgPath string, dst string) error {
+	modFilePath := filepath.Join(dst, ModfileName)
+	if _, err := os.Stat(modFilePath); os.IsNotExist(err) {
+		fmt.Fprintln(os.Stderr, "gno: downloading", pkgPath)


we usually don't use os.StdXXX, but stdouterr from a struct (io something) which is passed when needed.

done in e648305

Signed-off-by: Norman <[email protected]>

This makes the imports utils split imports by file kinds, allowing to make explicit decisions about what imports to use at the various callsites - Create `FileKind` enum to categorize gno files, with variants `PackageSource`, `Test`, `XTest` and `Filetest` - Create `GetFileKind` util to derive the `FileKind` from a file name and body - Create `ImportsMap` type that maps `FileKind`s to lists of imports. It has a single method `Merge` to select and merge various imports from multiple `FileKind`s - Modify the`packages.Imports` helper to return an `ImportsMap` instead of a `[]string` and adapt callsites by using`ImportMap.Merge` to preserve existing behavior This is something I need for #3304 and #2932 but to help reviews I made an atomic PR here instead --------- Signed-off-by: Norman Meier <[email protected]> Co-authored-by: Morgan <[email protected]>

Signed-off-by: Norman <[email protected]>

…t-cycle

Signed-off-by: Norman <[email protected]>

A step towards the importer package (#2932) and future of `gno.mod` (#2904) - BREAKING CHANGE: remove `require` statement support from `gno.mod` - BREAKING CHANGE: remove `-v` (verbose) and `--remote` flags in `gno mod download` - Don't require version specification in `gno.mod`'s `replace` statements - Use `.gno` files `import` statements to find dependencies - Extract and refacto imports gathering utils in `gnovm/pkg/packages` - Add `gnovm/cmd/gno/internal/pkgdownload.PackageFetcher` interface - Implement `PackageFetcher` using `vm/qfile` queries in `gnovm/cmd/gno/internal/pkgdownload/rpcpackagefetcher` - Rewrite single package download routine in `gnovm/cmd/gno/internal/pkgdownload` - Move and refacto dependencies download routine in `gnovm/cmd/gno/download_deps.go` - Add a `--remote-overrides` flag for `gno mod download` that takes `chain-domain=rpc-url` comma-separated pairs to override endpoints used to fetch packages - Add and use a testing implementation of `PackageFetcher` called `examplesPackageFetcher` that serves package from the `examples` directory for testing purposes (download tests before this PR use the portal loop public endpoint) - Make `ReadMemPackage` and it's dependencies error-out instead of panicking - Create panicking `MemPackage` utils that wrap the erroring ones and use them at existing callsites I decided to do this first to avoid having multiple ways to resolve dependencies lying around in the codebase and causing confusion in subsequent steps <details><summary>Contributors' checklist...</summary> - [x] Added new tests, or not needed, or not feasible - [ ] Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory - [ ] Updated the official documentation or not needed - [x] No breaking changes were made, or a `BREAKING CHANGE: xxx` message was included in the description - [x] Added references to related issues and PRs - [ ] Provided any useful hints for running manual tests </details> --------- Signed-off-by: Norman Meier <[email protected]> Co-authored-by: Morgan Bazalgette <[email protected]>

This makes the imports utils split imports by file kinds, allowing to make explicit decisions about what imports to use at the various callsites - Create `FileKind` enum to categorize gno files, with variants `PackageSource`, `Test`, `XTest` and `Filetest` - Create `GetFileKind` util to derive the `FileKind` from a file name and body - Create `ImportsMap` type that maps `FileKind`s to lists of imports. It has a single method `Merge` to select and merge various imports from multiple `FileKind`s - Modify the`packages.Imports` helper to return an `ImportsMap` instead of a `[]string` and adapt callsites by using`ImportMap.Merge` to preserve existing behavior This is something I need for #3304 and #2932 but to help reviews I made an atomic PR here instead --------- Signed-off-by: Norman Meier <[email protected]> Co-authored-by: Morgan <[email protected]>

thehowl and others added 8 commits May 26, 2024 13:41

copy over from gnoutil pr

01b0719

remove util_test

232834c

Merge branch 'master' of github.com:gnolang/gno into dev/morgan/gno-r…

f378217

…esolver

linter fixes

8afeacf

tmp: gno list

808061d

Signed-off-by: Norman Meier <[email protected]>

tmp

2c5d604

Signed-off-by: Norman Meier <[email protected]>

tmp: working

dfc0881

Signed-off-by: Norman Meier <[email protected]>

chore: merge remote-tracking branch 'origin/master' into dev/morgan/g…

a35050c

…no-resolver

github-actions bot assigned n0izn0iz Oct 10, 2024

github-actions bot added 🧾 package/realm Tag used for new Realms or Packages. 📦 🤖 gnovm Issues or PRs gnovm related 📦 ⛰️ gno.land Issues or PRs gno.land package related labels Oct 10, 2024

n0izn0iz added 4 commits October 10, 2024 14:47

feat: match mapping

7c7165f

Signed-off-by: Norman Meier <[email protected]>

fix: test and lint

1c3a830

Signed-off-by: Norman Meier <[email protected]>

fix: partially fix transpile

f95e4bb

Signed-off-by: Norman Meier <[email protected]>

tmp: support loaded pkgs in test store

2d990e5

Signed-off-by: Norman Meier <[email protected]>

n0izn0iz mentioned this pull request Oct 11, 2024

rfc: the future of gno.mod #2904

Open

n0izn0iz added 6 commits October 11, 2024 12:59

tmp: better Module info

6995485

Signed-off-by: Norman Meier <[email protected]>

tmp: follow go resolver logic

fb8d518

Signed-off-by: Norman Meier <[email protected]>

fix: resolver usage in test

1c51509

Signed-off-by: Norman Meier <[email protected]>

chore: mod tidy

da65417

Signed-off-by: Norman Meier <[email protected]>

chore: ignore draft

4a8401a

Signed-off-by: Norman Meier <[email protected]>

chore: rename importer -> packages like go, break import cycle and mo…

e792771

…ve download to own file Signed-off-by: Norman Meier <[email protected]>

n0izn0iz changed the title ~~feat(gnovm): package resolver lib~~ feat(gnovm): packages loader Oct 11, 2024

chore: lint and fmt

08fbd84

Signed-off-by: Norman Meier <[email protected]>

moul reviewed Oct 11, 2024

View reviewed changes

gnovm/pkg/packages/download.go Outdated Show resolved Hide resolved

moul reviewed Oct 11, 2024

View reviewed changes

n0izn0iz added 11 commits January 8, 2025 09:00

chore: don't override name if not requested

82b7ef8

Signed-off-by: Norman <[email protected]>

chore: skip stdlibs and package with no import paths in txs

9d6b2d8

Signed-off-by: Norman <[email protected]>

chore: fix lint

1405da2

Signed-off-by: Norman <[email protected]>

tmp

7bb33b2

Signed-off-by: Norman <[email protected]>

fix: linter errors locations

c95d2bf

Signed-off-by: Norman <[email protected]>

feat: single file pattern support + misc bugs

6d6ee0d

Signed-off-by: Norman <[email protected]>

chore: fmt and lint go

c5754b5

Signed-off-by: Norman <[email protected]>

fix: don't ignore files without valid package clause

a73ba04

Signed-off-by: Norman <[email protected]>

fix: mod why

cef59f3

Signed-off-by: Norman <[email protected]>

fix: pass all tests except transpile

b73027b

Signed-off-by: Norman <[email protected]>

tmp

5b5386f

Signed-off-by: Norman <[email protected]>

n0izn0iz added 11 commits January 9, 2025 21:50

fix: always use relative paths for files in pkgs

07375f8

Signed-off-by: Norman <[email protected]>

chore: revert transpile changes

95963dc

Signed-off-by: Norman <[email protected]>

chore: Merge remote-tracking branch 'origin/master' into forbid-impor…

43c95d4

…t-cycle

chore: remove merge artifact

6e1a134

Signed-off-by: Norman <[email protected]>

chore: Merge branch 'forbid-import-cycle' into gno-resolver

6f6bb36

fix: sort

1d67048

Signed-off-by: Norman <[email protected]>

chore: fix debugger

c0eacfb

Signed-off-by: Norman <[email protected]>

feat: detect stdlibs

740e3e3

Signed-off-by: Norman <[email protected]>

chore: Merge remote-tracking branch 'origin/master' into gno-resolver

7523824

fix: gnodev, gnogensis

9380f1c

Signed-off-by: Norman <[email protected]>

chore: lint and fmt

2f9c35a

Signed-off-by: Norman <[email protected]>

github-actions bot added the 📦 🌐 tendermint v2 Issues or PRs tm2 related label Jan 10, 2025

n0izn0iz added 2 commits January 10, 2025 05:45

chore: generate

85c800a

Signed-off-by: Norman <[email protected]>

test: gnodev

5b06874

Signed-off-by: Norman <[email protected]>

zxxma mentioned this pull request Jan 13, 2025

Teritori's Builder Journey gnolang/hackerspace#7

Open

Kouteki mentioned this pull request Jan 15, 2025

Minutes: Core Staff Weekly Syncs [every Monday] gnolang/meetings#36

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(gnovm): packages loader #2932

feat(gnovm): packages loader #2932

n0izn0iz commented Oct 10, 2024 •

edited

Loading

codecov bot commented Oct 10, 2024 •

edited

Loading

moul Oct 11, 2024

n0izn0iz Oct 11, 2024

n0izn0iz Oct 11, 2024 •

edited

Loading

moul Oct 11, 2024

n0izn0iz Oct 11, 2024

moul Oct 11, 2024

n0izn0iz Oct 11, 2024

feat(gnovm): packages loader #2932

Are you sure you want to change the base?

feat(gnovm): packages loader #2932

Conversation

n0izn0iz commented Oct 10, 2024 • edited Loading

codecov bot commented Oct 10, 2024 • edited Loading

Codecov Report

moul Oct 11, 2024

Choose a reason for hiding this comment

n0izn0iz Oct 11, 2024

Choose a reason for hiding this comment

n0izn0iz Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

moul Oct 11, 2024

Choose a reason for hiding this comment

n0izn0iz Oct 11, 2024

Choose a reason for hiding this comment

moul Oct 11, 2024

Choose a reason for hiding this comment

n0izn0iz Oct 11, 2024

Choose a reason for hiding this comment

n0izn0iz commented Oct 10, 2024 •

edited

Loading

codecov bot commented Oct 10, 2024 •

edited

Loading

n0izn0iz Oct 11, 2024 •

edited

Loading