fix: remove import cycles #3304

n0izn0iz · 2024-12-08T21:09:37Z

Depends on #3323

add test in examples/no_cycles_test.go to detect import cycles in stdlibs and examples
add matchString native injection in the testing stdlib to avoid a cycle in regexp tests
remove other import cycles

Go never allows import cycles. Our stdlibs have a lot of import cycles, and some examples import self which is not allowed in golang either.
Keeping support for import cycles in stdlib and importing self will require a lot of hacky and weird logic in generic package loading code so I try to tackle this first.

TODO:

fix tests
~~check cycles with the test stdlibs overlay applied~~ -> be explicit about the lack of support for modifying imports in testing stdlibs overlay

Signed-off-by: Norman Meier <[email protected]>

Gno2D2 · 2024-12-08T21:10:52Z

🛠 PR Checks Summary

All Automated Checks passed. ✅

Manual Checks (for Reviewers):

IGNORE the bot requirements for this PR (force green CI check)

✅ Automated Checks (for Contributors):

🟢 Maintainers must be able to edit this pull request (more info)

☑️ Contributor Actions:

Fix any issues flagged by automated checks.
Follow the Contributor Checklist to ensure your PR is ready for review.
- Add new tests, or document why they are unnecessary.
- Provide clear examples/screenshots, if necessary.
- Update documentation, if required.
- Ensure no breaking changes, or include BREAKING CHANGE notes.
- Link related issues/PRs, where applicable.

☑️ Reviewer Actions:

Complete manual checks for the PR, including the guidelines and additional checks if applicable.

📚 Resources:

Debug

Automated Checks
Maintainers must be able to edit this pull request (more info)
If
🟢 Condition met
└── 🟢 The pull request was created from a fork (head branch repo: n0izn0iz/gno)
Then
🟢 Requirement satisfied
└── 🟢 Maintainer can modify this pull request
Manual Checks
**IGNORE** the bot requirements for this PR (force green CI check)
If
🟢 Condition met
└── 🟢 On every pull request
Can be checked by

Any user with comment edit permission

codecov · 2024-12-08T21:11:01Z

Codecov Report

Attention: Patch coverage is 50.00000% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
gnovm/tests/stdlibs/testing/native_testing.go	62.50%	2 Missing and 1 partial ⚠️
gnovm/stdlibs/testing/testing.go	0.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

Signed-off-by: Norman Meier <[email protected]>

examples/gno.land/p/moul/md/md_test.gno

thehowl · 2024-12-10T10:42:12Z

contribs/nocycles/main.go

+	// find stdlibs
+	libs := []string{}
+	gnoRoot := gnoenv.RootDir()
+	stdlibsDir := filepath.Join(gnoRoot, "gnovm", "stdlibs")


for stdlibs, instead, i think you could change misc/genstd (note: this binary as well, if it were to stay, would also have to be misc/), by having its import order generator also consider test files

Indeed, when I cherry-pick c6728c5 on master, I get:

go run github.com/gnolang/gno/misc/genstd
panic: cyclical package initialization on "bytes" (bytes -> io -> bytes)

I like having code that is dedicated to detecting any cycles in stdlibs and examples though, I'll move the nocycles code into a test somewhere I think when I get to the polishing step on this PR

moved the test to examples/no_cycles_test.go

regarding genstd changes:

actually finding a single global order while including all test files is not possible, consider the following imports (which are allowed in go):

foopkg/foo_test.go imports barpkg

barpkg/bar_test.go imports foopkg

examining each packages tests deps will yield a different order:

foopkg tests:

barpkg

foopkg

barpkg tests:

foopkg

barpkg

we could fill a map[string][]string to get the testing order for each lib but I'm not sure this is really helpful, do you want me to add this?

Is this PR not meant to find that order?

The order should exist when excluding the Xtest files, no?

no, it only exists when you include solely the PackageSource files, the example above is with normal test files

The go stdlib makes it possible to find an order by putting all deps which would lead to circular dependencies in Xtest files. Could we not also do that, and is that not what you did in this PR itself?

Signed-off-by: Norman Meier <[email protected]>

Signed-off-by: Norman <[email protected]>

This makes the imports utils split imports by file kinds, allowing to make explicit decisions about what imports to use at the various callsites - Create `FileKind` enum to categorize gno files, with variants `PackageSource`, `Test`, `XTest` and `Filetest` - Create `GetFileKind` util to derive the `FileKind` from a file name and body - Create `ImportsMap` type that maps `FileKind`s to lists of imports. It has a single method `Merge` to select and merge various imports from multiple `FileKind`s - Modify the`packages.Imports` helper to return an `ImportsMap` instead of a `[]string` and adapt callsites by using`ImportMap.Merge` to preserve existing behavior This is something I need for #3304 and #2932 but to help reviews I made an atomic PR here instead --------- Signed-off-by: Norman Meier <[email protected]> Co-authored-by: Morgan <[email protected]>

…t-cycle

Signed-off-by: Norman <[email protected]>

This makes the imports utils split imports by file kinds, allowing to make explicit decisions about what imports to use at the various callsites - Create `FileKind` enum to categorize gno files, with variants `PackageSource`, `Test`, `XTest` and `Filetest` - Create `GetFileKind` util to derive the `FileKind` from a file name and body - Create `ImportsMap` type that maps `FileKind`s to lists of imports. It has a single method `Merge` to select and merge various imports from multiple `FileKind`s - Modify the`packages.Imports` helper to return an `ImportsMap` instead of a `[]string` and adapt callsites by using`ImportMap.Merge` to preserve existing behavior This is something I need for #3304 and #2932 but to help reviews I made an atomic PR here instead --------- Signed-off-by: Norman Meier <[email protected]> Co-authored-by: Morgan <[email protected]>

thehowl

Some comments on the non-trivial changes, good overall.

thehowl · 2025-01-15T19:19:32Z

examples/no_cycles_test.go

Could this not be a check in gno lint instead, which is already run on all of examples?

thehowl · 2025-01-15T19:20:33Z

gnovm/pkg/packages/imports_test.go


 import (
 	"os"
 	"path/filepath"
 	"testing"

 	"github.com/gnolang/gno/gnovm/pkg/gnolang"
+	. "github.com/gnolang/gno/gnovm/pkg/packages"


Don't think this is necessary; we should avoid dot imports unless we have weird cyclical imports in tests like the Go stdlib does.

thehowl · 2025-01-15T19:23:18Z

gnovm/stdlibs/strings/printtrie.gno

Should this not go in a printtrie_test.gno file, where it was originally? So it's not exported.

thehowl · 2025-01-15T19:26:31Z

gnovm/tests/stdlibs/testing/native_testing.go

Can you make this thread safe, since we want to do gnovm parallel testing?

n0izn0iz added 2 commits December 8, 2024 21:53

fix: prevent import cycles in stdlibs

b8bf790

Signed-off-by: Norman Meier <[email protected]>

fix: remove import cycles

fd7bbc4

Signed-off-by: Norman Meier <[email protected]>

github-actions bot assigned n0izn0iz Dec 8, 2024

github-actions bot added 🧾 package/realm Tag used for new Realms or Packages. 📦 🤖 gnovm Issues or PRs gnovm related labels Dec 8, 2024

n0izn0iz and others added 10 commits December 9, 2024 11:07

chore: improve cycle detector

9be6649

Signed-off-by: Norman Meier <[email protected]>

chore: inject match string to break import cycle

b7ba5a9

Signed-off-by: Norman Meier <[email protected]>

Merge branch 'master' into forbid-import-cycle

920c49b

chore: revert fuzz move

c45a741

Signed-off-by: Norman Meier <[email protected]>

chore: fuzztesting -> testing

360aa64

Signed-off-by: Norman Meier <[email protected]>

chore: regen

d7c3d10

Signed-off-by: Norman Meier <[email protected]>

Merge branch 'master' into forbid-import-cycle

c69fc43

fix: strconv tests

096f8cd

Signed-off-by: Norman Meier <[email protected]>

Merge branch 'master' into forbid-import-cycle

4517f0f

fix: math/rand, some math/bits, strings

ced3475

Signed-off-by: Norman Meier <[email protected]>

thehowl reviewed Dec 10, 2024

View reviewed changes

n0izn0iz and others added 10 commits December 10, 2024 14:30

fix: typo

eff15b0

Signed-off-by: Norman Meier <[email protected]>

chore: rename var

761ef83

Signed-off-by: Norman Meier <[email protected]>

fix: don't hide self imports

ee265f5

Signed-off-by: Norman Meier <[email protected]>

Merge branch 'master' into forbid-import-cycle

401d4e3

chore: add test case for self import

0a05f85

Signed-off-by: Norman Meier <[email protected]>

chore: make genstd consider test files

d9f4d3a

Signed-off-by: Norman Meier <[email protected]>

chore: make genstd consider test files

c6728c5

Signed-off-by: Norman Meier <[email protected]>

fix(genstd): allow native injections in all packages and regen

4374a39

Signed-off-by: Norman Meier <[email protected]>

tmp

284e56b

Signed-off-by: Norman Meier <[email protected]>

feat: categorize imports

6c5d83e

Signed-off-by: Norman Meier <[email protected]>

n0izn0iz mentioned this pull request Dec 11, 2024

feat(gnovm): categorize imports #3323

Merged

Merge branch 'master' into imports-map

d05c43b

Norman added 11 commits December 25, 2024 14:16

chore: revert genstd changes

65fdad5

Signed-off-by: Norman <[email protected]>

chore: revert genstd changes

f9e8c94

Signed-off-by: Norman <[email protected]>

chore: revert comments changes

d5ee04e

Signed-off-by: Norman <[email protected]>

chore: revert more comment changes

bf7ff6f

Signed-off-by: Norman <[email protected]>

chore: protect bits error values

8c7959a

Signed-off-by: Norman <[email protected]>

chore: revert multi_test changes

57a2d6b

Signed-off-by: Norman <[email protected]>

chore: revert unneeded math rand changes

5c95bd1

Signed-off-by: Norman <[email protected]>

chore: remove uneeded change

d98eea9

Signed-off-by: Norman <[email protected]>

chore: only implement matchString in testing context

288351b

Signed-off-by: Norman <[email protected]>

chore: improve comment

4692891

Signed-off-by: Norman <[email protected]>

chore: regen

6c06b7f

Signed-off-by: Norman <[email protected]>

n0izn0iz changed the title ~~fix: remove import cycles~~ fix: remove import cycles and self-imports Dec 25, 2024

n0izn0iz changed the title ~~fix: remove import cycles and self-imports~~ fix: remove import cycles Dec 25, 2024

Norman and others added 6 commits December 25, 2024 20:02

chore: add comment about overlay imports

e720c60

Signed-off-by: Norman <[email protected]>

Merge branch 'master' into imports-map

0490b9e

chore: Merge remote-tracking branch 'origin/master' into imports-map

2945568

Merge branch 'master' into imports-map

31983c8

chore: Merge branch 'imports-map' into forbid-import-cycle

44444c0

Merge branch 'master' into forbid-import-cycle

13a5e72

chore: Merge remote-tracking branch 'origin/master' into forbid-impor…

43c95d4

…t-cycle

n0izn0iz marked this pull request as ready for review January 9, 2025 21:13

chore: remove merge artifact

6e1a134

Signed-off-by: Norman <[email protected]>

n0izn0iz requested a review from thehowl January 9, 2025 21:18

n0izn0iz and others added 2 commits January 10, 2025 01:02

Merge branch 'master' into forbid-import-cycle

10ea5f4

chore: trigger CI

b0c1f9f

Signed-off-by: Norman <[email protected]>

Kouteki added the in focus Core team is prioritizing this work label Jan 13, 2025

Kouteki mentioned this pull request Jan 15, 2025

Minutes: Core Staff Weekly Syncs [every Monday] gnolang/meetings#36

Open

thehowl reviewed Jan 15, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: remove import cycles #3304

fix: remove import cycles #3304

n0izn0iz commented Dec 8, 2024 •

edited

Loading

Gno2D2 commented Dec 8, 2024 •

edited

Loading

✅ Automated Checks (for Contributors):

☑️ Contributor Actions:

☑️ Reviewer Actions:

📚 Resources:

If

Then

If

Can be checked by

codecov bot commented Dec 8, 2024 •

edited

Loading

thehowl Dec 10, 2024

n0izn0iz Dec 10, 2024 •

edited

Loading

n0izn0iz Dec 23, 2024 •

edited

Loading

n0izn0iz Dec 25, 2024 •

edited

Loading

thehowl Jan 15, 2025

n0izn0iz Jan 15, 2025

thehowl Jan 16, 2025

thehowl left a comment

thehowl Jan 15, 2025

thehowl Jan 15, 2025

thehowl Jan 15, 2025

thehowl Jan 15, 2025

fix: remove import cycles #3304

Are you sure you want to change the base?

fix: remove import cycles #3304

Conversation

n0izn0iz commented Dec 8, 2024 • edited Loading

Gno2D2 commented Dec 8, 2024 • edited Loading

🛠 PR Checks Summary

Manual Checks (for Reviewers):

✅ Automated Checks (for Contributors):

☑️ Contributor Actions:

☑️ Reviewer Actions:

📚 Resources:

If

Then

If

Can be checked by

codecov bot commented Dec 8, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

n0izn0iz Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

n0izn0iz Dec 23, 2024 • edited Loading

Choose a reason for hiding this comment

n0izn0iz Dec 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thehowl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

n0izn0iz commented Dec 8, 2024 •

edited

Loading

Gno2D2 commented Dec 8, 2024 •

edited

Loading

codecov bot commented Dec 8, 2024 •

edited

Loading

n0izn0iz Dec 10, 2024 •

edited

Loading

n0izn0iz Dec 23, 2024 •

edited

Loading

n0izn0iz Dec 25, 2024 •

edited

Loading