Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: remove import cycles #3304

Open
wants to merge 88 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
b8bf790
fix: prevent import cycles in stdlibs
n0izn0iz Dec 8, 2024
fd7bbc4
fix: remove import cycles
n0izn0iz Dec 8, 2024
9be6649
chore: improve cycle detector
n0izn0iz Dec 9, 2024
b7ba5a9
chore: inject match string to break import cycle
n0izn0iz Dec 9, 2024
920c49b
Merge branch 'master' into forbid-import-cycle
n0izn0iz Dec 9, 2024
c45a741
chore: revert fuzz move
n0izn0iz Dec 9, 2024
360aa64
chore: fuzztesting -> testing
n0izn0iz Dec 9, 2024
d7c3d10
chore: regen
n0izn0iz Dec 9, 2024
c69fc43
Merge branch 'master' into forbid-import-cycle
n0izn0iz Dec 9, 2024
096f8cd
fix: strconv tests
n0izn0iz Dec 9, 2024
4517f0f
Merge branch 'master' into forbid-import-cycle
n0izn0iz Dec 9, 2024
ced3475
fix: math/rand, some math/bits, strings
n0izn0iz Dec 10, 2024
eff15b0
fix: typo
n0izn0iz Dec 10, 2024
761ef83
chore: rename var
n0izn0iz Dec 10, 2024
ee265f5
fix: don't hide self imports
n0izn0iz Dec 10, 2024
401d4e3
Merge branch 'master' into forbid-import-cycle
n0izn0iz Dec 10, 2024
0a05f85
chore: add test case for self import
n0izn0iz Dec 10, 2024
d9f4d3a
chore: make genstd consider test files
n0izn0iz Dec 10, 2024
c6728c5
chore: make genstd consider test files
n0izn0iz Dec 10, 2024
4374a39
fix(genstd): allow native injections in all packages and regen
n0izn0iz Dec 10, 2024
284e56b
tmp
n0izn0iz Dec 11, 2024
6c5d83e
feat: categorize imports
n0izn0iz Dec 11, 2024
d05c43b
Merge branch 'master' into imports-map
n0izn0iz Dec 11, 2024
702e3d9
Merge branch 'master' into imports-map
n0izn0iz Dec 11, 2024
1d17b9f
Merge branch 'master' into imports-map
n0izn0iz Dec 13, 2024
c4d82dc
Merge branch 'master' into imports-map
n0izn0iz Dec 17, 2024
27bb085
Merge branch 'master' into imports-map
n0izn0iz Dec 18, 2024
3e04787
chore: Merge remote-tracking branch 'origin/master' into imports-map
n0izn0iz Dec 19, 2024
a0dd0d8
chore: move code for easier review
n0izn0iz Dec 19, 2024
d2d043e
chore: pass fset to GetFileKind
n0izn0iz Dec 19, 2024
efda22d
fix: usage of packages.Imports
n0izn0iz Dec 19, 2024
fcd66a5
chore: don't expose SortImports
n0izn0iz Dec 19, 2024
37d96ce
chore: error msg with loc
n0izn0iz Dec 19, 2024
e591002
Merge branch 'master' into imports-map
n0izn0iz Dec 19, 2024
315b37f
Merge branch 'master' into imports-map
n0izn0iz Dec 19, 2024
cd91edb
Merge branch 'master' into imports-map
n0izn0iz Dec 20, 2024
c0998d7
chore: nit
n0izn0iz Dec 21, 2024
41ddaa6
chore: revert merge fail
n0izn0iz Dec 21, 2024
4cfda1e
chore: FileKindCompiled -> FileKindPackageSource
n0izn0iz Dec 21, 2024
2100a57
chore: FileKindXtest -> FilKindXTest
n0izn0iz Dec 21, 2024
0be8c69
chore: add FileKind doc
n0izn0iz Dec 21, 2024
1fcf4e3
Merge branch 'master' into imports-map
n0izn0iz Dec 21, 2024
3973f84
chore: Merge branch 'imports-map' into forbid-import-cycle
n0izn0iz Dec 22, 2024
5c7e31b
fix: nocycles binary after upgrade
n0izn0iz Dec 23, 2024
479145b
Merge branch 'master' into forbid-import-cycle
n0izn0iz Dec 23, 2024
35aca6d
chore: use new util to split imports
n0izn0iz Dec 23, 2024
22f1589
feat: correctly check cycles
n0izn0iz Dec 23, 2024
5c4208f
chore: improve explainers
n0izn0iz Dec 23, 2024
64beee7
feat: embed cycle test in actual test
n0izn0iz Dec 23, 2024
5a8beef
chore: don't panic in test
n0izn0iz Dec 23, 2024
a3fc750
Merge branch 'master' into forbid-import-cycle
n0izn0iz Dec 23, 2024
39def3e
chore: improve comment
n0izn0iz Dec 23, 2024
1f7d1fc
chore: improve doc
n0izn0iz Dec 23, 2024
0ebf4f5
Merge branch 'master' into imports-map
n0izn0iz Dec 25, 2024
7c11caa
chore: revert pkg change
Dec 25, 2024
f7c34e0
chore: refacto test
Dec 25, 2024
f01b564
chore: refacto test 2
Dec 25, 2024
8ae5f73
chore: remove dev artifact
Dec 25, 2024
65fdad5
chore: revert genstd changes
Dec 25, 2024
f9e8c94
chore: revert genstd changes
Dec 25, 2024
d5ee04e
chore: revert comments changes
Dec 25, 2024
bf7ff6f
chore: revert more comment changes
Dec 25, 2024
8c7959a
chore: protect bits error values
Dec 25, 2024
57a2d6b
chore: revert multi_test changes
Dec 25, 2024
5c95bd1
chore: revert unneeded math rand changes
Dec 25, 2024
d98eea9
chore: remove uneeded change
Dec 25, 2024
288351b
chore: only implement matchString in testing context
Dec 25, 2024
4692891
chore: improve comment
Dec 25, 2024
6c06b7f
chore: regen
Dec 25, 2024
e720c60
chore: add comment about overlay imports
Dec 25, 2024
0490b9e
Merge branch 'master' into imports-map
n0izn0iz Jan 3, 2025
2945568
chore: Merge remote-tracking branch 'origin/master' into imports-map
n0izn0iz Jan 6, 2025
31983c8
Merge branch 'master' into imports-map
n0izn0iz Jan 7, 2025
44444c0
chore: Merge branch 'imports-map' into forbid-import-cycle
n0izn0iz Jan 7, 2025
13a5e72
Merge branch 'master' into forbid-import-cycle
n0izn0iz Jan 7, 2025
43c95d4
chore: Merge remote-tracking branch 'origin/master' into forbid-impor…
n0izn0iz Jan 9, 2025
6e1a134
chore: remove merge artifact
n0izn0iz Jan 9, 2025
10ea5f4
Merge branch 'master' into forbid-import-cycle
n0izn0iz Jan 10, 2025
b0c1f9f
chore: trigger CI
n0izn0iz Jan 10, 2025
d6fa81c
chore: Merge remote-tracking branch 'origin/master' into forbid-impor…
n0izn0iz Jan 20, 2025
8728744
chore: don't use dot import
n0izn0iz Jan 20, 2025
c87d4dd
chore: unexport printtrie
n0izn0iz Jan 20, 2025
348f34b
chore: make native matchString thread-safe
n0izn0iz Jan 20, 2025
04818d8
Merge branch 'master' into forbid-import-cycle
n0izn0iz Jan 20, 2025
784d508
chore: add todo about moving the cycle detector in gno lint
n0izn0iz Jan 20, 2025
ebc87af
chore: Merge remote-tracking branch 'origin/master' into forbid-impor…
n0izn0iz Jan 20, 2025
ac3a38d
Merge branch 'master' into forbid-import-cycle
thehowl Jan 21, 2025
cbeb758
Merge branch 'master' into forbid-import-cycle
n0izn0iz Jan 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 146 additions & 0 deletions contribs/nocycles/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
package main

import (
"fmt"
"io/fs"
"os"
"path/filepath"
"strings"

"github.com/gnolang/gno/gnovm"
"github.com/gnolang/gno/gnovm/pkg/gnoenv"
"github.com/gnolang/gno/gnovm/pkg/gnolang"
"github.com/gnolang/gno/gnovm/pkg/gnomod"
"github.com/gnolang/gno/gnovm/pkg/packages"
)

func main() {
// set to true to dump the final pkg list
verbose := true

// find stdlibs
libs := []string{}
gnoRoot := gnoenv.RootDir()
stdlibsDir := filepath.Join(gnoRoot, "gnovm", "stdlibs")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for stdlibs, instead, i think you could change misc/genstd (note: this binary as well, if it were to stay, would also have to be misc/), by having its import order generator also consider test files

Copy link
Contributor Author

@n0izn0iz n0izn0iz Dec 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, when I cherry-pick c6728c5 on master, I get:

go run github.com/gnolang/gno/misc/genstd
panic: cyclical package initialization on "bytes" (bytes -> io -> bytes)

I like having code that is dedicated to detecting any cycles in stdlibs and examples though, I'll move the nocycles code into a test somewhere I think when I get to the polishing step on this PR

Copy link
Contributor Author

@n0izn0iz n0izn0iz Dec 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved the test to examples/no_cycles_test.go

Copy link
Contributor Author

@n0izn0iz n0izn0iz Dec 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

regarding genstd changes:

actually finding a single global order while including all test files is not possible, consider the following imports (which are allowed in go):

  • foopkg/foo_test.go imports barpkg
  • barpkg/bar_test.go imports foopkg

examining each packages tests deps will yield a different order:

  • foopkg tests:
    • barpkg
    • foopkg
  • barpkg tests:
    • foopkg
    • barpkg

we could fill a map[string][]string to get the testing order for each lib but I'm not sure this is really helpful, do you want me to add this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this PR not meant to find that order?

The order should exist when excluding the Xtest files, no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, it only exists when you include solely the PackageSource files, the example above is with normal test files

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The go stdlib makes it possible to find an order by putting all deps which would lead to circular dependencies in Xtest files. Could we not also do that, and is that not what you did in this PR itself?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and is that not what you did in this PR itself?

No, this PR aims to remove cycles when you follow the normal golang rules. Mainly the source of cycles were due to the dependencies of the testing package which must only have XTest files otherwise they have a cycle with testing since they must import it.

I'm not sure the golang stdlibs have a global order if you consider all Test sources, would need to check

There is tests in a few libraries (in gno and go) that test the lib internals and can't be direcly converted to XTest

fs.WalkDir(os.DirFS(stdlibsDir), ".", func(path string, d fs.DirEntry, err error) error {
if !d.IsDir() {
return nil
}
if path == "." {
return nil
}
libs = append(libs, path)
return nil
})

// read stdlibs
pl := gnomod.PkgList{}
for _, lib := range libs {
memPkg := gnolang.MustReadMemPackage(filepath.Join(stdlibsDir, lib), lib)
pkg, xpkg, err := splitMemPackage(memPkg)
if err != nil {
panic(fmt.Errorf("split %q: %w", lib, err))
}
{
imports, err := packages.Imports(pkg)
if err != nil {
panic(fmt.Errorf("read %q: %w", lib, err))
}
pl = append(pl, gnomod.Pkg{
Dir: "",
Name: lib,
Imports: imports,
})
}
if !xpkg.IsEmpty() {
imports, err := packages.Imports(xpkg)
if err != nil {
panic(fmt.Errorf("read %q: %w", lib, err))
}
pl = append(pl, gnomod.Pkg{
Dir: "",
Name: "_test_" + lib,
Imports: imports,
})
}
}

// load all examples
examples, err := gnomod.ListPkgs(filepath.Join(gnoRoot, "examples"))
if err != nil {
panic(fmt.Errorf("load examples: %w", err))
}
for _, example := range examples {
if example.Draft {
continue
}
pkgPath := example.Name
memPkg := gnolang.MustReadMemPackage(example.Dir, example.Name)
pkg, xpkg, err := splitMemPackage(memPkg)
if err != nil {
panic(fmt.Errorf("split %q: %w", pkgPath, err))
}
{
imports, err := packages.Imports(pkg)
if err != nil {
panic(fmt.Errorf("read %q: %w", pkgPath, err))
}
pl = append(pl, gnomod.Pkg{
Dir: example.Dir,
Name: pkgPath,
Imports: imports,
})
}
if !xpkg.IsEmpty() {
imports, err := packages.Imports(xpkg)
if err != nil {
panic(fmt.Errorf("read %q: %w", pkgPath, err))
}
pl = append(pl, gnomod.Pkg{
Dir: example.Dir,
Name: "_test_" + pkgPath,
Imports: imports,
})
}
}

// detect import cycles
if _, err := pl.Sort(); err != nil {
panic(err)
}

if verbose {
for _, p := range pl {
fmt.Println(p.Name)
}
}
}

func splitMemPackage(pkg *gnovm.MemPackage) (*gnovm.MemPackage, *gnovm.MemPackage, error) {
corePkg := gnovm.MemPackage{
Name: pkg.Name,
Path: pkg.Path,
}
xtestPkg := gnovm.MemPackage{
Name: pkg.Name + "_test",
Path: pkg.Path,
}

for _, file := range pkg.Files {
if !strings.HasSuffix(file.Name, "_test.gno") {
corePkg.Files = append(corePkg.Files, file)
continue
}
pkgName, err := packages.FilePackageName(file.Name, file.Body)
if err != nil {
return nil, nil, fmt.Errorf("get package name in file %q: %w", file.Name, err)
}
if !strings.HasSuffix(pkgName, "_test") {
corePkg.Files = append(corePkg.Files, file)
continue
}
xtestPkg.Files = append(xtestPkg.Files, file)
}

return &corePkg, &xtestPkg, nil
}
2 changes: 1 addition & 1 deletion examples/gno.land/p/moul/md/md_test.gno
thehowl marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package md
package md_test

import (
"testing"
Expand Down
2 changes: 1 addition & 1 deletion examples/gno.land/r/gnoland/faucet/faucet_test.gno
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package faucet
package faucet_test

import (
"std"
Expand Down
21 changes: 13 additions & 8 deletions gnovm/pkg/gnomod/pkg.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,39 +39,45 @@ type (
// sortPkgs sorts the given packages by their dependencies.
func (pl PkgList) Sort() (SortedPkgList, error) {
visited := make(map[string]bool)
onStack := make(map[string]bool)
stack := []string{}
sortedPkgs := make([]Pkg, 0, len(pl))

// Visit all packages
for _, p := range pl {
if err := visitPackage(p, pl, visited, onStack, &sortedPkgs); err != nil {
if err := visitPackage(p, pl, visited, stack, &sortedPkgs); err != nil {
return nil, err
}
}

return sortedPkgs, nil
}

var injectedTestingLib = []string{"encoding/json", "fmt", "os", "internal/os_test"}

// visitNode visits a package's and its dependencies dependencies and adds them to the sorted list.
func visitPackage(pkg Pkg, pkgs []Pkg, visited, onStack map[string]bool, sortedPkgs *[]Pkg) error {
if onStack[pkg.Name] {
return fmt.Errorf("cycle detected: %s", pkg.Name)
func visitPackage(pkg Pkg, pkgs []Pkg, visited map[string]bool, stack []string, sortedPkgs *[]Pkg) error {
if slices.Contains(stack, pkg.Name) {
return fmt.Errorf("cycle detected: %s -> %s", strings.Join(stack, " -> "), pkg.Name)
}
if visited[pkg.Name] {
return nil
}

visited[pkg.Name] = true
onStack[pkg.Name] = true
stack = append(stack, pkg.Name)

// Visit package's dependencies
for _, imp := range pkg.Imports {
if slices.Contains(injectedTestingLib, imp) {
continue
}

found := false
for _, p := range pkgs {
if p.Name != imp {
continue
}
if err := visitPackage(p, pkgs, visited, onStack, sortedPkgs); err != nil {
if err := visitPackage(p, pkgs, visited, stack, sortedPkgs); err != nil {
return err
}
found = true
Expand All @@ -82,7 +88,6 @@ func visitPackage(pkg Pkg, pkgs []Pkg, visited, onStack map[string]bool, sortedP
}
}

onStack[pkg.Name] = false
*sortedPkgs = append(*sortedPkgs, pkg)
return nil
}
Expand Down
9 changes: 9 additions & 0 deletions gnovm/pkg/packages/imports.go
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,12 @@ func FileImports(filename string, src string) ([]*FileImport, *token.FileSet, er
}
return res, fs, nil
}

func FilePackageName(filename string, src string) (string, error) {
fs := token.NewFileSet()
f, err := parser.ParseFile(fs, filename, src, parser.PackageClauseOnly)
if err != nil {
return "", err
}
return f.Name.Name, nil
}
2 changes: 1 addition & 1 deletion gnovm/stdlibs/crypto/ed25519/ed25519_test.gno
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package ed25519
package ed25519_test

import (
"crypto/ed25519"
Expand Down
2 changes: 1 addition & 1 deletion gnovm/stdlibs/crypto/sha256/sha256_test.gno
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package sha256
package sha256_test

import (
"crypto/sha256"
Expand Down
2 changes: 1 addition & 1 deletion gnovm/stdlibs/encoding/binary/binary_test.gno
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package binary
package binary_test

import (
"bytes"
Expand Down
38 changes: 38 additions & 0 deletions gnovm/stdlibs/generated.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion gnovm/stdlibs/hash/marshal_test.gno
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
// BinaryMarshaler, BinaryUnmarshaler,
// and lock in the current representations.

package hash
package hash_test

import (
"bytes"
Expand Down
10 changes: 10 additions & 0 deletions gnovm/stdlibs/io/export_test.gno
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
// Copyright 2020 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

package io

// exported for test
var ErrInvalidWrite = errInvalidWrite
var ErrWhence = errWhence
var ErrOffset = errOffset
Loading
Loading