-
Notifications
You must be signed in to change notification settings - Fork 388
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: fixing some nondeterministic tests #132
Conversation
I also have experience of flappy tests as soon as we're doing protocol using p2p; it's a real nightmare what we did:
|
|
$ time go test -v ./pkgs/bft/blockchain
[…]
TestBadBlockStopsPeer
[…]
panic: Failed to process committed block (5:5CE69BF4527330E3B6055F2AA9936FA1C03CEADEF8F057E90A93BB9AD4269906): Wrong Block.Header.LastBlockID. Expected 3B14C912CF5C850CA146D747D263159459E320907A2C6B70ECF3B8E863538271:1:F3F7E09BF331, got 32A5F608D8548CDBBF4522F7AF25E6FC82FD9963CB00F2F1777EB99F2920111B:1:A0A9F448A38A
goroutine 44473 [running]:
github.com/gnolang/gno/pkgs/bft/blockchain.(*BlockchainReactor).poolRoutine(0xc00bed0f00)
/home/moul/go/src/moul.io/gno4/pkgs/bft/blockchain/reactor.go:343 +0x1425
created by github.com/gnolang/gno/pkgs/bft/blockchain.(*BlockchainReactor).OnStart
/home/moul/go/src/moul.io/gno4/pkgs/bft/blockchain/reactor.go:117 +0x87
FAIL github.com/gnolang/gno/pkgs/bft/blockchain 201.628s
FAIL |
$ time go test -v ./pkgs/bft/consensus/ -count 2 -run TestHandshakeReplayNone -test.timeout=60m
[…]
.level 1 .msg Executed block height 6 validTxs 0 invalidTxs 0
.level 2 .msg CONSENSUS FAILURE!!! [err leveldb: closed stack goroutine 200 [running]:
runtime/debug.Stack()
/home/moul/.gvm/gos/go1.18/src/runtime/debug/stack.go:24 +0x65
github.com/gnolang/gno/pkgs/bft/consensus.(*ConsensusState).receiveRoutine.func2()
/home/moul/go/src/moul.io/gno4/pkgs/bft/consensus/state.go:609 +0x4c
panic({0x94ce60, 0xc00020b4d0})
/home/moul/.gvm/gos/go1.18/src/runtime/panic.go:838 +0x207
github.com/gnolang/gno/pkgs/db.(*GoLevelDB).Set(0x9a1d60?, {0xded478?, 0x199?, 0x7f04e24c4a68?}, {0xc002fdb0b0?, 0xc000780400?, 0x199?})
/home/moul/go/src/moul.io/gno4/pkgs/db/go_level_db.go:67 +0x7f
github.com/gnolang/gno/pkgs/bft/abci/example/kvstore.saveState({{0xabb6b0, 0xc00310a538}, 0x0, 0x6, {0xc002fe3918, 0x8, 0x8}})
/home/moul/go/src/moul.io/gno4/pkgs/bft/abci/example/kvstore/kvstore.go:45 +0xe7
github.com/gnolang/gno/pkgs/bft/abci/example/kvstore.(*KVStoreApplication).Commit(0xc0007c0080)
/home/moul/go/src/moul.io/gno4/pkgs/bft/abci/example/kvstore/kvstore.go:106 +0x12a
github.com/gnolang/gno/pkgs/bft/abci/example/kvstore.(*PersistentKVStoreApplication).Commit(0xc002fee120?)
/home/moul/go/src/moul.io/gno4/pkgs/bft/abci/example/kvstore/persistent_kvstore.go:85 +0x5b
github.com/gnolang/gno/pkgs/bft/abci/client.(*localClient).CommitSync(0xe66440?)
/home/moul/go/src/moul.io/gno4/pkgs/bft/abci/client/local_client.go:188 +0xd4
github.com/gnolang/gno/pkgs/bft/proxy.(*appConnConsensus).CommitSync(0x203000?)
/home/moul/go/src/moul.io/gno4/pkgs/bft/proxy/app_conn.go:81 +0x82
github.com/gnolang/gno/pkgs/bft/state.(*BlockExecutor).Commit(_, {{0x9f45b7, 0xb}, {0x9f45b7, 0xb}, {0x9f1f20, 0x6}, {0xc000815a70, 0xf}, 0x6, ...}, ...)
/home/moul/go/src/moul.io/gno4/pkgs/bft/state/execution.go:176 +0x269
github.com/gnolang/gno/pkgs/bft/state.(*BlockExecutor).ApplyBlock(_, {{0x9f45b7, 0xb}, {0x9f45b7, 0xb}, {0x9f1f20, 0x6}, {0xc000815a70, 0xf}, 0x6, ...}, ...)
/home/moul/go/src/moul.io/gno4/pkgs/bft/state/execution.go:133 +0x52e
github.com/gnolang/gno/pkgs/bft/consensus.(*ConsensusState).finalizeCommit(0xc0000dd500, 0x6)
/home/moul/go/src/moul.io/gno4/pkgs/bft/consensus/state.go:1353 +0x97e
github.com/gnolang/gno/pkgs/bft/consensus.(*ConsensusState).tryFinalizeCommit(0xc0000dd500, 0x6)
/home/moul/go/src/moul.io/gno4/pkgs/bft/consensus/state.go:1281 +0x305
github.com/gnolang/gno/pkgs/bft/consensus.(*ConsensusState).enterCommit.func1()
/home/moul/go/src/moul.io/gno4/pkgs/bft/consensus/state.go:1227 +0x8e
github.com/gnolang/gno/pkgs/bft/consensus.(*ConsensusState).enterCommit(0xc0000dd500, 0x6, 0x0)
/home/moul/go/src/moul.io/gno4/pkgs/bft/consensus/state.go:1258 +0xab0
github.com/gnolang/gno/pkgs/bft/consensus.(*ConsensusState).addVote(0xc0000dd500, 0xc002600f00, {0x0, 0x0})
/home/moul/go/src/moul.io/gno4/pkgs/bft/consensus/state.go:1646 +0x8cf
github.com/gnolang/gno/pkgs/bft/consensus.(*ConsensusState).tryAddVote(0xc0000dd500, 0x475886?, {0x0?, 0xc00260dbf0?})
/home/moul/go/src/moul.io/gno4/pkgs/bft/consensus/state.go:1489 +0x27
github.com/gnolang/gno/pkgs/bft/consensus.(*ConsensusState).handleMsg(0xc0000dd500, {{0xab47c0?, 0xc0000af3a8?}, {0x0?, 0xeda159371?}})
/home/moul/go/src/moul.io/gno4/pkgs/bft/consensus/state.go:695 +0x388
github.com/gnolang/gno/pkgs/bft/consensus.(*ConsensusState).receiveRoutine(0xc0000dd500, 0x0)
/home/moul/go/src/moul.io/gno4/pkgs/bft/consensus/state.go:654 +0x465
created by github.com/gnolang/gno/pkgs/bft/consensus.(*ConsensusState).OnStart
/home/moul/go/src/moul.io/gno4/pkgs/bft/consensus/state.go:346 +0x4cf
[wal_generator wal_generator]]
.level 1 .msg Starting baseWAL impl baseWAL
.level 1 .msg Starting Group impl Group
.level 1 .msg Searching for height height 1 min 0 max 0
.level 1 .msg Found height 1 index 0
.level 1 .msg Executed block height 1 validTxs 0 invalidTxs 0
.level 1 .msg Committed state height 1 txs 0 appHash 0000000000000000
.level 1 .msg Executed block height 2 validTxs 0 invalidTxs 0
.level 1 .msg Committed state height 2 txs 0 appHash 0000000000000000
.level 1 .msg Executed block height 3 validTxs 0 invalidTxs 0
.level 1 .msg Committed state height 3 txs 0 appHash 0000000000000000
.level 1 .msg Executed block height 4 validTxs 0 invalidTxs 0
.level 1 .msg Committed state height 4 txs 0 appHash 0000000000000000
.level 1 .msg Executed block height 5 validTxs 0 invalidTxs 0
.level 1 .msg Committed state height 5 txs 0 appHash 0000000000000000
.level 1 .msg Executed block height 6 validTxs 0 invalidTxs 0
.level 1 .msg Committed state height 6 txs 0 appHash 0000000000000000
replay_test.go:679: Error on abci handshake: error on replay: App block height (12) is higher than core (6)
.level 1 .msg Stopping baseWAL impl baseWAL
.level 1 .msg Stopping Group impl Group
--- FAIL: TestHandshakeReplayNone (0.24s)
FAIL
FAIL github.com/gnolang/gno/pkgs/bft/consensus 1.377s
FAIL Seems to be stable when called with Full logs here -> https://gist.github.com/moul/d15e000f8be82f8fc45325c3a8df1a19 |
From https://github.com/moul/gno/runs/6054431723?check_suite_focus=true
The timeout based tests are definitely a problem.
The privval socket system maybe requires a redo, or merging fixes from mainline tendermint at some point.
The p2p/transport system is also pretty confusing.
This PR can stay open while we find more issues.