[pull] master from stellar:master #19
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![pull[bot]](https://avatars.githubusercontent.com/in/12910?s=60&v=4){kind=small}
# Checklist - [x] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [x] Rebased on top of master (no merge commits) - [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [x] Compiles - [x] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
# Description Update Core to protocol 22 and ungate the tests that rely on the new protocol. # Checklist - [ ] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [ ] Rebased on top of master (no merge commits) - [ ] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [ ] Compiles - [ ] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
This modifies the approach we use to linking multiple sorobans. The new approach builds each soroban separately into its own `rlib` using a `--locked` cargo build, followed by manually providing them as `--extern` definitions to the top-level rust build of libstellar_core.rlib. It is an approach to solving problems of cargo solving/merging/advancing dependency versions when doing soroban multi-versioning (a.k.a. #4278). The approach here is deeply indebted to @leighmcculloch -- he both had the initial idea and overcame almost every obstacle I encountered along the way. I am just the automake-wrestling keyboard monkey in this PR. Advantages: - We get the soroban lockfiles that were tested upstream, no more no less. - We get to use git submodules to just point directly to the soroban source trees. - Cargo treats all the builds separately: each soroban, then stellar-core itself. Never merges deps of any of them. Disadvantages: - Submodules make a lot of people sad - The necessary automake code _definitely_ makes me sad - It breaks vscode or any other IDE trying to edit contract.rs (which, granted, is only 500 lines of code) ## Dep-tree checking There's some existing machinery in stellar-core that bakes-in the Cargo.lock file and then compares the dep-tree of each soroban host in it to fixed (manually maintained) dep-tree files we generate with the `cargo lock tree` cargo-extension. This machinery no longer works with this new scheme: - There is no single lockfile anymore - The lockfiles in the submodules contain lots of additional deps and churn (dev-deps especially) So instead I've decided to redo this task using a _slightly_ weaker tool: `cargo tree`, which is built-in to cargo (not `cargo lock tree`). This loses some precision (`cargo tree` only outputs package version numbers, not checksums) but it allows us to specify the features, and exclude the dev-deps, of each submodule. Along the way I've changed it from a dynamic check to a static one: the build just won't succeed if the expected deptrees (checked-in to the stellar-core repo) don't match the actual ones (extracted at build time from the submodules). The resulting errors look like this: ``` --- rust/src/dep-trees/p21-expect.txt 2024-09-07 20:38:22.056593002 -0700 +++ ../src/rust/src/dep-trees/p21-actual.txt 2024-09-07 20:38:11.852700915 -0700 @@ -5,7 +5,7 @@ │ ├── curve25519-dalek-derive v0.1.0 (proc-macro) │ │ ├── proc-macro2 v1.0.69 │ │ │ └── unicode-ident v1.0.9 -│ │ ├── quote v1.0.32 +│ │ ├── quote v1.0.33 │ │ │ └── proc-macro2 v1.0.69 (*) │ │ └── syn v2.0.39 │ │ ├── proc-macro2 v1.0.69 (*) dep trees differ, please update p21-expect.txt or roll back submodule ``` I think this is a generally superior developer experience for us, despite the minor loss in precision around dep identities. In practice I think the package version numbers are precise enough. ## Dep-tree differences If you take a look at the dep tree being checked in with this change for the p21 host and compare to the dep tree baked into master's current lockfile for the `[email protected]` package, you will see some slight differences: specifically you'll see that this PR _downgrades_ `smallvec 1.13.2 -> 1.10.0`, `libm 0.2.8 -> 0.2.7` and `wasmparser-nostd 0.100.2 -> 0.100.1`, and the removal of `ahash`. These downgrades are actually a _revert_ of changes that happened recently in e967b18 where I generalized support for multiple versions of soroban and simultaneously brought the p22 env into core: at that point I was forced (by cargo's aggressive version unification) to allow those _upgrades_ to the dep-tree of the p21 host, even though I kinda didn't want to. I accepted them at the time as "probably unobservable and worth the bet" but, in fact, the presence of such unwanted upgrades was one of the motivating factors for _this PR_, that reverts them by downgrading them. Luckily we have not _released_ anything with those unwanted-upgrades yet, so reverting and downgrading them to the exact versions that (a) shipped in p21 and (b) are baked into the lockfile of the env git repo at the point in history when p21 was released is the right thing to do here. But it was good to check!
This is an attempt (very terrible and unfun) to get the new `cargo tree` dep-tree commands to be more stable across platforms and invocations. Unfortunately it makes them gigantic.
Adds instructions on installing `clang-format-12` on Ubuntu 24.04. # Checklist - [ ] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [ ] Rebased on top of master (no merge commits) - [ ] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [ ] Compiles - [ ] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
# Description 1. Picks up the latest env version. 2. Bumps the xdr. 3. Adds new cost types for V22. I just finished up what Jay had started, so I still need to review the cost type changes. <!--- Describe what this pull request does, which issue it's resolving (usually applicable for code changes). ---> # Checklist - [ ] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [ ] Rebased on top of master (no merge commits) - [ ] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [ ] Compiles - [ ] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
I tried to use a clever feature of GNU make (the `&:` rule type) and it does not seem to cause the sequencing it implies in the manual. We need some sequencing to avoid running multiple cargos in parallel since they in turn run rustup and rustup can't handle concurrent execution. So let us try this with just "more stamp files", the old fashioned way.
This also brings in p22 updates for k256 and dalek libraries.
# Description Fix the upgrade validation to handle settings upgrades for the previous protocol. Also rewrote the condition so our test cases will catch this issue next time settings are added. <!--- Describe what this pull request does, which issue it's resolving (usually applicable for code changes). ---> # Checklist - [ ] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [ ] Rebased on top of master (no merge commits) - [ ] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [ ] Compiles - [ ] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
…rtain methods static
Resolves #4317 Concludes #4128 The implementation of this proposal requires massive changes to the stellar-core codebase, and touches almost every subsystem. There are some paradigm shifts in how the program executes, that I will discuss below for posterity. The same ideas are reflected in code comments as well, as it’ll be important for code maintenance and extensibility ## Database access Currently, only Postgres DB backend is supported, as it required minimal changes to how DB queries are structured (Postgres provides a fairly nice concurrency model). SQLite concurrency support is a lot more rudimentary, with only a single writer allowed, and the whole database is locked during writing. This necessitates further changes in core (such as splitting the database into two). Given that most network infrastructure is on Postgres right now, SQLite support can be added later. ### Reduced responsibilities of SQL SQL tables have been trimmed as much as possible to avoid conflicts, essentially we only store persistent state such as the latest LCL and SCP history, as well as legacy OFFER table. ## Asynchronous externalize flow There are three important subsystems in core that are in charge of tracking consensus, externalizing and applying ledgers, and advancing the state machine to catchup or synced state: - Herder: receives SCP messages, forwards them to SCP, decides if a ledger is externalized, triggers voting for the next ledger - LedgerManager: implements closing of a ledger, sets catchup vs synced state, advances and persists last closed ledger. - CatchupManager: Keep track of any externalized ledgers that are not LCL+1. That is, keep track of future externalizing ledgers, attempt applying them to keep core in sync, and trigger catchup if needed. Prior to this change, the externalize flow had two different flows: - If core received LCL+1, it would immediately apply it. Which means the flow externalize → closeLedger → set “synced” state happened in one synchronous function. After application, core triggers the next ledger, usually asynchronously, as it needs to wait to meet the 5s ledger requirement. - If core received ledger LCL+2..LCL+N it would asynchronously buffer it, and continue buffering new ledgers. If core can’t close the gap and apply everything sequentially, it would go into catchup flow. With the new changes, the triggering ledger close flow moved to CatchupManager completely. Essentially, CatchupManager::processLedger became a centralized place to decide whether to apply a ledger, or trigger catchup. Because ledger close happens in the background, the transition between externalize and “closeLedger→set synced” becomes asynchronous. ## Concurrent ledger close List of core items that moved to the background followed by explanation why it is safe to do so: ### Emitting meta Ledger application is the only process that touches the meta pipe, no conflicts with other subsystems ### Writing checkpoint files Only the background thread writes in-progress checkpoint files. Main thread deals exclusively with “complete” checkpoints, which after completion must not be touched by any subsystem except publishing. ### Updating ledger state The rest of the system operates strictly on read-only BucketList snapshots, and is unaffected by changing state. Note: there are some calls to LedgerTxn in the codebase still, but those only appear on startup during setup (when node is not operational) or in offline commands. ### Incrementing current LCL Because ledger close moved to the background, guarantees about ledger state and its staleness are now different. Previously, ledger state queried by subsystems outside of apply was always up-to-date. With this change, it is possible the snapshot used by main thread may become slightly stale (if background just closed a new ledger, but main thread hasn't refreshed its snapshot yet). There are different use cases of main thread's ledger state, which must be treated with caution and evaluated individually: - When it is safe: in cases, where LCL is used more like a heuristic or an approximation. Program correctness does not depend on the exact state of LCL. Example: post-externalize cleanup of transaction queue. We load LCL’s close time to purge invalid transactions from the queue. This is safe because if LCL has been updated while we call this, the queue is still in a consistent state. In fact, anything in the transaction queue is essentially an approximation, so a slightly stale snapshot should be safe to use. - When it is not safe: when LCL is needed in places where the latest ledger state is critical, like voting in SCP, validating blocks, etc. To avoid any unnecessary headaches, we introduce a new invariant: “applying” is a new state in the state machine, which does not allow voting and triggering next ledgers. Core must first complete applying to be able to vote on the “latest state”. In the meantime, if ledgers arrive while applying, we treat them like “future ledgers” and apply the same procedures in herder that we do today (don’t perform validation checks, don’t vote on them, and buffer them in a separate queue). The state machine remains on the main thread _only_, which ensures SCP can safely execute as long as the state transitions are correct (for example, executing a block production function can safely grab the LCL at the beginning of the function without worrying that it might change in the background). ### Reflecting state change in the bucketlist Close ledger is the only place in the code that updates the BucketList. Other subsystems may only read it. Example is garbage collection, which queries the latest BucketList state to decide which buckets to delete. These are protected with a mutex (the same LCL mutex used in LM, as bucketlist is conceptually a part of LCL as well).
* Update catchup tests to reflect the new LM policy for being in sync * Rename CatchupManager to LedgerApplyManager to more accurately capture module's responsibilities
# Description Ensure all transactions succeed in loadgen test. This hardens the test a bit and ensures that we don't accidentally drop/fail any of the transactions. Also increase the TPS a bit and add a more realistic ops distribution. # Checklist - [ ] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [ ] Rebased on top of master (no merge commits) - [ ] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [ ] Compiles - [ ] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
# Description Resolves #4618 This hardens Binary Fuse Filter constructor code to use protected vector lookups instead of C style array dereferencing. # Checklist - [x] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [x] Rebased on top of master (no merge commits) - [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [x] Compiles - [x] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
Closes #4620 This changes bumps the minimum overlay version to match the version in `v22.0.0`. Now that the network has upgraded to protocol 22, there's no reason to keep older clients connected.
Closes #4620 This changes bumps the minimum overlay version to match the version in `v22.0.0`. Now that the network has upgraded to protocol 22, there's no reason to keep older clients connected.
…tting. Most of the code is guarded behind the vnext protocol macro and this is a no-op for now. This also required the XDR bump, which in turn required an env bump, which is why I've created a new submodule for p23 env.
…tting (#4609) # Description Add support for the max number of dependent tx clusters per ledger setting. Most of the code is guarded behind the vnext protocol macro and this is a no-op for now. This also required the XDR bump, which in turn required an env bump, which is why I've created a new submodule for p23 env. # Checklist - [ ] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [ ] Rebased on top of master (no merge commits) - [ ] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [ ] Compiles - [ ] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
# Description Resolves #4584 This PR adds support for the Hot Archive to TX apply. Specifically, `InvokeHostFunctionOp` will check both the live BucketList and Hot Archive BucketList for archived entries. The `RestoreFootprintOp` can now restore entries from both the live BucketList and Hot Archive. Additionally, restore meta has also changed for p23. The `LEDGER_ENTRY_RESTORED` type has been added for the restore op. When an entry is restored. both the `LedgerEntry` of the restored data and the corresponding `TTL` entry are emitted as `LEDGER_ENTRY_RESTORED LedgerEntryChangeType`. If the entry has not yet been evicted (such that the entry and it's TTL still exist in the live BucketList), the preexisting TTL value will be emitted as `LEDGER_ENTRY_STATE`. If the entry has been evicted such that the TTL value has been previously deleted, only a single change type of `LEDGER_ENTRY_RESTORED` will be emitted for the new value. For the data being restored, `LEDGER_ENTRY_STATE` is never emitted. Rebased on #4585. # Checklist - [x] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [x] Rebased on top of master (no merge commits) - [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [x] Compiles - [x] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
# Description Update Soroban settings files for https://github.com/stellar/stellar-protocol/blob/master/limits/slp-0001.md. This maintains the 5 to 1 ratio between ledger and tx settings. <!--- Describe what this pull request does, which issue it's resolving (usually applicable for code changes). ---> # Checklist - [ ] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [ ] Rebased on top of master (no merge commits) - [ ] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [ ] Compiles - [ ] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
Resolves #4625 and #4614. This change fixes the tests in the `--disable-postgres` build by disabling parallel ledger close tests when postgres is disabled. This is necessary because parallel ledger close requires postgres. As part of this, I also removed the `TESTDB_BUCKET_DB_PERSISTENT_POSTGRES` test mode, as it's made redundant by the `TESTDB_POSTGRESQL` mode.
Resolves #4625 and #4614. This change fixes the tests in the `--disable-postgres` build by disabling parallel ledger close tests when postgres is disabled. This is necessary because parallel ledger close requires postgres. As part of this, I also removed the `TESTDB_BUCKET_DB_PERSISTENT_POSTGRES` test mode, as it's made redundant by the `TESTDB_POSTGRESQL` mode. # Checklist - [x] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [x] Rebased on top of master (no merge commits) - [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [x] Compiles - [x] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )