Reusable Diffusion - Make diffusion fully polymorphic #5016

bolt12 · 2024-11-27T13:10:08Z

Description

This PR completely refactors ouroboros-network in order to make diffusion layer fully general/polymorphic and extensible. This will allow 3rd party users to instantiate their own diffusion and their own diffusion parameters, creating their own overlay network.

More details about this refactor refer to https://github.com/IntersectMBO/ouroboros-network/wiki/Reusable-Diffusion-Investigation

TODOs:

Better document PRP invariants and refactor/cleanup insertion functions
remove daBlockFetchMode
Add tests for the general components: Churn, OG, etc..
Generalise peerChurnGovernor to only have one implementation and not 2
Consensus uses a entirely different project for Cardano Node consensus layer instantiation, decide what's the best approach for the networking code.

bolt12 · 2024-11-28T10:04:42Z

ouroboros-network-api/ouroboros-network-api.cabal

+    Cardano.Node.ConsensusMode
+    Cardano.Node.PeerSelection.Bootstrap
+    Cardano.Node.PeerSelection.LocalRootPeers
+    Cardano.Node.PeerSelection.PeerTrustable
+    Cardano.Node.Types


Check what consensus uses.

What did you decide to use?

ouroboros-network/ouroboros-network.cabal

ouroboros-network/src/Cardano/Node/PublicRootPeers.hs

ouroboros-network/src/Ouroboros/Network/Diffusion/P2P.hs

ouroboros-network/src/Cardano/PeerSelection/Churn.hs

ouroboros-network/sim-tests-lib/Test/Ouroboros/Network/Diffusion/Policies.hs

ouroboros-network/sim-tests-lib/Test/Ouroboros/Network/PeerSelection/Cardano/MockEnvironment.hs

ouroboros-network/src/Ouroboros/Network/PeerSelection/Governor/Types.hs

ouroboros-network/src/Cardano/Network/ArgumentsExtra.hs

ouroboros-network/src/Ouroboros/Network/PeerSelection/Governor/Types.hs

ouroboros-network/src/Ouroboros/Network/PeerSelection/PublicRootPeers.hs

ouroboros-network/src/Ouroboros/Network/PeerSelection/Governor.hs

ouroboros-network/src/Ouroboros/Network/PeerSelection/Governor/Types.hs

ouroboros-network/src/Cardano/Network/PeerSelection/Governor/Types.hs

ouroboros-network/src/Ouroboros/Network/Diffusion/Common.hs

ouroboros-network/src/Ouroboros/Network/Diffusion/MinimalP2P.hs

yihuang · 2024-12-04T13:00:09Z

a step toward haskell blockchain sdk? 😄

bolt12 · 2024-12-05T10:35:57Z

@yihuang as you can see from the PR description this is some interesting step towards a fully polymorphic diffusion that might be used by others to create their own overlay network! Thank you for noticing 😃

There's a cyclic dependency issue in this commit that is fixed in the next one. The problem is that I pulled out `ConsensusModePeerTargets` which depend on `PeerSelectionTargets` and then the module that defines `PeerSelectionTargets` requires `ConsensusModePeerTargets`. This is fixed by abstracting over Cardano Node specific types such as `ConsensusModePeerTargets`.

Moved Diffusion Cardano specifics to their own folder, separating utilities and functions accordingly. Changed Diffusion selection by splitting P2P in two: P2PCardano and P2P. The former instantiates diffusion using Cardano Node specifics. The latter is able to accommodate with other use cases. Added a Minimal/Node.hs that is incomplete but will hold the minimal Node diffusion instantiation example that highlights how one can use their own custom types.

- Removed PeerSelectionActionsArgs: This data type was bit redundant. - Reorganized DNSActions and LedgerPeersArgs - Improved API for PeerSelectionActions

Make PeerSelection{Views, Counters} extensible by third party users. Generalised PublicRootPeers First refactor of Outbound Governor

- Generalize MinimalP2P Module - `MinimalP2P` is _the_ module for diffusion initialization. - Updated parameter plumbing to support generalization and polymorphism. - Enhance ArgumentsExtra - Added additional parameters to `ArgumentsExtra` to facilitate a more general and polymorphic diffusion setup. This includes `requestPublicRootPeers`, `peerChurnGovernor`, etc.. - Refactor PeerSelection.Governor.Monitor - Moved Cardano-specific monitoring actions to a separate module. - Implemented general `localRoots` and `targetPeers` monitoring actions. - Pulled out Cardano specific implementation details that allowed to generalise the minimal set of monitoring actions. Added fields to `ExtraGuardedDecisions` so that 3rd party users can pass their own `localRoots` and `targetPeers` actions and also backdoors/callbacks like `abortGovernor`, `updateWithState` to not only support Cardano specific needs but to give more power/flexibility to 3rd party users. - Polymorphize Diffusion Module - Removed `CardanoP2P` dependencies from the `Diffusion` module. - Ensured the `Diffusion` module remains fully polymorphic and adaptable to different use cases. - Refactored test suite to compile with new changes.

Make addresses polymorphic

crocodile-dentist · 2025-01-07T15:14:00Z

ouroboros-network/src/Ouroboros/Network/PeerSelection/PublicRootPeers.hs

+instance ( Ord peeraddr
+         , extraPeers ~ CardanoPublicRootPeers peeraddr
+         ) => Semigroup (PublicRootPeers extraPeers peeraddr) where


Suggested change

instance ( Ord peeraddr

, extraPeers ~ CardanoPublicRootPeers peeraddr

) => Semigroup (PublicRootPeers extraPeers peeraddr) where

instance ( Ord peeraddr

) => Semigroup (PublicRootPeers (CardanoPublicRootPeers peeraddr) peeraddr) where

ref: cb0da21

crocodile-dentist · 2025-01-07T15:14:49Z

ouroboros-network/src/Ouroboros/Network/PeerSelection/PublicRootPeers.hs

+instance ( Ord peeraddr
+         , extraPeers ~ CardanoPublicRootPeers peeraddr
+         ) => Monoid (PublicRootPeers extraPeers peeraddr) where


same suggestion as above.

coot

I reviewed the first two patches:

git hash	commit title
`bae4f75`	Pull out Cardano specifics to different modules
`cb0da21`	Add extension points to Diffusion data structures

coot · 2025-01-17T09:15:03Z

ouroboros-network-api/ouroboros-network-api.cabal

+    Cardano.Node.ConsensusMode
+    Cardano.Node.PeerSelection.Bootstrap
+    Cardano.Node.PeerSelection.LocalRootPeers
+    Cardano.Node.PeerSelection.PeerTrustable
+    Cardano.Node.Types


What did you decide to use?

coot · 2025-01-17T09:25:26Z

ouroboros-network-api/src/Cardano/Node/Types.hs

+newtype MinBigLedgerPeersForTrustedState =
+  MinBigLedgerPeersForTrustedState { getMinBigLedgerPeersForTrustedState :: Int }


Suggested change

newtype MinBigLedgerPeersForTrustedState =

MinBigLedgerPeersForTrustedState { getMinBigLedgerPeersForTrustedState :: Int }

newtype NumberOfBigLedgerPeers =

NumberOfBigLedgerPeers { getNumberOfBigLedgerPeers :: Int }

the rest is explained in the haddock.

coot · 2025-01-17T09:33:59Z

ouroboros-network/sim-tests-lib/Test/Ouroboros/Network/Diffusion/Testnet/Cardano.hs

+                targetNumberOfRootPeers = 1,
+                targetNumberOfKnownPeers = 1,
+                targetNumberOfEstablishedPeers = 1,
+                targetNumberOfActivePeers = 1,


why the targets are changed?

ref: cb0da21

coot · 2025-01-17T09:34:13Z

ouroboros-network/sim-tests-lib/Test/Ouroboros/Network/Diffusion/Testnet/Cardano.hs

+                targetNumberOfRootPeers = 0,
+                targetNumberOfKnownPeers = 0,
+                targetNumberOfEstablishedPeers = 0,
+                targetNumberOfActivePeers = 0,


Why the targets are changed?

ref: cb0da21

coot · 2025-01-17T09:42:07Z

ouroboros-network/sim-tests-lib/Test/Ouroboros/Network/Diffusion/Testnet/Cardano/Node.hs

+        , caeConsensusMode          = aConsensusMode na
+        , caePeerTargets            = aPeerTargets na


we should drop the a prefix

Suggested change

, caeConsensusMode = aConsensusMode na

, caePeerTargets = aPeerTargets na

, caeConsensusMode = consensusMode na

, caePeerTargets = peerTargets na

ref: cb0da21

coot · 2025-01-17T11:19:39Z

ouroboros-network/src/Ouroboros/Network/PeerSelection/PublicRootPeers.hs

+                => Set peeraddr
+                -> PublicRootPeers extraPeers peeraddr
+fromLedgerPeers lp =
+  (empty mempty) { getLedgerPeers = lp }


empty mempty sounds a bit like Humpty Dumpty 😁

coot · 2025-01-17T11:20:57Z

ouroboros-network/src/Ouroboros/Network/PeerSelection/PublicRootPeers.hs

+instance ( Ord peeraddr
+         , extraPeers ~ CardanoPublicRootPeers peeraddr
+         ) => Semigroup (PublicRootPeers extraPeers peeraddr) where


ref: cb0da21

coot · 2025-01-17T11:26:03Z

ouroboros-network/src/Cardano/Node/PublicRootPeers.hs

+data CardanoPublicRootPeers peeraddr =
+  CardanoPublicRootPeers
+    { cprpGetPublicConfigPeers :: !(Map peeraddr PeerAdvertise)
+    , cprpGetBootstrapPeers    :: !(Set peeraddr)
+    }
+  deriving (Eq, Show, Typeable)


This type should be called CardanoExtraPeers, and then we could provide & use the following:

type CardanoPublicRootPeers peeraddr = PublicRootPeers (CardanoExtraPeers peeraddr) peeraddr

ref: bae4f75

coot · 2025-01-17T11:31:37Z

ouroboros-network/src/Cardano/Node/PeerSelection/Governor/PeerSelectionState.hs

+data CardanoPeerSelectionState =
+  CardanoPeerSelectionState {


I'd call this CardanoExtraState and then we can have & use

type CardanoPeerSelectionState = PeerSelectionState CardanoExtraState ...

ref: bae4f75

coot · 2025-01-17T11:37:33Z

ouroboros-network/src/Ouroboros/Network/PeerSelection/PeerSelectionActions.hs

+  -> PeerSelectionActionsArgs (CardanoPeerSelectionActions m) (CardanoLedgerPeersConsensusInterface m) (CardanoPublicRootPeers peeraddr) PeerTrustable peeraddr peerconn exception m
+  -> WithLedgerPeersArgs (CardanoLedgerPeersConsensusInterface m) m
  -> PeerSelectionActionsDiffusionMode peeraddr peerconn m
  -> (   (Async m Void, Async m Void)
-      -> PeerSelectionActions peeraddr peerconn m
+      -> PeerSelectionActions (CardanoPeerSelectionActions m) (CardanoPublicRootPeers peeraddr) PeerTrustable (CardanoLedgerPeersConsensusInterface m) peeraddr peerconn m
      -> m a)


With some type aliases this could look simpler:

-> CardanoPeerSelectionActionsArgs ... -> WithLedgerPeersArgs (CardanoLedgerPeersConsensusInterface m) m -> PeerSelectionActionsDiffusionMode peeraddr peerconn m -> ( (Async m Void, Async m Void) -> CardanoPeerSelectionActions ... -> m a)

ref: cb0da21

bolt12 added the diffusion Issues / PRs related to diffusion layer label Nov 27, 2024

bolt12 self-assigned this Nov 27, 2024

bolt12 requested a review from a team as a code owner November 27, 2024 13:10

bolt12 force-pushed the bolt12/reusable-diffusion-3 branch 6 times, most recently from 3cdb9a3 to f00952f Compare November 27, 2024 16:53

bolt12 commented Nov 28, 2024

View reviewed changes

bolt12 commented Nov 29, 2024

View reviewed changes

bolt12 force-pushed the bolt12/reusable-diffusion-3 branch from c14d5cb to 3b30f43 Compare December 2, 2024 16:42

bolt12 requested review from coot and crocodile-dentist December 3, 2024 15:13

bolt12 force-pushed the bolt12/reusable-diffusion-3 branch 6 times, most recently from f922210 to 3cdf98d Compare December 9, 2024 10:09

bolt12 added 5 commits January 6, 2025 13:56

Add extension points to Diffusion data structures

cb0da21

Refactor localRootPeers and PeerSelectionActions

86f68f3

- Removed PeerSelectionActionsArgs: This data type was bit redundant. - Reorganized DNSActions and LedgerPeersArgs - Improved API for PeerSelectionActions

Renamed Cardano.Node to Cardano.Network

856abeb

bolt12 force-pushed the bolt12/reusable-diffusion-3 branch 3 times, most recently from fb13f01 to bd855c2 Compare January 7, 2025 11:30

bolt12 force-pushed the bolt12/reusable-diffusion-3 branch from bd855c2 to ce99e1d Compare January 7, 2025 11:39

bolt12 added 12 commits January 7, 2025 12:07

Refactored ConsensusModePeerTargets

f1b04ef

Refactor PeerSelection{Views, Counters}

409b01d

Make PeerSelection{Views, Counters} extensible by third party users. Generalised PublicRootPeers First refactor of Outbound Governor

Update CHaP and Fix build

e9a9938

Remove daBlockFetchMode

f4293b5

Address Review Feedback

eef654f

Fix Churn no timeouts test

cd3aa60

Moved Cardano folder to Ouroboros folder

331ccac

Increase shortDelay when churning to fix test

5b0e2fa

Recover sigusr1 signal handler

9ef2b15

Make addresses polymorphic

Refactor extraDebugState appropriately

ecf1f10

Added P2PDecisionType and Refactored requestPublicRootPeers type

5400a7c

bolt12 force-pushed the bolt12/reusable-diffusion-3 branch from ce99e1d to 5400a7c Compare January 7, 2025 12:07

crocodile-dentist reviewed Jan 7, 2025

View reviewed changes

coot reviewed Jan 17, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reusable Diffusion - Make diffusion fully polymorphic #5016

Reusable Diffusion - Make diffusion fully polymorphic #5016

bolt12 commented Nov 27, 2024 •

edited

Loading

bolt12 Nov 28, 2024

coot Jan 17, 2025

yihuang commented Dec 4, 2024 •

edited

Loading

bolt12 commented Dec 5, 2024

crocodile-dentist Jan 7, 2025 •

edited

Loading

coot Jan 17, 2025

crocodile-dentist Jan 7, 2025

coot left a comment •

edited

Loading

coot Jan 17, 2025

coot Jan 17, 2025

coot Jan 17, 2025

coot Jan 17, 2025

coot Jan 17, 2025

coot Jan 17, 2025

coot Jan 17, 2025

coot Jan 17, 2025

coot Jan 17, 2025

coot Jan 17, 2025

		newtype MinBigLedgerPeersForTrustedState =
		MinBigLedgerPeersForTrustedState { getMinBigLedgerPeersForTrustedState :: Int }

		, caeConsensusMode = aConsensusMode na
		, caePeerTargets = aPeerTargets na

Reusable Diffusion - Make diffusion fully polymorphic #5016

Are you sure you want to change the base?

Reusable Diffusion - Make diffusion fully polymorphic #5016

Conversation

bolt12 commented Nov 27, 2024 • edited Loading

Description

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yihuang commented Dec 4, 2024 • edited Loading

bolt12 commented Dec 5, 2024

crocodile-dentist Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coot left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bolt12 commented Nov 27, 2024 •

edited

Loading

yihuang commented Dec 4, 2024 •

edited

Loading

crocodile-dentist Jan 7, 2025 •

edited

Loading

coot left a comment •

edited

Loading