Emit Transfer/Mint/Burn/Clawback Contract Events for Classic Transfers

[leighmcculloch]

What

I propose we unify transfer events on the network in smart contract events, by exploring emitting transfer contract events, like those defined by SEP-41, CAP-46-6, and those potentially defined by #1534, for classic transfers (payments, path payments, etc).

Why

To create a singe unified data stream of all movements of assets / tokens on Stellar.

Today there is no single way to consume transfer events on the network. Transfers are divided by the classic vs smart contract divide, and then also on a separate dimension by stellar assets and custom tokens.

The RPC is the way developers are intended to stream and ingest live events occurring on Stellar. The RPC exposes smart contract events. However any application using the RPC and wanting to ingest transfers must implement complex logic merging the results of streaming smart contract events and transactions containing payments, path payments, offers, so on.

The shortest paths to a payments implementation today is to go back to Horizon's /effects endpoints, or Horizon's /payments endpoint, however the migration path we're wanting to create is in the opposite direction from Horizon to the RPC. There should be a way for developers to ingest all transfers from RPC, otherwise the path for migrating from Horizon to RPC is not realistic.

We could explore ways to add payment information to RPC, such as a similar Horizon-like /effects endpoint or /payments endpoint, however I think that is solving the problem at the wrong layer and repeating mistakes of the past. We've made that mistake before with Horizon, by solving all indexing problems at the Horizon layer which encouraged folks to build on Horizon rather than innovate on new and or better data sources.

RPC's API is designed to provide a raw view into the network and live / current network data. It exposes raw transaction XDR, and raw events. RPC doesn't contain "business logic" or even "indexer logic" for understanding network data. This is critical because if you build on RPC, you're building on the same data that's also available from Galexie, other CDP tooling, or raw meta. You have access to the same data, so applications can select between them without being disadvantaged, and without artificial / convenience forcing functions causing folks to interact with one when another would be more appropriate.

What's critical is that network data be a good foundation to build on. In many ways it is, but for understanding transfers between accounts it is unfortunately not. Soroban transfers have resolved this problem for transfers originating on Soroban through Soroban events, and if classic adopted the same events network data would tell a great coherent story about transfers.

If we solve the problem at the network layer, then events on Stellar can support a unified view into the movement of value on the network.

How

All transfers / payments on Stellar classic are the result of interactions with Stellar Assets. All Stellar Assets have a reserved contract address and a reserved installation of the built-in Stellar Asset Contract. For example, on mainnet (pubnet) the native asset has contract address CAS3J7GYLGXMF6TDJBBYYSE3HQ6BBSMLNUQ34T6TZMYMW2EVH34XOWMA. Stellar classic operations can emit Soroban events using the asset's reserved contract address, even if the reserved contract address hasn't been deployed to yet for that contract. All classic transfers of value are mappable to Soroban events, because the Soroban Stellar Asset Contract was modelled on the Stellar asset classic behaviours.

A proposed mapping exists at: https://docs.google.com/document/d/1511nhYdUiaGAJ-CT7TzVeAdBaTm4HLBgEFndfjno4jc

And is being discussed here: #1553 (comment)

Links

Discussion on Discord: https://discord.com/channels/897514728459468821/1285736582426394760

[Shaptic]

Stellar classic operations could emit Soroban events using that contract address

Define "emit"? I don't think this is even logistically possible from Core's perspective. Since smart contracts operate in their own land of execution and thus emit events into metadata via host functions, how would classic operations replicate this? Are you suggesting they inject ContractEvents into the meta despite (a) not coming from a contract, (b) possibly referring to a nonexistent contract address, and (c) having no understanding of ScVals? I should let them speak to its feasibility, though.

Don't get me wrong, this sounds really nice, but I just can't see how it's possible.

[leighmcculloch]

Are you suggesting they inject ContractEvents into the meta despite (a) not coming from a contract

"Not coming from a contract" is an implementation detail. The Stellar Asset Contract is a code API into existing classic data. Events are data about how data has changed, and transfer events provide information about how classic data has changed. The proposal here is to emit events about changes in the same data.

possibly referring to a nonexistent contract address

This is unfortunate, but I think a trade-off that's worth the benefits. It's not a nonexistent contract address in anyway that is unsafe. Contract addresses are reserved. The event contains enough information to identify the asset reliably, so even if the SAC contract isn't deployed, the asset is still identifiable.

Because of state archival, it's always possible for a contract to emit an event, and shortly after for that event to reference a contract that's not "live" on chain, so problems that arise from a SAC not existing would be similar to the problems that arise from state archival anyway.

having no understanding of ScVals

ScVal's are just an XDR type. Core knows about all XDR types. For example: https://github.com/search?q=repo:stellar/stellar-core%20ScVal&type=code

this sounds really nice

💯 ❤️

[Shaptic]

Great points all around. Couple more points to follow up:

• I'd like to suggest renaming ContractEvent for XDR clarity since it wouldn't always come from contracts.
• When you say "transfer events," you mean the exact same format as the SAC today, right?
• Does this proposal need a detailed enumeration of the things that happen on the native operations' side that would result in a transfer event?

[leighmcculloch]

Yup a formal proposal / CAP would need a detailed enumeration of classic happenings to events.

[dmkozh]

While this seems easy enough to support for simple payments, I wonder what should we do for things like path payments, DEX trades, claimable balances etc. In Soroban world these would contain a number of transfer events, as well as maybe some other events. I don't think that's feasible for classic, as we wouldn't want to have an operation to emit a lot of events (these won't have a fee, unlike in Soroban case).

I wonder if a subset of operations that move balances is sufficient to support. Basically, it would be nice to identify the requirements/use cases that we can support with certain 'classic' events.

[leighmcculloch]

I wonder if a subset of operations that move balances is sufficient to support. Basically, it would be nice to identify the requirements/use cases that we can support with certain 'classic' events.

+1. If there's general consensus that this is a direction to explore, then I think the next step would be to do some assessment of all possible movements of value on classic, and mapping / excluding, so that we could talk about the implications more specifically.

[Shaptic]

I don't think that's feasible for classic, as we wouldn't want to have an operation to emit a lot of events (these won't have a fee, unlike in Soroban case).

Yeah, this separates it from Horizon's /payments which are 1:1 and is more like its /effects which are 1:N, where we derive all of the side-effects for an operation (e.g., account_debited, offer_updated, account_credited for a simple path payment). N is typically > 1 but I don't think long paths are very common which is the only operation that can have a large number of events. So isn't it generally okay that there won't be fees since there's no VM cost being incurred and the events can't get arbitrarily large?

Alternatively +1 to exploring mapping / excluding.

[dmkozh]

The event fee actually goes towards the meta storage (we don't want it to be possible to just spam meta with arbitrary events for free), but I guess the missing fee is probably acceptable given that the event size will be proportional to the diffs we emit anyway. But I'm still not sure if implementing very granular event reporting is worth the effort, which is why I'd like to understand what's actually useful.

[mollykarcher]

RPC's API is designed to provide a raw view into the network and live / current network data. It exposes raw transaction XDR, and raw events. RPC doesn't contain "business logic" for understanding network data.

Just wanted to big ➕ this! There have been a handful of conversations lately (internal + external) that have assumed or implied that we will be rebuilding or recreating the unified data models that we built out in Horizon, but now in RPC. Repeating this behavior would only recreate a lot of the issues we saw with Horizon, and allow RPC to become the defacto backend/monolith for any and all applications that want on-chain data. So yes, I'd wholeheartedly agree that the right layer to solve this at feels like the source-of-truth of the events data (ie. Core).

I wonder if a subset of operations that move balances is sufficient to support

I think this is the right bit to start with as there's generally a lot of interest around a unified understanding of "what is a payment" or "what moves value" (lots of interest/talk around an open-source payment indexer, for example).

[JakeUrban]

I'm a big fan of this proposal. I want to point out that there are two layers in this unification effort -- the source of the data and the format of the data.

Providing a stream for all stellar asset transactions at the protocol level via events, as opposed to at the application level (like Horizon's /payments endpoint), makes sources of this data interchangeable. Clients can use RPC, a txmeta lake (produced by galaxie), captive core, or some other source of txmeta.

However, I think it is equally important that we unify the format of the data provided. We could add events for all stellar asset transfers, providing a unified the stream for this data, but if the format of these events vary, clients will inevitably support a subset of these formats. We have this same problem right now with Horizon's /payments endpoint -- most clients support the simple payment format, but don't support the path payment or SAC transfer formats.

Unfortunately I think we've already introduced multiple data formats in cases where a single format could have sufficed. For example, mint, burn, and clawback are all effectively transfers, but clients may unintentionally omit support for these events in their implementation because the first topic is the name of the contract function, rather than "transfer". If clients wanted to know more about the nature of the transfer (i.e. whether it was a mint, burn, clawback, simple transfer, path payment, etc.) they could inspect other information included in the associated TransactionMetaV3.

So lets take path payments for example. I would advocate that the event(s) emitted as a result of a path payment operation use the ["transfer", from, to, asset]: amount format. Clients interested in this event can inspect the associated txmeta to determine if it was a path payment if they'd like.

I'd like to see events in this format emitted for the following classic operations:

• payment
• both path payment types
• create account
• merge account

Now there are some classic operations where using the same format may not be the best path because there is not an obvious from and to address.

• create claimable balance
• claim claimable balance
• amm deposits
• amm withdraws

In these cases, we may need to use a different format. The good news is that most applications won't need to care about these events because they don't use claimable balances or native AMMs.

Finally, I think we should also consider emitting events for transaction fees. These are a kind XLM transfer, and anyone using events to update balances won't be able to maintain an accurate XLM balance without this event.

[dmkozh]

These are a kind XLM transfer, and anyone using events to update balances won't be able to maintain an accurate XLM balance without this event.

This is a very important point that I've alluded to above. If we want to make it possible to track the exact balances only via events, then the things will become much more complicated, as we'd need to cover every balance movement at once. It's not obvious to me that this should be a requirement. I think an argument can be made that getting the balance is a very common operation and in general a user shouldn't ingest each and every event concerning a given account (and also interpret these correctly), just to figure out something as simple as balance is. Thus I'm not sure this should be a requirement.

[leighmcculloch]

Just to clarify some history and why on these things:

mint, burn, and clawback are all effectively transfers, but clients may unintentionally omit support for these events in their implementation because the first topic is the name of the contract function, rather than "transfer"

Transfers is the movement of an amount between two addresses and SEP-41 does use "transfer" as the topic of any event that moves an amount between accounts. For example, the transfer_from function generates a "transfer" event.

The mint, burn, clawback events are separate because they create and destroy amounts, they do not move amounts between the issuer and the holder, even though they do affect the balance of a holder. On Classic stellar these operations were represented by transfers, but in Soroban it was decided not to represent those happenings as transfers. An application can still ingest these events also though.

[Shaptic]

Just to clarify

On Classic stellar these operations were represented by transfers

More accurately, Horizon interprets the side effects of these operations as payments and presents them as such, where a payment is literally just a concept we made up. Just to echo the general sentiment that standardization of a transfer at the source of truth would go a long way 👍

[leighmcculloch]

I've written up a document taking the list that @JakeUrban posted at #1553 (comment) and documenting what I think each of the events would need to be emitted to align the events with the events that the Stellar Asset Contract already emits today.

https://docs.google.com/document/d/1511nhYdUiaGAJ-CT7TzVeAdBaTm4HLBgEFndfjno4jc

Please leave comments about specifics directly on the document ☝🏻, or more general broad comments here 👇🏻.

cc @JakeUrban appreciate your 👀 again.

cc @sisuresh @dmkozh @anupsdf @MonsieurNicolas

Cross-posted to Discord: https://discord.com/channels/897514728459468821/1285736582426394760/1300244089102532718

[dmkozh]

All proposed events are the same size as existing events that get published by the Stellar Asset Contract. Events are afaik designed to be as small as possible.

Keep in mind that classic events are free, unlike soroban events. There is also no limit for the size of events emitted by an operation.

Could this be the push towards multiple categories of meta, or optional/configurable data within meta, where an operator can enable "events meta" only, and disable other types of meta. Meta may become smaller rather than capturing the full transaction meta that exists today to derive the same events off-chain.

Yeah, that seems reasonable. I'm not sure if the sizes are going to be always be smaller, but events should have 'asymptotically' the same size as the ledger diffs, so we should be fine if we were to emit only events or only diffs most of the time.

Ideally, an event emitted by core would describe both the state transition (i.e. the transfer) and the new state (the updated balances). This is what we're aiming to do with the "replace ledger entry changes with events".

This sounds suspiciously similar to the current ledger diffs. If the end state (and maybe the start state) of the entry are necessary, let's use the appropriate meta format (ledger diffs). Balance is just a single example of this; there are also account/asset properties beyond the balance (e.g. authorization). The Soroban approach to this is to use events to track the updates and getters to get the interesting data (such as balance).

[sisuresh]

Another issue with emitting a Balance event is that it would be unique to classic. Tracking balances for smart wallets will require a different method, leading to the fragmentation we want to avoid.

[leighmcculloch]

Tracking balances for smart wallets will require a different method

Why would that be? Custom tokens that share a concept of a balance could emit an event communicating changes in balance.

[leighmcculloch]

This proposal is focused on bringing consistency between Stellar classic and Soroban so that classic operations that move assets get emitted in the same way that asset movement on Soroban does. This proposal makes no changes to Soroban emitted events, and only brings classic into consistency.

I think it's been helpful for us to zoom out to see if the scope should be broader, but so far the ideas for tracking balances have all discussed what could be implemented as additive features, so I think we should explore them as a separate proposal.

As part of that separate proposal process, we should consider why other ecosystems have not needed balance events. When looking at what data providers do for balance tracking in EVM chains, there's some evidence (see below) that in other ecosystems this is done via tracking transfer events which contain no balance information. So we may be solving a problem at a layer that's not necessary to solve at this layer.

---

Dune does balance tracking via tracking ERC20 transfer events:

• For ERC20: Analysis is based on token transfer events detailed in the tokens_ethereum_base_transfers. The address balance is updated based on the to/from addresses in the transfer events. If the token is a non-standard ERC20 token, the balance might not be updated until we detect a standard ERC20 transfer event. Known issues currently include ERC4626 tokens and rebasing tokens like stETH or $ampleforth. We are working on improving the range of balance changes that we detect.
• For ERC721 and ERC1155: Inspection of NFT transfer activities via the nft_ethereum_transfers, focusing on the to/from addresses.

Ref: https://docs.dune.com/data-catalog/curated/evm/asset-tracking/balances#balance-calculation-methodology

[sisuresh]

Tracking balances for smart wallets will require a different method

Why would that be? Custom tokens that share a concept of a balance could emit an event communicating changes in balance.

Yeah sorry I wasn't clear. This proposal is about adding events to classic ops, so I meant if that wasn't done for smart wallets, then we'd have to figure something out. We could add that event to the token standard (although any existing non-upgradeable tokens would be out of luck).

[JakeUrban]

@jopmiddelkamp I'd be curious on your take from Beans' perspective. Or if you'd continue using a third-party indexer, maybe you can ask your indexer to take a look at this?

@earrietadev what is xBull's solution for indexing value transfers? Would this be helpful?

I know from Freighter and Vibrant's perspective, this would be quite helpful because it would make our implementation much simpler.

Today Vibrant listens for classic payments and path payments using RPC's getTransactions method, and it doesn't support SAC transfers. If we wanted to support them, we'd need to use getEvents in addition to getTransactions for the classic ops. Vibrant also doesn't index some of the transfer types that are unlikely but still possible to occur, like account merges. Unifying all transfers using events would ensure they don't miss anything without putting a lot of thought into ensuring all cases are considered.

Its a similar situation for Freighter.

[earrietadev]

For xBull because of the different ways to know a transfer of funds was made (payments, create account, merge account, path payments, trades, etc), I ended up using account effects instead (which also includes SAC transfers) and just sending users to stellar.expert if they want to see more information.

At the end of the day, most users just want to know how much left their accounts and how much is entering their accounts, they don't really care "how" or "why" the transfer was executed (especially if they are keeping business payment records).

I think using just events will be similar to just using effects so in my opinion that makes everything easier.

A note regarding fees, it will be extremely useful if the amount of fee paid is also an event because that's also an amount of funds leaving the account which with the current tools we need to make a separate call (like getting the transaction and filter if the same account paid the fee) just to show that or when you try to see a historical variation of the XLMs amounts in an account.

[leighmcculloch]

I think using just events will be similar to just using effects so in my opinion that makes everything easier.

Events will also have transfers for custom tokens, unlike effects which currently don't.

[jopmiddelkamp]

@jopmiddelkamp I'd be curious on your take from Beans' perspective. Or if you'd continue using a third-party indexer, maybe you can ask your indexer to take a look at this?

@earrietadev what is xBull's solution for indexing value transfers? Would this be helpful?

I know from Freighter and Vibrant's perspective, this would be quite helpful because it would make our implementation much simpler.

Today Vibrant listens for classic payments and path payments using RPC's getTransactions method, and it doesn't support SAC transfers. If we wanted to support them, we'd need to use getEvents in addition to getTransactions for the classic ops. Vibrant also doesn't index some of the transfer types that are unlikely but still possible to occur, like account merges. Unifying all transfers using events would ensure they don't miss anything without putting a lot of thought into ensuring all cases are considered.

Its a similar situation for Freighter.

I’ve reached out to BlockDeamon to see if one of their Stellar developers could review this, as we’re utilizing them as our third-party indexer.

[piyalbasu]

Just to provide a little context from Freighter's side: we hoped to use an indexer to help us parse transfer events, but it's been tough to find the right fit for us. Currently, we're opting to construct token transfers in an account's history using the XDR from Horizon's transaction's endpoint and checking to see if it's a transfer or a mint. If it is, we parse out the params for display (to, from, amount, symbol, etc). This only works for sent token transfers, as there's no equivalent transaction for received.

Our current approach requires quite a bit of manual effort to get this to work at all. I think this type of manual data parsing is a big stumbling block for new wallet developers, especially ones that want to support both classic and Soroban. The logic becomes pretty hairy very quickly and devs end up spending time maintaining logic that was "out of the box" with Horizon rather than innovating. This proposal would be a big win for wallet engineers.

I'd also add that from Freighter's perspective, I'd lean more towards having as much context around the event as possible. While it's true that users probably care most about the account effect (change of balance), for a wallet like Freighter that's geared towards power users, we tend to want to show all the context so they can know exactly what happened. Is it reasonable to have the emitted event be verbose by default and offer maybe a helper in an SDK to parse that down to something simpler that's closer to how account effects work now?

[leighmcculloch]

Hi folks, conversation seems to have slowed in this proposal, suggesting some consensus exists, with the discussions on balances having identified a new problem we should think about and potentially solve independently.

One of the most critical decisions with this proposal is where we solve the problem.

The proposal as it is has the problem being solved in Stellar-Core.

Alternatives that were considered:

• RPC
  • no because that would create artificial motivators for people to use the RPC over other data sources
• Some new service that sits very close to core and generates events ingested by all existing systems that ingest meta
  • no because the events that core already emits are what we'd want more of, and already speak directly into the desired domain, and this proposal is a tweak to emit them in all cases rather than only the contract subset
  • maybe because decoupling event production from core might help maintainers from an implementation level iterate faster
  • no because it would be difficult to decouple as the data would have to be provided by core and so this decoupling would actually be coupled transformation

Does anyone have alternative views on this?

For folks who think this is being solved in the right place, please leave a positive emoji of some kind.

[sisuresh]

Reviving this thread - It sounds like there's support for adding the events described in this doc to stellar-core. We still need to figure out exactly what's required. Do the unified events need to be emitted retroactively? Also, the soroban events vector in tx meta can currently be verified because it's hashed into the ledger, so these classic events (which won't be hashed into the ledger), will need to either be a different field in meta, or put into the diagnostic events vector.

Related to this, we need to make a change to make sure the events are semantically correct for transfers involving the issuer in the SAC - discussion. This will result in different logic to consume the events between protocol versions (a transfer call from an issuer will emit a mint event in one protocol, and then a transfer event in the next protocol).

[sisuresh]

As I've mentioned above, protocol specification + events should be in theory sufficient for arriving at the next state of the balance entries, though it doesn't proof that the movements of value match the events (the same can be said for the current ledger entry diff meta). But having them actually hashed into ledgers makes verification much easier and also ensures that every event is correct.

Yeah the proof aspect of this isn't really important IMO, and I doubt anyone is going through the hassle of verification. We initially didn't hash Soroban events into the ledger, but changed our minds towards the end of Soroban development because hashing the events would let us catch a divergence during replay if I remember correctly. The same argument can be made for classic events, but my initial reasoning to avoid hashing the events was that it'd be simpler, and the end user (most likely) wouldn't care. Having said that, I'm fine with hashing the events. It might make sense to hash all of the events in a tx result extension at the operation layer, and stop hashing anything in InvokeHostFunctionResult for consistency.

[sydneynotthecity]

I'm not sure I understand. Option 1 proposes to emit events in the same meta field for every protocol version, why would it not mean that the downstream schema is consistent? The only difference is really whether these events get hashed into ledger or not (if we're going to hash them at all).

Ah, nevermind then. I thought you were proposing that the events in general would not be emited as a contract event until P23. If it's only the hash that's the difference, i'm also in favor of option 1.

[chowbao]

So it looks like the new classic events will be available for full history (through an optional flag/setting). Will full history (backfilled classic events) be included when this CAP is implemented and released into P23/P24? I wanted to confirm because without backfilled classic events for all of history it wouldn't be possible to aggregate and get correct balances only using events

[Shaptic]

Would it make sense to only backfill from Soroban (P20) onward? Everyone that cares already has transfer data indexed/parsed/stored since then, and events become a "new" concept if you go pre-20 as opposed to "aligned" if you go 20 onward.

[dmkozh]

I'm not sure it makes sense, given that events are not sufficient to infer the balance if not tracked from the genesis. It's possible to optimize ingestion by bootstrapping if necessary - Core can catchup in any ledger range and there is nothing special to do about it. The only benefit of explicitly cutting the backfill is to not deal with some quirks of the old protocols, but I'm not sure if these are bad enough (and Core needs to maintain them anyway).

[sisuresh]

Here's the initial CAP - https://github.com/stellar/stellar-protocol/blob/master/core/cap-0067.md. There are some notable points that'll probably change -

1. The CAP specifies that the events will be emitted at the Transaction layer, and not the Operation layer. I'll think about alternative solutions that emit the events at the Operation layer.
2. The CAP doesn't emit the set_authorized event on trustline flag changes, but the Stellar Asset Contract currently does. While the set_authorized event is not necessary to track balance changes, it makes classic consistent with soroban, so we may add that event to this CAP.
3. The CAP still needs to specify how retroactive events will work, and go through any edge cases due to older protocol behavior. Note that the retroactive events cannot be hashed into the ledger.

[chowbao]

What's the process for comments in the initial CAP that you merged? Like there is no PR anymore to comment on. Is the expectation that all discussion should be done in this discussion even if it was something like a rewording suggestion or a typo fix suggestion?

[sisuresh]

Yeah discussions should be done here. For grammar or typo fixes, you can either open a PR fixing them, or you can just let me know if it's one or two changes and I can add them to my next update.

[Shaptic]

I noticed the following line in the proposed CAP-67 for this discussion after #1617:

By using the existing events<> vector in SorobanTransactionMeta. We can avoid making any xdr changes. This does have some tradeoffs, mainly that all events for a given transaction will be emitted in a single vector, making it impossible to distinguish which operation emitted a specific event. The alternative would be to move Soroban meta from the transaction layer into the operation layer of transaction meta, but that would be a breaking change.

(Also noted by @sisuresh in (1), above.) This seems prohibitive to downstream ingesters, unless the events themselves are expanded to make it completely unnecessary to inspect the operation itself (which would require more details in each classic event's data, imo).

Could we make events<> a nested vector, instead, in a binary-compatible way, or will that not work because of a prepended length byte? I'm not 100% sure how XDR encodes things. We could also add classic events into a new SorobanTransactionMetaExt (V2).

[sisuresh]

I think a cleaner solution would be to introduce a TransactionMetaV4, and pull anything related to events out of SorobanTransactionMeta. The events could be in an OperationMetaV2 or something like that under TransactionMetaV4. What do you think?

[Shaptic]

Wouldn't that still be breaking to SorobanTransactionMeta if events aren't in it anymore? Or do you mean introducing a V2 of it for MetaV4?

[sisuresh]

Yeah we'd need a SorobanTransactionMetaV2 as well. This path will require some work for downstream consumers, but the meta will make more sense.

[Shaptic]

MetaV4 introduces more opportunities to 💥 in downstream integrations, but if we're thorough I think we can mitigate this. Would Soroban transactions still use V3 (which would increase backwards compatibility) or is V4 the universal payload?

[chowbao]

I like the MetaV4 (and other +V*). I would also prefer V4 the universal payload. Basically going with a "clean slate" going forward approach