Apps: txn.Access list for access to more resources #6286

jannotti · 2025-03-26T16:56:33Z

Summary

Today, app developers find it frustrating that they can only list 8 resources in a transaction. This number is artificially low because there are rules that allow access to many more than 8 items when, for example, 4 account and 4 apps are listed.

This PR introduced a single unified Access field on app calls, which contains all accounts, apps, asas, and boxes that the transaction can touch. Because no extra rules allow access to extra resources, we can expand the allowable size of such a list. 32 seems likely. (edit: currently thinking 16, but also upping the box quota per reference to 2k)

This will probably increase performance, since it will now be reasonable to perform perfect prefetching of all resources an app call might touch.

This PR does not yet augment goal to create such transactions. It probably should, as that is also the best way to write e2e tests.

This PR does not actually implement the improved pre-fetching. Should it?

Test Plan

joe-p · 2025-03-26T19:10:14Z

This shouldn't need a new AVM version, correct? Since AVM9 we've had group resource sharing, which means an app has no way of knowing what resources are available. Although I suppose it could check if it's an outer call and check the rest of the group, but seems highly improbable to account for all the sharing rules even if an app wanted to do that.

jannotti · 2025-03-26T19:48:04Z

I think you're asking "will only new programs, AVM v12, be able to use this?"

I think it should be ok to let old programs use this, including letting these resources be accessed by other programs in the same group that are a low version. I'm pretty sure we made the same decision with resource pooling. After the consensus upgrade, programs suddenly got access to things they didn't "know" they had access to by looking into the arrays.

I'll have to confirm whether we put in any limitations. For example, I think it would be unsafe to let v3 (!) programs see extra ASAs.

codecov · 2025-03-26T23:03:38Z

Codecov Report

Attention: Patch coverage is 57.22802% with 287 lines in your changes missing coverage. Please review.

Project coverage is 51.81%. Comparing base (f8ec83f) to head (5b820c5).
Report is 26 commits behind head on master.

Files with missing lines	Patch %	Lines
cmd/goal/application.go	20.66%	118 Missing and 1 partial ⚠️
data/basics/testing/nearzero.go	67.67%	28 Missing and 4 partials ⚠️
libgoal/transactions.go	78.00%	22 Missing ⚠️
data/transactions/application.go	83.96%	16 Missing and 5 partials ⚠️
util/fn.go	0.00%	17 Missing ⚠️
shared/pingpong/pingpong.go	0.00%	16 Missing ⚠️
cmd/goal/asset.go	0.00%	12 Missing ⚠️
cmd/goal/interact.go	0.00%	12 Missing ⚠️
daemon/algod/api/server/v2/utils.go	0.00%	11 Missing ⚠️
data/transactions/logic/eval.go	88.13%	4 Missing and 3 partials ⚠️
... and 7 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6286      +/-   ##
==========================================
+ Coverage   51.68%   51.81%   +0.13%     
==========================================
  Files         649      651       +2     
  Lines       86857    87236     +379     
==========================================
+ Hits        44893    45204     +311     
- Misses      39099    39169      +70     
+ Partials     2865     2863       -2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

joe-p · 2025-03-27T00:05:00Z

Right that's what I would think as well. The main reason I ask is because if we don't require a specific AVM version existing applications, even if immutable, can leverage this feature by simply updating their client-side code.

This commit ensures that assets are made available properly with tx.Access

jannotti · 2025-04-02T15:49:28Z

Right that's what I would think as well. The main reason I ask is because if we don't require a specific AVM version existing applications, even if immutable, can leverage this feature by simply updating their client-side code.

Having worked on it some more:

Yes, any transaction can use tx.Access instead of the existing arrays, even if the transaction is an app call to an old app. Old apps that use "slots" with opcodes will index into tx.Access instead of the respective array. That element of tx.Access has to be the proper type. So int 3; int 4; asset_holding_get will expect an account in tx.Access[3-1] and an asset in tx.Access[4-1]. (The -1 is because tx.Access is always 1-based so that 0 can mean Sender or current app for address and apps.)

As for sharing, we only enabled sharing for v9 programs and higher, so I made that true here as well. Your v9 or higher programs will be able to see the resources made available from other transactions, whether those transactions made the resources available with the "old-style" foreign arrays or with tx.Access.

Like tx.Boxes, I do not plan on making tx.Access explicitly visible to apps, there will not be txna Access 3. And inner transactions will not be able to populate tx.Access for their inners. The first limitation is to prevent apps from the bad practice of passing arguments in tx.Access. That should be done explicitly, so that availability can be handled in various ways (maybe another transaction has the resource in tx.Access, for example). The second limitation is to reduce complexity - generally you shouldn't need to pass tx.Access because of resource sharing.

algorandskiy

Left some comments
Empty box ref in Access gaining extra reads budged should be somehow better documented

data/transactions/application.go

algorandskiy · 2025-04-08T18:00:38Z

data/transactions/application.go

+		rr.Holding.Empty() && rr.Locals.Empty() && rr.Box.Empty()
+}
+
+// wellFormed checks that a ResourceRef is either empty or of only one kind, and


probably requires some clarification that rr is one of access elements. Maybe a better interface would be a free function taking access and looping over it?

I added some clarification.

The main wellFormed() function for app calls is pretty much what you're describing (though it has other checks too.

It does this:

for _, rr := range ac.Access { if err := rr.wellFormed(ac.Access, proto); err != nil { return err } }

Would you like it better if the whole loop was pulled into it's own function, instead of the "inner" part of the loop as rr.wellFormed()?

data/transactions/application_test.go

algorandskiy · 2025-04-08T18:24:02Z

data/transactions/logic/eval.go

-			return basics.AppIndex(cx.txn.Txn.ForeignApps[ref-1]), nil
+			return cx.txn.Txn.ForeignApps[ref-1], nil
+		}
+		// it seems most consistent to allow slot access into tx.Access if it is used with an old app.


this looks inconsistent - we do not allow both ForeignApps and Access use in a transaction but here (and in resolveApp below) both are checked

I could not decide which makes for clearer code. Only one of A or B can be true, so is it nice to write:

if isA { A } else { B }

or nice to just write

A B

I decided to go with the later to save space. Either A or B becomes a no-op because the length of one of the arrays is 0.

algorandskiy · 2025-04-08T18:32:36Z

data/txntest/txn.go

@@ -117,6 +118,10 @@ func (tx *Txn) internalCopy() {
 	for i := 0; i < len(tx.Boxes); i++ {
 		tx.Boxes[i].Name = append([]byte(nil), tx.Boxes[i].Name...)
 	}
+	tx.Access = append([]transactions.ResourceRef(nil), tx.Access...)
+	for i := 0; i < len(tx.Access); i++ {
+		tx.Access[i].Box.Name = append([]byte(nil), tx.Access[i].Box.Name...)


check for non-empty tx.Access[i].Box ?

This is just copying the name for immutability. If it's nil, it stays nil. That's why I used the weird construct of append onto nil. That function does it a lot, for the Note, ApplicationArgs etc.

Check out this beauty, co-authored by chatgpt.

algorandskiy · 2025-04-09T20:09:55Z

data/basics/testing/nearzero.go

+// the same type in which each element of the slice is the same type as the
+// sample, but exactly one field (or sub-field) is set to a non-zero value. It
+// returns one example for every sub-field.
+func NearZeros(t *testing.T, sample any) []any {


maybe add a simple random test creating a struct with reflect with a few fields of random type and make sure this function returns that exact number of element in a slice?

You mean a test for NearZeros itself? I agree, I'll write some.

This required changes to libgoal interfaces that we previously designed to take all the foreign arrays, ready for packing into the transaction. Now they receive a "RefBundle" of things the tx needs access to, so it should be possible to change that to use tx.Access. Still needs final e2e subs tests using goal.

jannotti self-assigned this Mar 26, 2025

jannotti added the Enhancement label Mar 26, 2025

jannotti force-pushed the access-list branch from 38bb0f4 to 21b22bc Compare March 26, 2025 22:38

jannotti added 3 commits April 2, 2025 10:17

Initial stuctures and consensus flags for txn.Access

e19795b

Committing before some state delta cleanup

98272a3

TestAssets test coverage for tx.Access

b8578c3

This commit ensures that assets are made available properly with tx.Access

jannotti force-pushed the access-list branch from 17dd112 to b8578c3 Compare April 2, 2025 14:19

jannotti force-pushed the access-list branch 5 times, most recently from 98e1981 to 1573e6b Compare April 2, 2025 18:56

wellFormed for tx.Acccess checks internal indices

872e497

jannotti force-pushed the access-list branch from 1573e6b to 872e497 Compare April 2, 2025 19:28

Testing and fixes for cross-product resources

dbbd1bd

jannotti force-pushed the access-list branch from d269419 to dbbd1bd Compare April 4, 2025 13:50

jannotti marked this pull request as ready for review April 4, 2025 15:39

Many app call well-formed tests for tx.Access

2ad657d

algorandskiy reviewed Apr 8, 2025

View reviewed changes

jannotti added 3 commits April 8, 2025 22:29

Reflection based testing for Empty() completeness

13bf5c7

Check out this beauty, co-authored by chatgpt.

Correctly check and test for handling multiple types in one elt

4609198

Test fixes for quota bump

62751b6

jannotti force-pushed the access-list branch from 373bca2 to 62751b6 Compare April 9, 2025 16:22

algorandskiy reviewed Apr 9, 2025

View reviewed changes

jannotti force-pushed the access-list branch from e48cdc5 to f247f5d Compare April 10, 2025 13:08

NearZero testing

728815a

jannotti force-pushed the access-list branch 7 times, most recently from f04ef4d to 9371062 Compare April 15, 2025 17:18

jannotti force-pushed the access-list branch from 9371062 to 5b820c5 Compare April 15, 2025 18:40

Merge remote-tracking branch 'upstream/master' into access-list

3e12d5a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Apps: txn.Access list for access to more resources #6286

Apps: txn.Access list for access to more resources #6286

jannotti commented Mar 26, 2025 •

edited

Loading

joe-p commented Mar 26, 2025

jannotti commented Mar 26, 2025 •

edited

Loading

codecov bot commented Mar 26, 2025 •

edited

Loading

joe-p commented Mar 27, 2025

jannotti commented Apr 2, 2025

algorandskiy left a comment

algorandskiy Apr 8, 2025

jannotti Apr 15, 2025

algorandskiy Apr 8, 2025

jannotti Apr 9, 2025

algorandskiy Apr 8, 2025

jannotti Apr 9, 2025

algorandskiy Apr 9, 2025

jannotti Apr 9, 2025

jannotti Apr 10, 2025 •

edited

Loading

Apps: txn.Access list for access to more resources #6286

Are you sure you want to change the base?

Apps: txn.Access list for access to more resources #6286

Conversation

jannotti commented Mar 26, 2025 • edited Loading

Summary

Test Plan

joe-p commented Mar 26, 2025

jannotti commented Mar 26, 2025 • edited Loading

codecov bot commented Mar 26, 2025 • edited Loading

Codecov Report

joe-p commented Mar 27, 2025

jannotti commented Apr 2, 2025

algorandskiy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jannotti Apr 10, 2025 • edited Loading

Choose a reason for hiding this comment

jannotti commented Mar 26, 2025 •

edited

Loading

jannotti commented Mar 26, 2025 •

edited

Loading

codecov bot commented Mar 26, 2025 •

edited

Loading

jannotti Apr 10, 2025 •

edited

Loading