The problem

block_on in general is a bad practice imo - it constantly causes issues because it doesn't compose. A function called with block_on (or an async context) cannot call another function with block_on. This is not symmetrical with async - a sync function can call an async function using block_on, but an async function cannot call a sync function that uses block_on.

This is revelant to the the tests in ic-nervous-system-integration-tests because PocketIc in blocking mode (the default) uses block_on. This excludes the possibility of using it in an async context.

For a concrete example, we can’t currently use sns-audit in integration tests since it must work with ic-agent (which is async). This means that any functions called from sns-audit must work in an async context. It would be very nice to be able to use sns-audit in integration tests, since it would mean that we could have a centralized place to do the checks for “did the SNS get deployed the way it was supposed to” that could be used on mainnet and in tests. Using sns-audit in tests that would also mean that we would not be able to make a change to how SNSes are deployed that is breaking w.r.t. sns-audit without also fixing sns-audit (which we currently basically do not test for IIUC)

The solution

Use PocketIC in non-blocking mode. What this means generally is that you need to add .await and async everywhere. There are a lot of changes in this PR, but they almost all very mechanical.

In some cases we had closures that called blocking pocket_ic functions. These generally could be fixed by using async move.

The only non-mechanical change is that some places that used iterators had to be changed to use streams or just use traditional for i in range iteration.