I have seen #7162, but this is all new greenfield stuff, I have never used those old versions of dapr.

This error means that two (or more) concurrent operations trying to mutate the same state are happening on your Cosmos collection and one got rejected due to record versioning. This is a retriable error that you can safely retry from your code. However, please note that we completely revamped the actor reminder system in Dapr 1.14 - it'd be great if you could upgrade to 1.14.1 and enable the Scheduler service with the following configuration:

apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: featureconfig
spec:
  features:
    - name: SchedulerReminders
      enabled: true

Then apply the configuration to your app with the following annotation: dapr.io/config: featureconfig.

You should not only see these errors resolved but also get improved performance. Notice that old reminder data will not be moved over.

Yes I am anxiously awaiting 1.14 to hit the AKS Dapr extension, not quite available yet. Great job on the work there!

As for retrying this error, not really sure it will help. I'm not seeing intermittent failure; it's failing every time on every workflow. I too thought there must be concurrency somewhere, but I can't find it. It seems there is only 1 instance of this workflow running and nothing else is touching that state.

Yes I am anxiously awaiting 1.14 to hit the AKS Dapr extension, not quite available yet. Great job on the work there!

As for retrying this error, not really sure it will help. I'm not seeing intermittent failure; it's failing every time on every workflow. I too thought there must be concurrency somewhere, but I can't find it. It seems there is only 1 instance of this workflow running and nothing else is touching that state.

Can you reach out to me on Discord? handle is yaron2. We can debug there and take findings here.

It seems this was due to Cosmos consistency level and multiple regions being enabled. Session consistency was the cause. We switched to Bounded Staleness and that resolved it, although there is a cost associated with that so it would be good if Session support could be added.

One question I have is, with the new reminder scheduling, if we delete and recreate our AKS cluster will reminder now be lost?

It seems this was due to Cosmos consistency level and multiple regions being enabled. Session consistency was the cause. We switched to Bounded Staleness and that resolved it, although there is a cost associated with that so it would be good if Session support could be added.

One question I have is, with the new reminder scheduling, if we delete and recreate our AKS cluster will reminder now be lost?

Yes, all existing reminder data will be lost.

This issue has been automatically marked as stale because it has not had activity in the last 60 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.