I have microservices (Kubernetes pods) running in multiple regions. I have one monitored collection. (all regions will be monitoring the same collection; each region has multiple pods)
I want each document (that was updated) to be processed independently by the deployment-units in each region. Here's how I plan to make this work:
- Each region's deployment-unit will use a distinct processor-name (say CFP-US, CFP-Europe, CFP-Australia)
- All pods running in a region, will use the processor-name for that region, same lease container, and distinct instance-names (say the pod's unique id). (The lease container will actually be the same for all regions).
Questions:
- Does the setup above achieve what I want (i.e. each region independently processing every document that was updated?)
- My collection has only one physical partition. My understanding is that in this case, only one lease will exist for all pods/instances in a region (since a lease is for a physical partition); and each region will have its own lease (since the processor-name is different for each region). Is my understanding correct?
- Say I have 3 pods/instances in a region. Since only lease exists, only pod receives CFP updates at a time while other 2 sit idle. How can I configure the leases/intervals so that all 3 pods get updates (almost) equally? i.e. pod-A gets updates (and pod-B, pod-C sit idle), then after some time, pod-B get updates (and pod-A, pod-C sit idle); then pod-C gets updates (and pod-B, pod-A sit idle); and so-on.
- The doc says that:
the number of instances should not be greater than the number of leases.
Other than the fact that some pods could be sitting idle, does having more instances than leases, cause any other issues? (particularly issues in the lease state, or in acquiring/renewing leases, etc.)
CodePudding user response:
- Yes. As long as they are different Deployment Units (https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/change-feed-processor?tabs=dotnet#deployment-unit) it doesn't matter where they are physically located, each Deployment Unit will receive the changes.
- Correct. Each Deployment Unit will create it's own independent set of leaes.
- You cannot. The distribution of leases-per-machine is by design, there is no configuration to force it.
- That is the ideal scenario, the reason is that all machines (using the LeaseAcquireInterval) frequently check if they can acquire a new lease. If the machine currently owning it releases it (for an unhandled error for example) then the lease could jump to any other machine. Depending on your telemetry, that might be undesirable.