Goal is to retrieve Azure DevOps users with their license and project entitlements in go.

I'm using Microsoft SDK.

Our Azure DevOps organization has more than 1500 users. So when I request each user entitlements, I have an error message due to Azure DevOps rate limit => 443: read: connection reset by peer

However, limiting top with 100/200 does the job, of course..

For a real solution, I though not using SDK anymore and using direct REST API calls with a custom http handler which would support rate limit. Or maybe using heimdall.

What is your advise for a good design guys ?

Thanks.

Here is code :

package main

import (
    "context"
    "fmt"
    "github.com/microsoft/azure-devops-go-api/azuredevops"
    "github.com/microsoft/azure-devops-go-api/azuredevops/memberentitlementmanagement"
    "log"
    "runtime"
    "sync"
    "time"
)

var organizationUrl = "https://dev.azure.com/xxx"
var personalAccessToken = "xxx"

type User struct {
    DisplayName         string
    MailAddress         string
    PrincipalName       string
    LicenseDisplayName  string
    Status              string
    GroupAssignments    string
    ProjectEntitlements []string
    LastAccessedDate    azuredevops.Time
    DateCreated         azuredevops.Time
}

func init() {
    runtime.GOMAXPROCS(runtime.NumCPU()) // Try to use all available CPUs.
}

func main() {
    // Time measure
    defer timeTrack(time.Now(), "Fetching Azure DevOps Users License and Projects")

    // Compute context
    fmt.Println("Version", runtime.Version())
    fmt.Println("NumCPU", runtime.NumCPU())
    fmt.Println("GOMAXPROCS", runtime.GOMAXPROCS(0))
    fmt.Println("Starting concurrent calls...")

    // Create a connection to your organization
    connection := azuredevops.NewPatConnection(organizationUrl, personalAccessToken)

    // New context
    ctx := context.Background()

    // Create a member client
    memberClient, err := memberentitlementmanagement.NewClient(ctx, connection)
    if err != nil {
        log.Fatal(err)
    }

    // Request all users
    top := 10000
    skip := 0
    filter := "Id"
    response, err := memberClient.GetUserEntitlements(ctx, memberentitlementmanagement.GetUserEntitlementsArgs{
        Top:        &top,
        Skip:       &skip,
        Filter:     &filter,
        SortOption: nil,
    })

    usersLen := len(*response.Members)

    allUsers := make(chan User, usersLen)

    var wg sync.WaitGroup
    wg.Add(usersLen)

    for _, user := range *response.Members {
        go func(user memberentitlementmanagement.UserEntitlement) {
            defer wg.Done()

            var userEntitlement = memberentitlementmanagement.GetUserEntitlementArgs{UserId: user.Id}
            account, err := memberClient.GetUserEntitlement(ctx, userEntitlement)
            if err != nil {
                log.Fatal(err)
            }

            var GroupAssignments string
            var ProjectEntitlements []string

            for _, assignment := range *account.GroupAssignments {
                GroupAssignments = *assignment.Group.DisplayName
            }

            for _, userProject := range *account.ProjectEntitlements {
                ProjectEntitlements = append(ProjectEntitlements, *userProject.ProjectRef.Name)
            }

            allUsers <- User{
                DisplayName:         *account.User.DisplayName,
                MailAddress:         *account.User.MailAddress,
                PrincipalName:       *account.User.PrincipalName,
                LicenseDisplayName:  *account.AccessLevel.LicenseDisplayName,
                DateCreated:         *account.DateCreated,
                LastAccessedDate:    *account.LastAccessedDate,
                GroupAssignments:    GroupAssignments,
                ProjectEntitlements: ProjectEntitlements,
            }
        }(user)
    }

    wg.Wait()
    close(allUsers)
    for eachUser := range allUsers {
        fmt.Println(eachUser)
    }
}

func timeTrack(start time.Time, name string) {
    elapsed := time.Since(start)
    log.Printf("%s took %s", name, elapsed)
}

CodePudding user response：

You can write custom version of GetUserEntitlement function.

https://github.com/microsoft/azure-devops-go-api/blob/dev/azuredevops/memberentitlementmanagement/client.go#L297-L314

It does not use any private members.

After getting http.Response you can check Retry-After header and delay next loop's iteration if it is present.

https://github.com/microsoft/azure-devops-go-api/blob/dev/azuredevops/memberentitlementmanagement/client.go#L306

P.S. Concurrency in your code is redundant and can be removed.

Update - explaining concurrency issue:

You cannot easily implement rate-limiting in concurrent code. It will be much simpler if you execute all requests sequentially and check Retry-After header in every response before moving to the next one.

With parallel execution: 1) you cannot rely on Retry-After header value because you may have another request executing at the same time returning a different value. 2) You cannot apply delay to other requests because some of them are already in progress.

CodePudding user response：

For a real solution, I though not using SDK anymore and using direct REST API calls with a custom http handler which would support rate limit. Or maybe using heimdall.

Do you mean you want to avoid the Rate Limit by using the REST API directly?

If so, then your idea will not work.

Most REST APIs are accessible through client libraries, and if you're using SDK based on a REST API or other thing based on a REST API, it will of course hit a rate limit.

Since the rate limit is based on users, I suggest that you can complete your operations based on multiple users (provided that your request is not too much that the server blocking your IP).