Limit number of goroutine picking task in loop-CodePudding

I am running a loop - 24 times which calls a downstream that is not able to handle all the work supplied concurrently by the 24 go-routines spawned by the loop I want to limit that only a specific number (like 3 or 4) go-routine will run in concurrently, this but all 24 tasks needs to be down

Sample code looks like below, if anyone can point me to the right pattern to fulfil this would be a great help

for i:=0; i<24; i   {
   go callSomeDownStream()
}

CodePudding user response：

You can use the channel of empty structs to control the number of concurrent worker goroutines

const MAX_CONCURRENT_JOBS = 3

func main() {
    waitChan := make(chan struct{}, MAX_CONCURRENT_JOBS)

    for i:=0; i < 24; i   {
        waitChan <- struct{}{}
        go func() {
            callSomeDownStream()
            <-waitChan
        }()
    }
}

CodePudding user response：

func callSomeDownStream(wg *sync.WaitGroup, queue chan struct{}) {
    defer func() {
        <-queue
        wg.Done()
    }()
    // do something
}

func main() {
    wg := sync.WaitGroup{}
    queue := make(chan struct{}, 3)
    for i := 0; i < 24; i   {
        queue <- struct{}{}
        wg.Add(1)
        go callSomeDownStream(&wg, queue)
    }
    wg.Wait()
    close(queue)
}

CodePudding user response：

I think this sounds like the worker-pools concept when we want to limit the number of concurrent jobs: https://gobyexample.com/worker-pools