Go | Why does this singular goroutine deadlock when using a WaitGroup while uploading a file?-CodePudding

Edit: The main problem turned out to be the actual uploading process and not the deadlock that occured, which was simply caused by a misplaced wg.Wait()

I am trying to upload a file to an online file hosting service (https://anonfiles.com/) via their API. There is an upload file size limit of 20GB.

I can upload a simple text file that is around 2KB with the code below. However, if I try to do the same with a larger file, lets say, around 2MB, I get the following error from their API: No file chosen.

I thought this was because the code (below) was not waiting for the go routine to properly finish, so I added a wait group. I then got this error from Go: fatal error: all goroutines are asleep - deadlock!.

I have tried removing the WaitGroup below that seems to be causing the deadlock; but then the code below the go routine will run before the go routine is actually finished.

With the WaitGroup removed, I can still upload files that are KB in size, but files that are larger do not upload to the file hosting correctly, since I receive the No file chosen error from their API.

package main

import (
    "fmt"
    "io"
    "log"
    "math/rand"
    "mime/multipart"
    "net/http"
    "os"
    "sync"
    "time"
)

func main() {
    client := http.Client{}

    // Upload a >2MB wallpaper.
    file, err := os.Open("./wallpaper.jpg")
    if err != nil {
        log.Fatal(err)
    }

    defer file.Close()

    reader, writer := io.Pipe()
    multipart := multipart.NewWriter(writer)

    /* 
    Added Waitgroup to make sure the routine properly finishes. Instead, causes deadlock.
    wg := new(sync.WaitGroup)
    wg.Add(1) 
    */

    go func() {
        fmt.Println("Starting Upload...")
        defer wg.Done()
        defer writer.Close()
        defer multipart.Close()

        part, err := multipart.CreateFormFile("file", file.Name())
        if err != nil {
            log.Fatal(err)
        }

        fmt.Println("Copying...")
        if _, err = io.Copy(part, file); err != nil {
            log.Fatal(err)
        }
    }()

    fmt.Println("The code below will run before the goroutine is finished; without the WaitGroup.")

    req, err := http.NewRequest(http.MethodPost, "https://api.anonfiles.com/upload", reader)
    if err != nil {
        log.Fatal(err)
    }

    req.Header.Add("Content-Type", multipart.FormDataContentType())

    resp, err := client.Do(req)
    if err != nil {
        log.Fatal(err)
    }

    wg.Wait()

    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(string(body))
}

I have researched several issues, but none seem to apply to my problem. What is causing this to lock up? What can be done differently? Perhaps this is some rookie mistake, any suggestions or help would be appreciated.

CodePudding user response：

TL;DR

Set the Content-Length header of the request.

A working demo is attached to the end of this answer.

Debugging

I think the deadlock issue is not important here. Your purpose is to upload files to https://anonfiles.com/. So I will focus on debugging the uploading issue.

First, let's upload a file with curl:

curl -F "[email protected]" https://api.anonfiles.com/upload

It works.

Then let's upload the same file with your demo, it fails with the misleading response:

{
  "status": false,
  "error": {
    "message": "No file chosen.",
    "type": "ERROR_FILE_NOT_PROVIDED",
    "code": 10
  }
}

Now let's replace the target https://api.anonfiles.com/upload with https://httpbin.org/post so that we can compare the requets:

  {
   "args": {}, 
   "data": "", 
   "files": {
     "file": "aaaaaaaaaa\n"
   }, 
   "form": {}, 
   "headers": {
-    "Accept": "*/*", 
-    "Content-Length": "197", 
-    "Content-Type": "multipart/form-data; boundary=------------------------bd4a81e725230fa6", 
     "Accept-Encoding": "gzip",
     "Content-Type": "multipart/form-data; boundary=2d4e7969789ed6ef6ff3e7b815db3aa040fd3994a34fbaedec85240dc5af",
     "Host": "httpbin.org", 
-    "User-Agent": "curl/7.81.0", 
-    "X-Amzn-Trace-Id": "Root=1-63747739-2c1dab1b122b7e3a4db8ca79"
     "Transfer-Encoding": "chunked",
     "User-Agent": "Go-http-client/2.0",
     "X-Amzn-Trace-Id": "Root=1-63747872-2fbc85f81c6dde7e5b2091c4"
   }, 
   "json": null, 
   "origin": "47.242.15.156", 
   "url": "https://httpbin.org/post"
 }

The outstanding difference is that curl sends "Content-Length": "197" while the go app sends "Transfer-Encoding": "chunked".

Let's try to modify the go app to send the Content-Length header:

package main

import (
    "bytes"
    "fmt"
    "io"
    "log"
    "mime/multipart"
    "net/http"
    "strings"
)

func main() {
    source := strings.NewReader(strings.Repeat("a", 1<<21))

    buf := new(bytes.Buffer)
    multipart := multipart.NewWriter(buf)

    part, err := multipart.CreateFormFile("file", "test.txt")
    if err != nil {
        log.Fatal(err)
    }

    if _, err := io.Copy(part, source); err != nil {
        log.Fatal(err)
    }
    multipart.Close()

    req, err := http.NewRequest(http.MethodPost, "https://api.anonfiles.com/upload", buf)
    if err != nil {
        log.Fatal(err)
    }

    req.Header.Add("Content-Type", multipart.FormDataContentType())

    // The following line is not required because the http client will set it
    // because the request body is a bytes.Buffer.
    // req.ContentLength = int64(buf.Len())

    client := http.Client{}
    resp, err := client.Do(req)
    if err != nil {
        log.Fatal(err)
    }

    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(string(body))
}

It works!

The disadvantage is that it has to copy the request body into the memory first. It seems to me that this is unavoidable because it needs to know the size of the request body.