Having trouble using requests to download images off of wiki-CodePudding

I am working on a project where I need to scrape images off of the web. To do this, I write the image links to a file, and then I download each of them to a folder with requests. At first, I used Google as the scrape site, but do to several reasons, I have decided that wikipedia is a much better alternative. However, after I tried the first time, many of the images couldn't be opened, so I tried again with the change that when I downloaded the images, I downloaded them to names with endings that matched the endings of the links. More images were able to be accessed like this, but many were still not able to be opened. When I tested downloading the images myself (individually outside of the function), they downloaded perfectly, and when I used my function to download them afterwards, they kept downloading correctly (i.e. I could access them). I am not sure i it is important, but the image endings that I generally come across are svg.png and png. I want to know why this is occurring and what I may be able to do to prevent it. I have left some of my code below. Thank you.

Function:

def download_images(file):
    object = file[0:file.index("IMAGELINKS") - 1]
    folder_name = object   "_images"
    dir = os.path.join("math_obj_images/original_images/", folder_name)
    if not os.path.exists(dir):
        os.mkdir(dir)
        with open("math_obj_image_links/"   file, "r") as f:
            count = 1
            for line in f:
                try:
                    if line[len(line) - 1] == "\n":
                        line = line[:len(line) - 1]
                    if line[0] != "/":
                        last_chunk = line.split("/")[len(line.split("/")) - 1]
                        endings = last_chunk.split(".")[1:]
                        image_ending = ""
                        for ending in endings:
                            image_ending  = "."   ending
                        if image_ending == "":
                            continue
                        with open("math_obj_images/original_images/"   folder_name   "/"   object   str(count)   image_ending, "wb") as f:
                            f.write(requests.get(line).content)
                        file = object   "_IMAGEENDINGS.txt"
                        path = "math_obj_image_endings/"   file
                        with open(path, "a") as f:
                            f.write(image_ending   "\n")
                        count  = 1
                except:
                    continue
            f.close()

Doing this outside of it worked:

with open("test"   image_ending, "wb") as f:
    f.write(requests.get(line).content)

Example of image link file: https://upload.wikimedia.org/wikipedia/commons/thumb/6/63/Triangle.TrigArea.svg/120px-Triangle.TrigArea.svg.png https://upload.wikimedia.org/wikipedia/commons/thumb/c/c9/Square_(geometry).svg/120px-Square_(geometry).svg.png https://upload.wikimedia.org/wikipedia/commons/thumb/3/33/Hexahedron.png/120px-Hexahedron.png https://upload.wikimedia.org/wikipedia/commons/thumb/2/22/Hypercube.svg/110px-Hypercube.svg.png https://wikimedia.org/api/rest_v1/media/math/render/svg/5f8ab564115bf2f7f7d12a9f873d9c6c7a50190e https://en.wikipedia.org/wiki/Special:CentralAutoLogin/start?type=1x1 https:/static/images/footer/wikimedia-button.png https:/static/images/footer/poweredby_mediawiki_88x31.png

CodePudding user response：

If all the files are indeed in PNG format and the suffix is always .png, you could try something like this:

import requests
from pathlib import Path
u1 = "https://upload.wikimedia.org/wikipedia/commons/thumb/6/63/Triangle.TrigArea.svg/120px-Triangle.TrigArea.svg.png"

r = requests.get(u1)
Path('u1.png').write_bytes(r.content)

CodePudding user response：

My previous answer works for PNG's only

For SVG files you need to check if the file contents start eith the string "<svg" and create a file with the .svg suffix.

The code below saves the downloaded files in the "downloads" subdirectory.

import requests
from pathlib import Path

# urls are stored in a file 'urls.txt'.
with open('urls.txt') as f:
    for i, url in enumerate(f.readlines()):
        url = url.strip()  #  MUST strip the line-ending char(s)!
        try:
            content = requests.get(url).content
        except:
            print('Cannot download url:', url)
            continue
        #  Check if this is an SVG file
        #  Note that content is bytes hence the b in b'<svg'
        if content.startswith(b'<svg'):
            ext = 'svg'
        elif url.endswith('.png'):
            ext = 'png'
        else:
            print('Cannot process contents of url:', url)
        Path('downloads', f'url{i}.{ext}').write_bytes(requests.get(url).content)

Contents of the urls.txt file: (the last url is an svg)

https://upload.wikimedia.org/wikipedia/commons/thumb/6/63/Triangle.TrigArea.svg/120px-Triangle.TrigArea.svg.png
https://upload.wikimedia.org/wikipedia/commons/thumb/c/c9/Square_(geometry).svg/120px-Square_(geometry).svg.png
https://upload.wikimedia.org/wikipedia/commons/thumb/3/33/Hexahedron.png/120px-Hexahedron.png
https://upload.wikimedia.org/wikipedia/commons/thumb/2/22/Hypercube.svg/110px-Hypercube.svg.png
https://wikimedia.org/api/rest_v1/media/math/render/svg/5f8ab564115bf2f7f7d12a9f873d9c6c7a50190e