File transfer using socket stream is too slow. Almost 100kbps. I used the python socket module to make this code. It sends data to the client when the client sends the file name. How can I increase the speed of this thing?

Below is the server code

import socket
import os

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(('localhost', 9999))
server.listen()

client, addr = server.accept()

msg = client.recv(1024).decode()

file = open("Server_files/" msg, "rb")
file_size = os.path.getsize("Server_files/" msg)

# client.send("Received_image.png".encode())
client.send(str(file_size).encode())

data = file.read()
client.sendall(data)
client.send(b"<END>")

file.close()
client.close()

Below is the client code

import tqdm
import socket

client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('localhost', 9999))

file_name = input("Enter the file name: ")

client.send(file_name.encode())

file_size = client.recv(1024).decode()
print(file_size)

file = open(file_name, "wb")

file_bytes = b""

done = False

progress = tqdm.tqdm(unit="B", unit_scale=True, unit_divisor=1000,
                     total=int(file_size))

while not done:
    data = client.recv(1024)
    if file_bytes[-5:] == b"<END>":
        done = True
    else:
        file_bytes  = data
    progress.update(1024)

file.write(file_bytes)

file.close()
client.close()

CodePudding user response：

Instead of continuously adding incoming data to file_bytes (which requires potentially a lot of RAM, and also requires Python to reallocate the buffer larger multiple times), you should just write the incoming data directly to file as you receive it.

Also it might help to increase your chunk size from 1024 to something larger, maybe 128*1024; that would allow the system calls to do more work in each call.

CodePudding user response：

Here's a complete solution (Python 3.10) that allows multiple files to be downloaded rapidly. It uses socket.makefile to buffer data, with to automatically close sockets, files, and flush tqdm progress, and writes data as received for speed.

Note that TCP is not a messaged-based protocol. It sends bytes and receives bytes in the same order, but not necessarily with the same "breaks". So send('filename') and send('filesize') can be recv(1024) as 'filenamefilesize' or 'filenamefi' and 'lesize' or any other break. So define a protocol:

Server:

1. read filename followed by '\n'.
2. If received 0 bytes, connection was closed.
3. send filesize as a string in base 10 followed by '\n'.
4. send exactly "filesize" bytes of file data.
5. repeat.

Client:

1. send filename followed by '\n'
2. read filesize followed by '\n', convert to integer of base 10.
3. read exactly "filesize" bytes of file data
4. if done, close connection, else repeat.

server.py

import socket
import os

BLOCK = 128 << 10 # 128KB

with socket.socket() as server:
    server.bind(('', 9999))
    server.listen()

    while True:
        client, addr = server.accept()
        try:
            with (client,
                  client.makefile('rb') as rfile,
                  client.makefile('wb') as wfile):

                while True:
                    filename = rfile.readline()
                    if not filename: break

                    fullname = os.path.join('Server_files', filename.decode().rstrip('\n'))
                    file_size = os.path.getsize(fullname)
                    wfile.write(f'{file_size}\n'.encode())
                    print(f'Sending {fullname}...')
                    with open(fullname, 'rb') as file:
                        while data := file.read(BLOCK):
                            wfile.write(data)
                    wfile.flush() # make sure anything remaining in makefile buffer is sent.
                    print(f' Complete ({file_size} bytes).')
        except ConnectionResetError:
            print('Client aborted.')

client.py

import socket
import tqdm

BLOCK = 1 << 20  # 1MB

with socket.socket() as client:
    client.connect(('localhost', 9999))

    with (client.makefile('rb') as rfile,
          client.makefile('wb') as wfile):

        while True:
            file_name = input('File name (just ENTER to quit): ')
            if not file_name: break
            wfile.write(f'{file_name}\n'.encode())
            wfile.flush() # make sure makefile buffer is fully sent
            file_size = int(rfile.readline().decode())

            with (tqdm.tqdm(unit='B',
                            unit_scale=True,
                            unit_divisor=1000,
                            total=file_size) as progress,
                  open(file_name, 'wb') as file):

                remaining = file_size
                while remaining:
                    data = rfile.read(BLOCK if remaining > BLOCK else remaining)
                    file.write(data)
                    progress.update(len(data))
                    remaining -= len(data)

Output example (note download speeds):

File name (just ENTER to quit): bigfile1
100%|██████████████████████████████████████████████████████████| 191M/191M [00:00<00:00, 706MB/s]
File name (just ENTER to quit): bigfile2
100%|██████████████████████████████████████████████████████████| 218M/218M [00:00<00:00, 718MB/s]
File name (just ENTER to quit):