Python web scraping: How to configure multipart/form-data payload-CodePudding

I'm trying to scrape this site: https://kfz.virtuelles-rathaus.de/igv2-man/servlet/Internetgeschaeftsvorfaelle

But I can't figure out how to set up a valid request body. The Status Code is 200, so the request itself works. In the response it says it couldn't process to data, because I used the Browser navigation. It wants me to use the Website buttons instead.

I think it is because the payload isn't configured correctly, but I can't get it to work.

I already tried following:

Setting the headers manually
Using requests_toolbelt.multipart.encoder
Creating the boundary for the multipart/data-format myself

The decodedPayload copied from Chrome Dev Tools

WKZ_UNTERSCH_Z: WT
WKZ_ERKENN_Z: SJ
WKZ_ZIFFERN: 454
WKZ_SUCHMERKMAL: NULL
BTN_WKZSUCHE: suchen
ZEITSTEMPEL: 2022040815031191

The encoded Payload copied from Chrome Dev Tools

------WebKitFormBoundaryj9dFOsSgrDr5dSwA
Content-Disposition: form-data; name="WKZ_UNTERSCH_Z"

WT
------WebKitFormBoundaryj9dFOsSgrDr5dSwA
Content-Disposition: form-data; name="WKZ_ERKENN_Z"

SJ
------WebKitFormBoundaryj9dFOsSgrDr5dSwA
Content-Disposition: form-data; name="WKZ_ZIFFERN"

454
------WebKitFormBoundaryj9dFOsSgrDr5dSwA
Content-Disposition: form-data; name="WKZ_SUCHMERKMAL"

NULL
------WebKitFormBoundaryj9dFOsSgrDr5dSwA
Content-Disposition: form-data; name="BTN_WKZSUCHE"

suchen
------WebKitFormBoundaryj9dFOsSgrDr5dSwA
Content-Disposition: form-data; name="ZEITSTEMPEL"

2022040815031191
------WebKitFormBoundaryj9dFOsSgrDr5dSwA--

My Code:

import requests
from datetime import datetime


def main():

    payload = {"WKZ_UNTERSCH_Z": "WT",
               "WKZ_ERKENN_Z": "GH",
               "WKZ_ZIFFERN": "454",
               "WKZ_SUCHMERKMAL": "NULL",
               "BTN_WKZSUCHE": "suchen",
               "ZEITSTEMPEL": datetime.strftime(datetime.now(), '%Y%m%d%H%S%M%f')[:-4]}

    url = 'https://kfz.virtuelles-rathaus.de/igv2-man/servlet/Internetgeschaeftsvorfaelle'

    # Initialize Session and get Cookie with session ID
    s = requests.Session()
    r = s.get(f'{url}?MANDANT=08337000&AUFRUF=WKZ')

    r = s.post(
        f'{url}', data=payload, verify=False)

    # Save Response for further scraping
    with open('z_1.html', 'w') as f:
        f.write(str(r.text))


if __name__ == '__main__':
    main()

I already thank you guys in advance for your help

EDIT:

The Body that's created when using the MultipartEncoder:

--b3dccffd58a47883c42249db16600856
Content-Disposition: form-data; name="WKZ_UNTERSCH_Z"

WT
--b3dccffd58a47883c42249db16600856
Content-Disposition: form-data; name="WKZ_ERKENN_Z"

SJ
--b3dccffd58a47883c42249db16600856
Content-Disposition: form-data; name="WKZ_ZIFFERN"

454
--b3dccffd58a47883c42249db16600856
Content-Disposition: form-data; name="WKZ_SUCHMERKMAL"

NULL
--b3dccffd58a47883c42249db16600856
Content-Disposition: form-data; name="BTN_WKZSUCHE"

suchen
--b3dccffd58a47883c42249db16600856
Content-Disposition: form-data; name="ZEITSTEMPEL"

2022040909485493
--b3dccffd58a47883c42249db16600856--

CodePudding user response：

I found the answer to my problem

I wasn't a problem with the data format, but I used the wrong timestamp.

The solution is to use the timestamp of the previous response and not the current time.