How do I extract details (body, from, subject) from the incoming SendGrid inbound parse webhook raw-CodePudding

I'm receiving the following payload in my SendGrid inbound parse webhook (the webhook points at my firebase cloud function):

data435 --xYzZY
Content-Disposition: form-data; name="dkim"

{@gmail.com : pass}
--xYzZY
Content-Disposition: form-data; name="email"

Received: by mx0099p1iad2.sendgrid.net with SMTP id 0QPMKulr9m Fri, 30 Sep 2022 06:39:28  0000 (UTC)
Received: from mail-lf1-f47.google.com (unknown [209.85.167.47]) by mx0099p1iad2.sendgrid.net (Postfix) with ESMTPS id 0780EE0961 for <[email protected]>; Fri, 30 Sep 2022 06:39:28  0000 (UTC)
Received: by mail-lf1-f47.google.com with SMTP id bu25so5446311lfb.3 for <[email protected]>; Thu, 29 Sep 2022 23:39:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date; bh=oR2ttmiMxoDGGpj0W7rOOSeDm7QR0gEvVzc1 jKfl08=; b=Eb4zbXHvt4GHz9 gdicE5V 7O7gDB xXHickuuXb7aoGlIAYIcZS8VArvs3f58VJ9E Z hM06A63eMHyaVhvaApdRy/e/wVHMzCg4cbjfO8GLxtN xQ qwZg6eFO90R8KN7iB4j fDbnA6oemv6kaTwnVrZgrcaVmezBUiH2ZfLykNqtgP9ysx 5aii53EZP1B0qB0Vo3k5r LDUqeo WyuX3pEVlp9GbE9cVbDpcx8dKCqiO5Hy3BSMMHtspgNIALLpX8JMLMjgsWxbe LV7S3r8NMsfehYaATc38JZZVlRYB/SDQeBF7ythydBxWxJ8onH l/9yggQSsyKdG JNf WiMw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date; bh=oR2ttmiMxoDGGpj0W7rOOSeDm7QR0gEvVzc1 jKfl08=; b=Spj3 a3z45RsOcrk946mJ3talAofD9bCbqhHksXvnYg144sSj9h6uFnI7D4e9r7Yrt 9Cs/ptj9x83wZu55/luMtdmBTXGm8XgTJyi1fOzZa6gANBJbf9FHde7avhpCBbpc udO cMERu0gQSM6shBiO0bOoMpywG07T6qW3glEEog5W/yji1NtZNzgZ7QhhZJAOpD4zwXKe JeLztSey/O8HO6/c41gi6HVX7LYEOztqEzKqXvvcXBkLjCkjZVZh37NcVpzgJ15PoP3V SB1sB25XpCGsbf4 7Rd cbJ8GjPtbgigfJtqN1vX7z1PAp4EaGyCV/TurruvEph56CL/ OkhA==
X-Gm-Message-State: ACrzQf1LFncmabsqOilbRnxolKJ1KtE0K7WwJYZ3D6ne5hGk9k5qTNxm ge5gg /XqquwMgtMzBXvkkAhdv0gwQ5UAan7jjqlLcjynnM=
X-Google-Smtp-Source: AMsMyM7irSVbK1v42mqnKCK3s25yqN2XM6XS7q8Ya//jBjOBnWAXlb33VmBIZAQNurF3DqzD4RXYcCvrkb2fztc1nig=
X-Received: by 2002:a05:6512:b85:b0:499:202a:bde9 with SMTP id b5-20020a0565120b8500b00499202abde9mr2950268lfv.550.1664519966548; Thu, 29 Sep 2022 23:39:26 -0700 (PDT)
MIME-Version: 1.0
From: 5WsApp Dev1 <[email protected]>
Date: Fri, 30 Sep 2022 09:39:15  0300
Message-ID: <CAO57T8mOpdo8eQd9BKiY4cPB0fSEsA=931erdHZOtskUD29QcQ@mail.gmail.com>
Subject: sub1234
To: [email protected]
Content-Type: multipart/alternative; boundary="00000000000053e68705e9df4158"

--00000000000053e68705e9df4158
Content-Type: text/plain; charset="UTF-8"

body1234

--00000000000053e68705e9df4158
Content-Type: text/html; charset="UTF-8"

<div dir="ltr">body1234</div>

--00000000000053e68705e9df4158--

--xYzZY
Content-Disposition: form-data; name="to"

[email protected]
--xYzZY
Content-Disposition: form-data; name="from"

5WsApp Dev1 <[email protected]>
--xYzZY
Content-Disposition: form-data; name="sender_ip"

209.85.167.47
--xYzZY
Content-Disposition: form-data; name="envelope"

{"to":["[email protected]"],"from":"[email protected]"}
--xYzZY
Content-Disposition: form-data; name="subject"

sub1234
--xYzZY
Content-Disposition: form-data; name="charsets"

{"to":"UTF-8","subject":"UTF-8","from":"UTF-8"}
--xYzZY
Content-Disposition: form-data; name="SPF"

pass
--xYzZY--

How do I extract details from it (body, from, to, subject)? I have tried libraries such "simpleParser" but it just doesn't extract the details.

CodePudding user response：

The trick is that you need to parse the parameters that come with the POST request (Content-Type: multipart/form-data). Then, you can easily extract the default parameters you are looking for. Here's how you would do that with fastify

import fastify, { FastifyInstance } from "fastify";
import fastifyMultipart from "fastify-multipart";

const server: FastifyInstance = fastify({});

server
  .register(fastifyMultipart, { addToBody: true })
  .all("/mail", async (request, reply) => {
    const sgBody = request.body as EmailBody;

    const sender = JSON.parse(sgBody.envelope).from;
   
    // Read the parameters here
   });

server.listen(3000, function (err, address) {
  if (err) {
    server.log.error(err);
    process.exit(1);
  }
  console.log(`Restarted at: ${address}`);
});

PS: This snippet comes from my sample repo on GitHub.

CodePudding user response：

OK, I found the solution. It seems that SendGrid doesn't send the payload in JSON format but as base64 that must be converted to string. Therefore, when using Google cloud functions, you need to extract the details from the string. Make sure you unchecked "POST the raw, full MIME message" at the defined host at https://app.sendgrid.com/settings/parse

I came around the problem the following way:

  const payload = Buffer.from(req.body).toString(); // Convert from base64 to string

  let array = payload.split("--xYzZY") //This is the default boundary for multipart MIME messages

  let body = array.find(t => t.includes("Content-Disposition: form-data; name=\"text\""))
  body = body.split('"text"')[1]

  let to = array.find(t => t.includes("Content-Disposition: form-data; name=\"to\""))
  to = to.split('"to"')[1].replace("\n", "")

  let from = array.find(t => t.includes("Content-Disposition: form-data; name=\"from\""))
  from = from.split('"from"')[1].replace("\n", "")

  let attachments = array.find(t => t.includes("Content-Disposition: form-data; name=\"attachments\""))
  attachments = attachments.split('"attachments"')[1]
  let numOfAttach = parseInt(attachments.replace("\n", ""))