Trouble piping to curl through sed-CodePudding

I have a program, rtl_433, that outputs lines of JSON - maybe once or twice a minute, while it's running. I need to pipe that data as HTTP POST data to curl.

What complicates matters is that this string needs to be encapsulated in single quotes, which they're not, so I need to add that before shipping it to curl.

Now, here's the thing: This works just fine:

echo '{"qwer":98}' | curl -vvv -u pi:<password> http://data:1880/rtl433 -H "Content-Type: application/json" -d @-

This takes the JSON string '{"qwer":98}' and sends it to the server (Node-RED), where it is received. Now, unfortunately, I have to add those single quotes myself, so I found a sed command that does exactly that, and I ran a test using ping:

$ ping 8.8.8.8 | sed -e "s/.*/'&'/"
'PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.'
'64 bytes from 8.8.8.8: icmp_seq=1 ttl=59 time=18.8 ms'
'64 bytes from 8.8.8.8: icmp_seq=2 ttl=59 time=19.1 ms'
'64 bytes from 8.8.8.8: icmp_seq=3 ttl=59 time=20.4 ms'
'64 bytes from 8.8.8.8: icmp_seq=4 ttl=59 time=18.9 ms'
^C

Perfect! This is exactly what I need - now I need to use sed on the output from rtl_433:

$ rtl_433 -M level -F json | sed -e "s/.*/'&'/"
rtl_433 version unknown inputs file rtl_tcp RTL-SDR SoapySDR
Use -h for usage help and see https://triq.org/ for documentation.
Trying conf file at "rtl_433.conf"...
Trying conf file at "/home/pi/.config/rtl_433/rtl_433.conf"...
Trying conf file at "/usr/local/etc/rtl_433/rtl_433.conf"...
Trying conf file at "/etc/rtl_433/rtl_433.conf"...
Registered 145 out of 175 device decoding protocols [ 1-4 8 11-12 15-17 19-21 23 25-26 29-36 38-60 63 67-71 73-100 102-105 108-116 119 121 124-128 130-149 151-161 163-168 170-175 ]
Detached kernel driver
Found Rafael Micro R820T tuner
Exact sample rate is: 250000.000414 Hz
[R82XX] PLL not locked!
Sample rate set to 250000 S/s.
Tuner gain set to Auto.
Tuned to 433.920MHz.
Allocating 15 zero-copy buffers

'{"time" : "2022-02-16 09:15:40", "model" : "AlectoV1-Rain", "id" : 130, "channel" : 0, "battery_ok" : 1, "rain_mm" : 273.500, "mic" : "CHECKSUM", "mod" : "ASK", "freq" : 433.911, "rssi" : -1.516, "snr" : 40.628, "noise" : -42.144}'

PERFECT! Now all I need to do is to pipe that into curl:

$ rtl_433 -M level -F json | sed -e "s/.*/'&'/" | curl -vvv -u pi:<password> http://data:1880/rtl433 -H "Content-Type: application/json" -d @-

And - nothing! Nothing at all. rtl_433 prints all the initial stuff - and then nothing for minutes. And Node-RED also receives absolutely nothing.

I am really at a loss here. Everything works by itself, but when I combine it together, I get nothing. Not even an error message. What am I missing?

EDIT: I saw a suggest to add --unbuffered to the sed command, but that didn't change anything.

CodePudding user response：

The problem here is that rtl_433 never exits, it just keeps printing lines for every message it receives.

This is a problem because curl is going to wait until it receives an EOF (end of file) marker to know when it has received all payload it needs to send to in the HTTP request, but of course it never gets this. The EOF would be automatically added when rtl_433 exited.

Your best bet is probable to move from a one line script to a proper bash command reads from rtl_433 a line at a time and then runs sed on that line before passing it to curl.

e.g.

#!/bin/bash

rtl_433 -M level -F json | {
  while IFS= read -r line
  do
    echo $line | sed -e "s/.*/'&'/" | curl -vvv -u pi:<password> http://data:1880/rtl433 -H "Content-Type: application/json" -d @-
  done
}