Getting Python output in real-time from C /boost::process-CodePudding

From a C program (running under Windows 10), I use boost::process to invoke Python in order to interpret a simple Python script. I want to redirect Python script's output in real-time to my C program's console.

My problem is that I'm getting the whole Python script output at once when the program completed, I'm not getting it in real-time.

Here is my MCVE:

Python script (script.py):

import time
from datetime import datetime

print( "Started Python script at t="   str(datetime.now().time()) )

time.sleep(1)
print( "Slept 1 sec, t="   str(datetime.now().time()) )
time.sleep(1)
print( "Slept 1 sec, t="   str(datetime.now().time()) )

print( "Stopped Python script at t="   str(datetime.now().time()) )

C program (main.cpp):

#include <boost/process.hpp>
#include <ctime>
#include <iostream>
#include <chrono>

namespace bp = boost::process;

std::ostream &operator<<(std::ostream &stream, const std::chrono::system_clock::time_point& time_point) 
{
    const auto time {std::chrono::system_clock::to_time_t (time_point)};
    const auto localtime {*std::localtime (&time)};
    const auto time_since_epoch {time_point.time_since_epoch()};
    const auto milliseconds_count {std::chrono::duration_cast<std::chrono::milliseconds> (time_since_epoch).count() % 1000};
    
    stream << "[" << std::put_time (&localtime, "%T") << "." << std::setw (3) << std::setfill ('0') << milliseconds_count << "] - ";
    return stream;
}

int main( int argc, char* argv[] )
{
    std::cout << std::chrono::system_clock::now() << "Creating child" << std::endl;
    bp::ipstream stream;
    bp::child c("python.exe", "script.py", bp::std_out > stream);
    std::cout << std::chrono::system_clock::now() << "Created child" << std::endl;

    std::cout << std::chrono::system_clock::now() << "Invoking getline" << std::endl;
    std::string line;
    while (getline(stream, line)) {
        std::cout << std::chrono::system_clock::now() << "From Python output: " << line << std::endl;
    }
    std::cout << std::chrono::system_clock::now() << "getline ended" << std::endl;

    c.wait();

    return 0;
}

This program outputs:

[12:50:34.684] - Creating child
[12:50:34.706] - Created child
[12:50:34.708] - Invoking getline
[12:50:36.743] - From Python output: Started Python script at t=12:50:34.742105
[12:50:36.745] - From Python output: Slept 1 sec, t=12:50:35.743111
[12:50:36.745] - From Python output: Slept 1 sec, t=12:50:36.743328
[12:50:36.746] - From Python output: Stopped Python script at t=12:50:36.743328
[12:50:36.747] - getline ended

As you can see, we get the whole 4 outputs after Python process ended (first call to getline freezes for 2 seconds - while Python runs the two time.sleep(1) - and then within 3ms we get the whole 4 lines). I would expect to get something like:

[12:50:34.684] - Creating child
[12:50:34.706] - Created child
[12:50:34.708] - Invoking getline
[12:50:34.XXX] - From Python output: Started Python script at t=12:50:34.742105
[12:50:35.XXX] - From Python output: Slept 1 sec, t=12:50:35.743111
[12:50:36.XXX] - From Python output: Slept 1 sec, t=12:50:36.743328
[12:50:36.XXX] - From Python output: Stopped Python script at t=12:50:36.743328
[12:50:36.XXX] - getline ended

I suspect the problem comes more from boost::process than from Python, but all the examples I could find for boost::process are reading std::cout the same way. Is there any thing I should change to have the getline loop run in real-time while Python prints output?

CodePudding user response：

Python streams are (like C streams or C ones) buffered (for performance reasons).

You may want to use some flush method in your Python code.

And your question could be operating system specific. For Linux, be also aware of fsync(2) and termios(3) (and pipe(7) and fifo(7)...). For other operating systems, read their documentation.