Redirect stdout from unix to hadoop without writing file in Unix-CodePudding

I have a process that would download a file, then unzip it a file and then send it to HDFS. We saw a way to optimize it as follows using the command --to-stdout and it worked wonders for us:

gunzip --to-stdout "${FULLNAME}" | hdfs dfs -put - "${path_folder}/${PROCESS}"

The question I have is regarding the stdout:

Is it possible to do something similar with a sql query? That is, throw the result you get and send it to the HDFS with a filename?

We have something like this that generates a file and then send it to the hdfs.

sqlplus -S user/password@dbname << EOF > file_name.txt
    set feedback off trimspool on
    select * from table_name;
    exit;
EOF

hdfs dfs -put file_name.txt ${path_folder}

What I would like to know is if there is a way to take the output from the output without redirecting it to a file but directly, like the example I put of the decompression, send it to the HDFS?

CodePudding user response：

Just redirect as in your example

sqlplus -S user/password@dbname <<EOF | hdfs dfs -put - ${path_folder}
    set feedback off trimspool on
    select * from table_name;
    exit;
EOF

If it can be redirected to a file then it can be redirected to a pipe.

CodePudding user response：

Like your original example, you can use the pipe (|) to redirect your sqlplus command too. Just put the pipe in the same line as the first EOF:

sqlplus -S user/password@dbname << EOF | hdfs dfs -put - ${path_folder}
    set feedback off trimspool on
    select * from table_name;
    exit;
EOF