paramiko module is not catching the output-CodePudding

I'm trying to get Kafka lags using paramiko module to catch sum of the kafka lag but the output always giving zero using python paramiko ssh module, while manually same command giving the correct result.

    # check list of consumers
        stdin, stdout, stderr = ssh.exec_command("/home/prduser/kafka_2.12-2.1.0/bin/kafka-consumer-groups.sh --bootstrap-server localhost:19092,localhost:29092,localhost:39092 --list")
        list_consumers = stdout.readlines()
        print(list_consumers)
# check lag for each consumer
        for consumer in list_consumers:
            print(consumer)
            # | awk '{sum  = $5} END {print sum}'
            stdin, stdout, stderr = ssh.exec_command("/home/prduser/kafka_2.12-2.1.0/bin/kafka-consumer-groups.sh --bootstrap-server localhost:19092,localhost:29092,localhost:39092 --describe --group {{0}} | awk '{{sum =$5}} END {{print sum}}' ".format(consumer))
            consumer_lag_out = stdout.readlines()
            print(consumer_lag_out)

sample output

manual shell command :

CodePudding user response：

You have double curly-brackets in the second ssh command. Change from --group {{0}} | awk to --group {0} | awk
~~The full line should be~~ (EDIT: see followup):

stdin, stdout, stderr = ssh.exec_command("/home/prduser/kafka_2.12-2.1.0/bin/kafka-consumer-groups.sh --bootstrap-server localhost:19092,localhost:29092,localhost:39092 --describe --group {0} | awk '{{sum =$5}} END {{print sum}}' ".format(consumer))

This way, {0} is replaced by format with consumer (which you redacted in the image, but looks like it should start with adapter.). Otherwise, consumer is ignored, and the actual command sent is:

/home/prduser/kafka_2.12-2.1.0/bin/kafka-consumer-groups.sh --bootstrap-server localhost:19092,localhost:29092,localhost:39092 --describe --group {0} | awk '{sum =$5} END {print sum}'

When format encounters double-curly-brackets, it simply strips the outer ones. It will not error in this case. For example:

>>> consumer = "adapter"
>>> print("value = {{0}}, some brackets = {{inside}}".format(consumer))
value = {0}, some brackets = {inside}
>>> print("value = {0}, some brackets = {{inside}}".format(consumer))
value = adapter, some brackets = {inside}

Following up with further comments, newlines should be stripped from original output. While the sample image is redacted so much so it hard to read, a simple POC can illustrate the situation:

>>> lines = ssh.exec_command("ls / | grep etc")[1].readlines()
>>> lines
['etc\n']
>>> consumer = lines[0]
>>> _, stdout, stderr = ssh.exec_command("ls /{0} | grep init.d".format(consumer))
>>> len(stdout.readlines())
106
>>> # It is unlikely we have 106 files with 'init.d' under /etc/ .. Let's check for errors:
>>> print(stderr.readlines())
["bash: -c: line 2: syntax error near unexpected token `|'\n", "bash: -c: line 2: ` | grep init.d'\n"]
>>> # Now let's try without newlines:
>>> _, stdout, stderr = ssh.exec_command("ls /{0} | grep init.d".format(consumer.strip()))
>>> print(stdout.readlines())
['init.d\n']
>>> # That's more like it.

Therefore, the full line should be:

stdin, stdout, stderr = ssh.exec_command("/home/prduser/kafka_2.12-2.1.0/bin/kafka-consumer-groups.sh --bootstrap-server localhost:19092,localhost:29092,localhost:39092 --describe --group {0} | awk '{{sum =$5}} END {{print sum}}' ".format(consumer.strip()))