What is the encoding of CompletedProcess.stdout coming from Powershell/Windows in Python?-CodePudding

I am getting this output from a ping request started from Python with subprocess.run():

>>> process.stdout
b"\r\nEnvoi d'une requ\x88te 'ping' sur www.google.fr [142.250.179.195] avec 32 octets de donn\x82es\xff:\r\nR\x82ponse de 142.250.179.195\xff: octets=32 temps=39 ms TTL=110\r\nR\x82ponse de 142.250.179.195\xff: octets=32 temps=46 ms TTL=110\r\nR\x82ponse de 142.250.179.195\xff: octets=32 temps=37 ms TTL=110\r\n\r\nStatistiques Ping pour 142.250.179.195:\r\n    Paquets\xff: envoy\x82s = 3, re\x87us = 3, perdus = 0 (perte 0%),\r\nDur\x82e approximative des boucles en millisecondes :\r\n    Minimum = 37ms, Maximum = 46ms, Moyenne = 40ms\r\n"

I run this script from Pycharm that runs Powershell on a Windows 10 21H2 in French language. So I expect encoding Windows-1252. Which is also the guess of chardet:

>>> chardet.detect(process.stdout)
{'encoding': 'Windows-1252', 'confidence': 0.73, 'language': ''}

However decoding this with Windows-1252 does not look very right:

>>> process.stdout.decode("windows-1252")
"\r\nEnvoi d'une requˆte 'ping' sur www.google.fr [142.250.179.195] avec 32 octets de donn‚esÿ:\r\nR‚ponse de 142.250.179.195ÿ: octets=32 temps=39 ms TTL=110\r\nR‚ponse de 142.250.179.195ÿ: octets=32 temps=46 ms TTL=110\r\nR‚ponse de 142.250.179.195ÿ: octets=32 temps=37 ms TTL=110\r\n\r\nStatistiques Ping pour 142.250.179.195:\r\n    Paquetsÿ: envoy‚s = 3, re‡us = 3, perdus = 0 (perte 0%),\r\nDur‚e approximative des boucles en millisecondes :\r\n    Minimum = 37ms, Maximum = 46ms, Moyenne = 40ms\r\n"

0x88 should be ê and 0x82 should be è

CodePudding user response：

Console applications on Windows typically use the console's active code page to encode their output, which by default is the system's legacy OEM code page (e.g., CP437 on US-English systems), not the legacy ANSI code page used by GUI applications (e.g, Windows-1252):

You can use the following code to determine the console's active code page and decode based on it:

import ctypes
import subprocess

# Get the console's active code page, as an integer.
oemCP = ctypes.windll.kernel32.GetConsoleOutputCP()

process = subprocess.run('ping.exe', capture_output=True)

# Decode based on the console's active code page.
print(process.stdout.decode("cp"   str(oemCP)))

A note re detecting the coding:

The prevalent single-byte code pages that are used as the OEM and ANSI code pages do not use BOMs, and any byte value is also a valid character.
This ultimately makes any attempt to detect what an unknown encoding is guesswork - though the probability of guessing right can be improved with sophisticated linguistic analysis.
I don't know what approach chardet.detect() uses, but in this case it guessed incorrectly; that it guessed can be inferred from the presence of a confidence value.

0x88 should be ê and 0x82 should be è

This actually applies to CP437, not to Windows-1252, as the following PowerShell code demonstrates:

PS> [System.Text.Encoding]::GetEncoding(437).GetString([byte[]] (0x88, 0x82))

êé