What specific syntax must be changed in the code below in order for the multi-line contents of the
$MY_SECRETSenvironment variable to be successfully written into theC:\\Users\\runneradmin\\somedir\\mykeys.yamlfile on a Windows runner in the GitHub workflow whose code is given below?
PROBLEM DEFINITION:
The echo "$MY_SECRETS" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml command is only printing the string literal MY_SECRETS into the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file instead of printing the multi-line contents of the MY_SECRETS variable.
We confirmed that this same echo command does successfully print the same multi-line secret in an ubuntu-latest runner, and we manually validated the correct contents of the secrets.LIST_OF_SECRETS environment variable. ... This problem seems entirely isolated to either the windows command syntax, or perhaps to the windows configuration of the GitHub windows-latest runner, either of which should be fixable by changing the workflow code below.
EXPECTED RESULT:
The multi-line secret should be printed into the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file.
The resulting printout of the contents of the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file should look like:
***
***
***
***
LOGS THAT DEMONSTRATE THE FAILURE:
The result of running myapp.py in the GitHub Actions log is:
ccc item is: $MY_SECRETS
As you can see, the string literal $MY_SECRETS is being wrongly printed out instead of the 4 *** secret lines.
WORKFLOW CODE:
The minimal code for the workflow to reproduce this problem is as follows:
name: write-secrets-to-file
on:
push:
branches:
- dev
jobs:
write-the-secrets-windows:
runs-on: windows-latest
steps:
- uses: actions/checkout@v3
- shell: python
name: Configure agent
env:
MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
run: |
import subprocess
import pathlib
pathlib.Path("C:\\Users\\runneradmin\\somedir\\").mkdir(parents=True, exist_ok=True)
print('About to: echo "$MY_SECRETS" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml')
output = subprocess.getoutput('echo "$MY_SECRETS" > C:\\Users\\runneradmin\\somedir\\mykeys.yaml')
print(output)
os.chdir('D:\\a\\myRepoName\\')
mycmd = "python myRepoName\\myapp.py"
p = subprocess.Popen(mycmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while(True):
# returns None while subprocess is running
retcode = p.poll()
line = p.stdout.readline()
print(line)
if retcode is not None:
break
MINIMAL APP CODE:
Then the minimal myapp.py program that demonstrates what was actually written into the C:\\Users\\runneradmin\\somedir\\mykeys.yaml file is:
with open('C:\\Users\\runneradmin\\somedir\\mykeys.yaml') as file:
for item in file:
print('ccc item is: ', str(item))
STRUCTURE OF MULTI-LINE SECRET:
The structure of the multi-line secret contained in the secrets.LIST_OF_SECRETS environment variable is:
var1:value1
var2:value2
var3:value3
var4:value4
These 4 lines should be what gets printed out when myapp.py is run by the workflow, though the print for each line should look like *** because each line is a secret.
@JAAAY'S SUGGESTION:
Here is how we have applied @JAAAY's suggestion to the requirements of this OP:
Repo file structure contains only 2 files as shown:
.github/
workflows/
test.yml
main.py
test.yml contains:
name: write-secrets-to-file
on:
push:
branches:
- main
jobs:
write-the-secrets-windows:
runs-on: windows-latest
steps:
- uses: actions/checkout@v3
- shell: python
name: Configure agent
env:
MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
run: |
import base64, subprocess, sys, os, pathlib
pathlib.Path("C:\\Users\\runneradmin\\somedir\\").mkdir(parents=True, exist_ok=True)
def powershell(cmd, input=None):
cmd64 = base64.encodebytes(cmd.encode('utf-16-le')).decode('ascii').strip()
stdin = None if input is None else subprocess.PIPE
process = subprocess.Popen(["powershell.exe", "-NonInteractive", "-EncodedCommand", cmd64], stdin=stdin, stdout=subprocess.PIPE)
if input is not None:
input = input.encode(sys.stdout.encoding)
output, stderr = process.communicate(input)
output = output.decode(sys.stdout.encoding).replace('\r\n', '\n')
return output
secrets = os.environ["MY_SECRETS"]
command = r"""$secrets = @'
{}
'@
$secrets | Out-File -FilePath C:\\Users\\runneradmin\\somedir\\mykeys.yaml""".format(secrets)
print('About to echo secrets to file.')
print(powershell(command))
output = subprocess.run(["powershell.exe", "-Command", ], capture_output=True, shell=True)
print(output)
output = subprocess.getoutput(["powershell.exe", "-Command", "Get-Content -Path C:\\Users\\runneradmin\\somedir\\mykeys.yaml"])
print(output)
mycmd = "python main.py"
p = subprocess.Popen(mycmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while(True):
# returns None while subprocess is running
retcode = p.poll()
line = p.stdout.readline()
print(line)
if retcode is not None:
break
main.py contents are:
with open('C:\\Users\\runneradmin\\somedir\\mykeys.yaml') as file:
for item in file:
print('ccc item is: ', str(item))
if "key1" in item:
print("Found key1")
Repository-level multi-line secret named LIST_OF_SECRETS contains:
key1:val1
key2:val2
And the seemingly relevant portion of the logs look like:
b'ccc item is: \xff\xfek\x00e\x00y\x001\x00:\x00v\x00a\x00l\x001\x00\r\n'
b'\r\n'
b'ccc item is: \x00k\x00e\x00y\x002\x00:\x00v\x00a\x00l\x002\x00\r\n'
b'\r\n'
b'ccc item is: \x00\r\n'
b'\r\n'
b'ccc item is: \x00\r\n'
b''
b''
b''
...many more lines
Do you see that it failed to find key1 and also that the secret lines are encoded in addition to being wrapped in a byte array?
These results are very different behavior than we are getting in other environments, including a Windows 10 laptop on prem, RHEL8 agents in a cloud provider using a different pipeline tool, and GitHub ubuntu-latest runners, all of which work perfectly without these problems.
CodePudding user response：
You need to use yaml library:
import yaml
data = {'MY_SECRETS':'''
var1:value1
var2:value2
var3:value3
var4:value4
'''}#add your secret
with open('file.yaml', 'w') as outfile: # Your file
yaml.dump(data, outfile, default_flow_style=False)
CodePudding user response：
I tried the following code and it worked fine :
LIST_OF_SECRETS
key1:val1
key2:val2
Github action (test.yml)
name: write-secrets-to-file
on:
push:
branches:
- main
jobs:
write-the-secrets-windows:
runs-on: windows-latest
steps:
- uses: actions/checkout@v3
- shell: python
name: Configure agent
env:
MY_SECRETS: ${{ secrets.LIST_OF_SECRETS }}
run: |
import base64, subprocess, sys, os
def powershell(cmd, input=None):
cmd64 = base64.encodebytes(cmd.encode('utf-16-le')).decode('ascii').strip()
stdin = None if input is None else subprocess.PIPE
process = subprocess.Popen(["powershell.exe", "-NonInteractive", "-EncodedCommand", cmd64], stdin=stdin, stdout=subprocess.PIPE)
if input is not None:
input = input.encode(sys.stdout.encoding)
output, stderr = process.communicate(input)
output = output.decode(sys.stdout.encoding).replace('\r\n', '\n')
return output
secrets = os.environ["MY_SECRETS"]
command = r"""$secrets = @'
{}
'@
$secrets | Out-File -FilePath .\mykeys.yaml""".format(secrets)
print('About to echo secrets to file.')
print(powershell(command))
output = subprocess.run(["powershell.exe", "-Command", ], capture_output=True, shell=True)
print(output)
output = subprocess.getoutput(["powershell.exe", "-Command", "Get-Content -Path .\mykeys.yaml"])
print(output)
Output
***
***
As you also mention in the question, Github will obfuscate any printed value containing the secrets with ***
EDIT : Updated the code to work with multiple line secrets. This answer was highly influenced by this one