I'm the author of pythonizer and I'm trying to convert perl-style function templates to python. When I generate what I think is the equivalent code, the value of the loop variable is the last value instead of the value that it was when the function template comes into existence. Any ideas on code to capture the proper loop variable values? For example:

# test function templates per the perlref documentation
use Carp::Assert;

sub _colors {
    return qw(red blue green yellow orange purple white black);
}

for my $name (_colors()) {
    no strict 'refs';
    *$name = sub { "<FONT COLOR='$name'>@_</FONT>" };
}

assert(red("careful") eq "<FONT COLOR='red'>careful</FONT>");
assert(green("light") eq "<FONT COLOR='green'>light</FONT>");

print "$0 - test passed!\n";

Gets translated to:

#!/usr/bin/env python3
# Generated by "pythonizer -v0 test_function_templates.pl" v0.978 run by snoopyjc on Thu May 19 10:49:12 2022
# Implied pythonizer options: -m
# test function templates per the perlref documentation
import builtins, perllib, sys

_str = lambda s: "" if s is None else str(s)
perllib.init_package("main")
# SKIPPED: use Carp::Assert;


def _colors(*_args):
    return "red blue green yellow orange purple white black".split()


_args = perllib.Array()
builtins.__PACKAGE__ = "main"
for name in _colors():
    pass  # SKIPPED:     no strict 'refs';

    def _f10(*_args):
        #nonlocal name
        return f"<FONT COLOR='{name}'>{perllib.LIST_SEPARATOR.join(map(_str,_args))}</FONT>"

    globals()[name] = _f10


print(red("careful"))
assert _str(red("careful")) == "<FONT COLOR='red'>careful</FONT>"
assert _str(green("light")) == "<FONT COLOR='green'>light</FONT>"

perllib.perl_print(f"{sys.argv[0]} - test passed!")

(I commented out the nonlocal because python complains it's a syntax error, and added the print statement). The added print statement writes out <FONT COLOR='black'>careful</FONT> instead of the proper <FONT COLOR='red'>careful</FONT>

How do I get it to capture the red value of the loop counter when the function red is generated?

CodePudding user response：

The function _f10 does not bind parameters name correctly.

So the name used in the function depends on the last result loop. name is in global scope when the loop is run, so you still get result, just not was is expected.

You should bind the name into the function, by adding the name argument to it and resolve partially the function, like this:

from functools import partial

def _f10(name, *_args):
    #nonlocal name
    return f"<FONT COLOR='{name}'>{perllib.LIST_SEPARATOR.join(map(_str,_args))}</FONT>"

globals()[name] = partial(_f10, name)

So each global is bound to a slightly different function (first argument is bound).

⇒ Finding the identifiers to bind into the function from the perl code could be difficult… You could still try to bind all local variables using locals() in a way, but that would be a bit tricky.

CodePudding user response：

As mentioned in other answers, the problem is name remains a reference to the variable rather than becoming a string literal as intended.

Another way to achieve this is to use a templated string as code, then execute the resulting string. A benefit to this approach is the future reader can validate exactly what's being executed by printing the resulting templated string.

Taking a step back and caring for the domain of the problem, I have created two solutions. First is what I think it would look like if I were to manually translate the code, second is taking your literal example and trying to make it work.

Manually Transcribed

Here I attempt to manually transcribe the Perl code to Python (knowing very little about Perl). I think this illustrates the closest 1:1 behavior to the original while attempting to maintain the spirit of how that's being accomplished in Perl.

This is my recommended solution as it results in a very elegant 1:1 ratio of lines of code accomplishing exactly the same work as the original Perl code per line (if you can excuse what would typically be seen as poor style in the Python paradigm)

import sys
from string import Template

def _colors(*_args):
    return "red blue green yellow orange purple white black".split()

for name in _colors():
    pass  # SKIPPED:     no strict 'refs';
    eval(compile(Template('''global $name\n$name = lambda x: f"<FONT COLOR='$name'>{x}</FONT>"''').substitute({'name':name}),'<string>', 'exec'))

assert red("careful") == "<FONT COLOR='red'>careful</FONT>"
assert green("light") == "<FONT COLOR='green'>light</FONT>"

print(f"{sys.argv[0]} - test passed!")

Updated OP Code

Here I attempt to replicate the literal code provided by OP to make that code work with as little modification to that as possible. (I prefer the manually transcribed version)

Note that I'm unable to test this as I do not have perllib installed.

#!/usr/bin/env python3
# Generated by "pythonizer -v0 test_function_templates.pl" v0.978 run by snoopyjc on Thu May 19 10:49:12 2022
# Implied pythonizer options: -m
# test function templates per the perlref documentation
import builtins, perllib, sys
from string import Template

_str = lambda s: "" if s is None else str(s)
perllib.init_package("main")
# SKIPPED: use Carp::Assert;


def _colors(*_args):
    return "red blue green yellow orange purple white black".split()


_args = perllib.Array()
builtins.__PACKAGE__ = "main"
for name in _colors():
    pass  # SKIPPED:     no strict 'refs';

    eval(compile(Template('''
def _f10(*_args):
    #nonlocal $name
    return f"<FONT COLOR='{$name}'>{perllib.LIST_SEPARATOR.join(map(_str,_args))}</FONT>"
globals()[$name] = _f10
''').substitute({'name':name}),'<string>', 'exec'))


print(red("careful"))
assert _str(red("careful")) == "<FONT COLOR='red'>careful</FONT>"
assert _str(green("light")) == "<FONT COLOR='green'>light</FONT>"

perllib.perl_print(f"{sys.argv[0]} - test passed!")

Additional Considerations

Security - Remote Code Execution

Typically the use of Eval/Exec/Compile should be done with great caution as any input values (in this case, colors) could be an arbitrary code block. That's very bad if the input values can be controlled by the end user in any way. That said this is presumably also true for Perl, and does not matter the solution you choose.

So if for any reason the input data is untrusted, you would want to do some more source validation etc. Normally I would be highly concerned, but IMO I think code execution may be an acceptable risk when translating code from one language to the other. You are probably already executing the original code to validate its functionality, so I'm making the assumption the source has 100% trust.

Clobbering

I'm sure you may be aware, but it is worth noting that there are some serious problems with auto-generating global objects like this. You should probably test what happens when you attempt to define a global with an existing keyword name causing a namespace collision. My expectation is that in Python it will generate an error, and in Perl it will work like a monkeypatch works in Python. You may want to consider adding a prefix to all global variables defined in this way, or make a decision on if this type of behavior of redefining keywords/builtins/existing names is allowable.