I am learning overloading in Python 3.X and to better understand the topic, I wrote the following code that works in 3.X but not in 2.X. I expected the below code to fail since I've not defined __call__ for class Test. But to my surprise, it works and prints "constructor called". Demo.

class Test: 
    def __init__(self):
        print("constructor called")
#Test.__getitem__()  #error as expected

Test.__call__() #this works in 3.X(but not in 2.X) and prints "constructor called"! WHY THIS DOESN'T GIVE ERROR in 3.x?

So my question is that how/why exactly does this code work in 3.x but not in 2.x. I mean I want to know the mechanics behind what is going on.

More importantly, why __init__ is being used here when I am using __call__?

CodePudding user response：

In 3.x:

About attribute lookup, type and object

Every time an attribute is looked up on an object, Python follows a process like this:

1. Is it directly a part of the actual data in the object? If so, use that and stop.

2. Is it directly a part of the object's class? If so, hold onto that for step 4.

3. Otherwise, check the object's class for __getattr__ and __getattribute__ overrides, look through base classes in the MRO, etc. (This is a massive simplification, of course.)

4. If something was found in step 2 or 3, check if it has a __get__. If it does, look that up (yes, that means starting over at step 1 for the attribute named __get__ on that object), call it, and use its return value. Otherwise, use what was returned directly.

Functions have a __get__ automatically; it is used to implement method binding. Classes are objects; that's why it's possible to look up attributes in them. That is: the purpose of the class Test: block is to define a data type; the code creates an object named Test which represents the data type that was defined.

But since the Test class is an object, it must be an instance of some class. That class is called type, and has a built-in implementation.

>>> type(Test)
<class 'type'>

Notice that type(Test) is not a function call. Rather, the name type is pre-defined to refer to a class, which every other class created in user code is (by default) an instance of.

In other words, type is the default metaclass: the class of classes.

>>> type
<class 'type'>

One may ask, what class does type belong to? The answer is surprisingly simple - itself:

>>> type(type) is type
True

Since the above examples call type, we conclude that type is callable. To be callable, it must have a __call__ attribute, and it does:

>>> type.__call__
<slot wrapper '__call__' of 'type' objects>

When type is called with a single argument, it looks up the argument's class (roughly equivalent to accessing the __class__ attribute of the argument). When called with three arguments, it creates a new instance of type, i.e., a new class.

How does type work?

Because this is digging right at the core of the language (allocating memory for the object), it's not quite possible to implement this in pure Python, at least for the reference C implementation (and I have no idea what sort of magic is going on in PyPy here). But we can approximately model the type class like so:

def _validate_type(obj, required_type, context):
    if not isinstance(obj, required_type):
        good_name = required_type.__name__
        bad_name = type(obj).__name__
        raise TypeError(f'{context} must be {good_name}, not {bad_name}')

class type:
    def __new__(cls, name_or_obj, *args):
        # __new__ implicitly gets passed an instance of the class, but
        # `type` is its own class, so it will be `type` itself.
        if len(args) == 0: # 1-argument form: check the type of an existing class.
            return obj.__class__
        # otherwise, 3-argument form: create a new class.
        try:
            bases, attrs = args
        except ValueError:
            raise TypeError('type() takes 1 or 3 arguments')
        _validate_type(name, str, 'type.__new__() argument 1')
        _validate_type(bases, tuple, 'type.__new__() argument 2')
        _validate_type(attrs, dict, 'type.__new__() argument 3')

        # This line would not work if we were actually implementing
        # a replacement for `type`, as it would route to `object.__new__(type)`,
        # which is explicitly disallowed. But let's pretend it does...
        result = super().__new__()
        # Now, fill in attributes from the parameters.
        result.__name__ = name_or_obj
        # Assigning to `__bases__` triggers a lot of other internal checks!
        result.__bases__ = bases
        for name, value in attrs.items():
            setattr(result, name, value)
        return result
    del __new__.__get__ # `__new__`s of builtins don't implement this.

    def __call__(self, *args):
        return self.__new__(self, *args)
    # this, however, does have a `__get__`.

What happens (conceptually) when we call the class (Test())?

1. Test() uses function-call syntax, but it's not a function. To figure out what should happen, we translate the call into Test.__class__.__call__(Test). (We use __class__ directly here, because translating the function call using type - asking type to categorize itself - would end up in endless recursion.)

2. Test.__class__ is type, so this becomes type.__call__(Test).

3. type contains a __call__ directly (type is its own class, remember?), so it's used directly - we don't go through the __get__ descriptor. We call the function, with Test as self, and no other arguments. (We have a function now, so we don't need to translate the function call syntax again. We could - given a function func, func.__class__.__call__.__get__(func) gives us an instance of an unnamed builtin "method wrapper" type, which does the same thing as func when called. Repeating the loop on the method wrapper creates a separate method wrapper that still does the same thing.)

4. This attempts the call Test.__new__(Test) (since self was bound to Test). Test.__new__ isn't explicitly defined in Test, but since Test is a class, we don't look in Test's class (type), but instead in Test's base (object).

5. object.__new__(Test) exists, and does magical built-in stuff to allocate memory for a new instance of the Test class, make it possible to assign attributes to that instance (even though Test is a subtype of object, which disallows that), and set its __class__ to Test.

Similarly, when we call type, the same logical chain turns type(Test) into type.__class__.__call__(type, Test) into type.__call__(type, Test), which forwards to type.__new__(type, Test). This time, there is a __new__ attribute directly in type, so this doesn't fall back to looking in object. Instead, with name_or_obj being set to Test, we simply return Test.__class__, i.e., type. And with separate name, bases, attrs arguments, type.__new__ instead creates an instance of type.

Finally: what happens when we call Test.__call__() explicitly?

If there's a __call__ defined in the class, it gets used, since it's found directly. This will fail, however, because there aren't enough arguments: the descriptor protocol isn't used since the attribute was found directly, so self isn't bound, and so that argument is missing.

If there isn't a __call__ method defined, then we look in Test's class, i.e., type. There's a __call__ there, so the rest proceeds like steps 3-5 in the previous section.

CodePudding user response：

In Python 3.x, every class is implicitely a child of the builtin class object. And at least in the CPython implementation, the object class has a __call__ method which is defined in its metaclass type.

That means that Test.__call__() is exactly the same as Test() and will return a new Test object, calling your custom __init__ method.

In Python 2.x classes are by default old-style classes and are not child of object. Because of that __call__ is not defined. You can get the same behaviour in Python 2.x by using new style classes, meaning by making an explicit inheritance on object:

# Python 2 new style class
class Test(object):
    ...