698

I'm just trying to streamline one of my classes and have introduced some functionality in the same style as the flyweight design pattern.

However, I'm a bit confused as to why __init__ is always called after __new__. I wasn't expecting this. Can anyone tell me why this is happening and how I can implement this functionality otherwise? (Apart from putting the implementation into the __new__ which feels quite hacky.)

Here's an example:

class A(object):
    _dict = dict()

    def __new__(cls):
        if 'key' in A._dict:
            print "EXISTS"
            return A._dict['key']
        else:
            print "NEW"
            return super(A, cls).__new__(cls)

    def __init__(self):
        print "INIT"
        A._dict['key'] = self
        print ""

a1 = A()
a2 = A()
a3 = A()

Outputs:

NEW
INIT

EXISTS
INIT

EXISTS
INIT

Why?

2
  • 1
    was trying to understand design pattern as well, and first time heard of :flyweight design pattern.. and very good link having example in almost all popular langugages.
    – Emma Y
    May 7, 2020 at 22:10
  • 5
    isn't it a singleton?
    – derek
    Mar 13, 2021 at 19:07

20 Answers 20

727

Use __new__ when you need to control the creation of a new instance.

Use __init__ when you need to control initialization of a new instance.

__new__ is the first step of instance creation. It's called first, and is responsible for returning a new instance of your class.

In contrast, __init__ doesn't return anything; it's only responsible for initializing the instance after it's been created.

In general, you shouldn't need to override __new__ unless you're subclassing an immutable type like str, int, unicode or tuple.

From April 2008 post: When to use __new__ vs. __init__? on mail.python.org.

You should consider that what you are trying to do is usually done with a Factory and that's the best way to do it. Using __new__ is not a good clean solution so please consider the usage of a factory. Here's a good example: ActiveState Fᴀᴄᴛᴏʀʏ ᴘᴀᴛᴛᴇʀɴ Recipe.

5
  • 2
    Ended up using __new__ inside a Factory class, which has become quite clean, so thanks for your input.
    – Dan
    Mar 25, 2009 at 18:09
  • 25
    Sorry, I disagree that the use of __new__ should be strictly limited to the cases stated. I've found it very useful for implementing extensible generic class factories -- see my answer to the question Improper use of __new__ to generate classes in Python? for an example of doing so.
    – martineau
    Feb 3, 2015 at 0:19
  • 11
    I strongly disagree with __new__ being a bad solution here. The factory pattern is necessary in languages that restrict constructors to act as initializers (preventing you from returning an existing object); like most design patterns (especially the plethora of them invented specifically for Java in the early days, due to language inflexibility), it's a way of working around language limitations in a consistent way. Python doesn't have that limitation; you use __new__ for this case, and @classmethod alternate constructors for construction from variant arguments. Jun 22, 2020 at 16:18
  • Link for Factory pattern Recipe does not work.
    – paradox
    Jun 18, 2021 at 14:08
  • I arrived here just after reading a RealPython blog about it. I liked it a lot: realpython.com/factory-method-python
    – Diego-MX
    May 4, 2023 at 14:31
204

__new__ is static class method, while __init__ is instance method. __new__ has to create the instance first, so __init__ can initialize it. Note that __init__ takes self as parameter. Until you create instance there is no self.

Now, I gather, that you're trying to implement singleton pattern in Python. There are a few ways to do that.

Also, as of Python 2.6, you can use class decorators.

def singleton(cls):
    instances = {}
    def getinstance():
        if cls not in instances:
            instances[cls] = cls()
        return instances[cls]
    return getinstance

@singleton
class MyClass:
  ...
13
  • 11
    @Tyler Long, I don't quite understand how does the "@singleton" work? Because the decorator returns a function but MyClass is a class.
    – Alcott
    Sep 19, 2011 at 13:33
  • 13
    Why the dictionary? Since cls will always be the same and you get a fresh dictionary for instance for each singleton you are creating a dictionary with only one item in it. Sep 20, 2011 at 0:20
  • 8
    @Alcott: No opinion needed -- the docs agree with you. Sep 20, 2011 at 0:29
  • 2
    @Alcott. yes the decorator returns a function. but both the class and the function are a callable. I think instances = {} should be a global variable.
    – Tyler Liu
    Nov 12, 2011 at 3:43
  • 5
    @TylerLong Alcott has a good point. This results in the name MyClass being bound to a function that returns instances of the original class. But there's now no way to refer to the original class at all, and MyClass being a function breaks isinstance/issubclass checks, access to class attributes/methods directly as MyClass.something, naming the class in super calls, etc, etc. Whether that's a problem or not depends on the class you're applying it to (and the rest of the program).
    – Ben
    Mar 21, 2012 at 22:48
181

In most well-known OO languages, an expression like SomeClass(arg1, arg2) will allocate a new instance, initialise the instance's attributes, and then return it.

In most well-known OO languages, the "initialise the instance's attributes" part can be customised for each class by defining a constructor, which is basically just a block of code that operates on the new instance (using the arguments provided to the constructor expression) to set up whatever initial conditions are desired. In Python, this corresponds to the class' __init__ method.

Python's __new__ is nothing more and nothing less than similar per-class customisation of the "allocate a new instance" part. This of course allows you to do unusual things such as returning an existing instance rather than allocating a new one. So in Python, we shouldn't really think of this part as necessarily involving allocation; all that we require is that __new__ comes up with a suitable instance from somewhere.

But it's still only half of the job, and there's no way for the Python system to know that sometimes you want to run the other half of the job (__init__) afterwards and sometimes you don't. If you want that behavior, you have to say so explicitly.

Often, you can refactor so you only need __new__, or so you don't need __new__, or so that __init__ behaves differently on an already-initialised object. But if you really want to, Python does actually allow you to redefine "the job", so that SomeClass(arg1, arg2) doesn't necessarily call __new__ followed by __init__. To do this, you need to create a metaclass, and define its __call__ method.

A metaclass is just the class of a class. And a class' __call__ method controls what happens when you call instances of the class. So a metaclass' __call__ method controls what happens when you call a class; i.e. it allows you to redefine the instance-creation mechanism from start to finish. This is the level at which you can most elegantly implement a completely non-standard instance creation process such as the singleton pattern. In fact, with less than 10 lines of code you can implement a Singleton metaclass that then doesn't even require you to futz with __new__ at all, and can turn any otherwise-normal class into a singleton by simply adding __metaclass__ = Singleton!

class Singleton(type):
    def __init__(self, *args, **kwargs):
        super(Singleton, self).__init__(*args, **kwargs)
        self.__instance = None
    def __call__(self, *args, **kwargs):
        if self.__instance is None:
            self.__instance = super(Singleton, self).__call__(*args, **kwargs)
        return self.__instance

However this is probably deeper magic than is really warranted for this situation!

3
  • 3
    Amazing :o simple and doesnt make working within your class annoying like decorating it would do.
    – Break
    Jan 24, 2021 at 7:40
  • 1
    self should be cls in metaclass Singleton Nov 19, 2022 at 12:33
  • @AmauryLeRouxDupeyron That's a stylistic choice, not a requirement (as is the name self in the first place, TBH). In this case I preferred to emphasise that the behaviour we're using here is perfectly ordinary class stuff, because metaclasses are perfectly ordinary classes. If the metaclass was more complicated and there needed to be code here that was run at the class level and thus had the object-level self in scope, then I'd use cls and self to keep the levels straight. But here there's only one level (directly) involved.
    – Ben
    May 27, 2023 at 4:28
32

To quote the documentation:

Typical implementations create a new instance of the class by invoking the superclass's __new__() method using "super(currentclass, cls).__new__(cls[, ...])"with appropriate arguments and then modifying the newly-created instance as necessary before returning it.

...

If __new__() does not return an instance of cls, then the new instance's __init__() method will not be invoked.

__new__() is intended mainly to allow subclasses of immutable types (like int, str, or tuple) to customize instance creation.

3
  • 29
    If __new__() does not return an instance of cls, then the new instance's __init__() method will not be invoked. That's an important point. If you return a different instance, original __init__ is never called. Aug 29, 2012 at 23:32
  • @tgray I realize your answer is from the docs, but I'm curious if you know any use-case of not returning an instance of cls. It seems like when the returned object of __new__ is checked for its class, the method would throw an error rather than allow the failed check to pass silently, because I do not understand why you'd ever want to return anything but an object of the class. Apr 20, 2020 at 17:16
  • @soporific312 I haven't seen any use-cases. This answer discusses some of the reasoning for the design, though they have also not seen any code that leverages that "feature".
    – tgray
    Apr 28, 2020 at 20:21
14

I realize that this question is quite old but I had a similar issue. The following did what I wanted:

class Agent(object):
    _agents = dict()

    def __new__(cls, *p):
        number = p[0]
        if not number in cls._agents:
            cls._agents[number] = object.__new__(cls)
        return cls._agents[number]

    def __init__(self, number):
        self.number = number

    def __eq__(self, rhs):
        return self.number == rhs.number

Agent("a") is Agent("a") == True

I used this page as a resource http://infohost.nmt.edu/tcc/help/pubs/python/web/new-new-method.html

1
  • 11
    Note: __new__ always returns an appropriate object, so __init__ is always called - even if the instance already exists. Oct 3, 2014 at 16:18
13

When __new__ returns instance of the same class, __init__ is run afterwards on returned object. I.e. you can NOT use __new__ to prevent __init__ from being run. Even if you return previously created object from __new__, it will be double (triple, etc...) initialized by __init__ again and again.

Here is the generic approach to Singleton pattern which extends vartec answer above and fixes it:

def SingletonClass(cls):
    class Single(cls):
        __doc__ = cls.__doc__
        _initialized = False
        _instance = None

        def __new__(cls, *args, **kwargs):
            if not cls._instance:
                cls._instance = super(Single, cls).__new__(cls, *args, **kwargs)
            return cls._instance

        def __init__(self, *args, **kwargs):
            if self._initialized:
                return
            super(Single, self).__init__(*args, **kwargs)
            self.__class__._initialized = True  # Its crucial to set this variable on the class!
    return Single

Full story is here.

Another approach, which in fact involves __new__ is to use classmethods:

class Singleton(object):
    __initialized = False

    def __new__(cls, *args, **kwargs):
        if not cls.__initialized:
            cls.__init__(*args, **kwargs)
            cls.__initialized = True
        return cls


class MyClass(Singleton):
    @classmethod
    def __init__(cls, x, y):
        print "init is here"

    @classmethod
    def do(cls):
        print "doing stuff"

Please pay attention, that with this approach you need to decorate ALL of your methods with @classmethod, because you'll never use any real instance of MyClass.

10

I think the simple answer to this question is that, if __new__ returns a value that is the same type as the class, the __init__ function executes, otherwise it won't. In this case your code returns A._dict('key') which is the same class as cls, so __init__ will be executed.

8

An update to @AntonyHatchkins answer, you probably want a separate dictionary of instances for each class of the metatype, meaning that you should have an __init__ method in the metaclass to initialize your class object with that dictionary instead of making it global across all the classes.

class MetaQuasiSingleton(type):
    def __init__(cls, name, bases, attibutes):
        cls._dict = {}

    def __call__(cls, key):
        if key in cls._dict:
            print('EXISTS')
            instance = cls._dict[key]
        else:
            print('NEW')
            instance = super().__call__(key)
            cls._dict[key] = instance
        return instance

class A(metaclass=MetaQuasiSingleton):
    def __init__(self, key):
        print 'INIT'
        self.key = key
        print()

I have gone ahead and updated the original code with an __init__ method and changed the syntax to Python 3 notation (no-arg call to super and metaclass in the class arguments instead of as an attribute).

Either way, the important point here is that your class initializer (__call__ method) will not execute either __new__ or __init__ if the key is found. This is much cleaner than using __new__, which requires you to mark the object if you want to skip the default __init__ step.

7
  • And how do you provide a parameter to the metaclass so that all instances of the class A have the value in their class that you gave to the metaclass? How to get a variable argument into the metaclass __init__?
    – NeilG
    Nov 30, 2023 at 5:02
  • 1
    @NeilG. You can pass kwargs when you call the metaclass (e.g. with a class statement) Nov 30, 2023 at 8:19
  • Ah! There!, @Mad_Physicist! Thanks! You mean like class A(metaclass=MetaQuasiSingleton(name="value")): ... ?
    – NeilG
    Dec 1, 2023 at 7:24
  • 1
    @NeilG. Probably more like class A(metaclass=MetaQuasiSingleton, name="value"): ... . Your proposed syntax calls MetaQuasiSingleton, which might return a class object as the metaclass or crash. Dec 1, 2023 at 14:42
  • 1
    @NeilG. Fair enough. Good luck. I'm sure you'll ask a good question when you need to. Dec 2, 2023 at 3:46
7
class M(type):
    _dict = {}

    def __call__(cls, key):
        if key in cls._dict:
            print 'EXISTS'
            return cls._dict[key]
        else:
            print 'NEW'
            instance = super(M, cls).__call__(key)
            cls._dict[key] = instance
            return instance

class A(object):
    __metaclass__ = M

    def __init__(self, key):
        print 'INIT'
        self.key = key
        print

a1 = A('aaa')
a2 = A('bbb')
a3 = A('aaa')

outputs:

NEW
INIT

NEW
INIT

EXISTS

NB As a side effect M._dict property automatically becomes accessible from A as A._dict so take care not to overwrite it incidentally.

1
  • You are missing an __init__ method that sets cls._dict = {}. You probably don't want a dictionary shared by all classes of this metatype (but +1 for the idea). Jul 18, 2017 at 19:25
6

However, I'm a bit confused as to why __init__ is always called after __new__.

I think the C++ analogy would be useful here:

  1. __new__ simply allocates memory for the object. The instance variables of an object needs memory to hold it, and this is what the step __new__ would do.

  2. __init__ initialize the internal variables of the object to specific values (could be default).

5

__new__ should return a new, blank instance of a class. __init__ is then called to initialise that instance. You're not calling __init__ in the "NEW" case of __new__, so it's being called for you. The code that is calling __new__ doesn't keep track of whether __init__ has been called on a particular instance or not nor should it, because you're doing something very unusual here.

You could add an attribute to the object in the __init__ function to indicate that it's been initialised. Check for the existence of that attribute as the first thing in __init__ and don't proceed any further if it has been.

5

Digging little deeper into that!

The type of a generic class in CPython is type and its base class is Object (Unless you explicitly define another base class like a metaclass). The sequence of low level calls can be found here. The first method called is the type_call which then calls tp_new and then tp_init.

The interesting part here is that tp_new will call the Object's (base class) new method object_new which does a tp_alloc (PyType_GenericAlloc) which allocates the memory for the object :)

At that point the object is created in memory and then the __init__ method gets called. If __init__ is not implemented in your class then the object_init gets called and it does nothing :)

Then type_call just returns the object which binds to your variable.

5

One should look at __init__ as a simple constructor in traditional OO languages. For example, if you are familiar with Java or C++, the constructor is passed a pointer to its own instance implicitly. In the case of Java, it is the this variable. If one were to inspect the byte code generated for Java, one would notice two calls. The first call is to an "new" method, and then next call is to the init method (which is the actual call to the user defined constructor). This two step process enables creation of the actual instance before calling the constructor method of the class which is just another method of that instance.

Now, in the case of Python, __new__ is a added facility that is accessible to the user. Java does not provide that flexibility, due to its typed nature. If a language provided that facility, then the implementor of __new__ could do many things in that method before returning the instance, including creating a totally new instance of a unrelated object in some cases. And, this approach also works out well for especially for immutable types in the case of Python.

5

Referring to this doc:

When subclassing immutable built-in types like numbers and strings, and occasionally in other situations, the static method __new__ comes in handy. __new__ is the first step in instance construction, invoked before __init__.

The __new__ method is called with the class as its first argument; its responsibility is to return a new instance of that class.

Compare this to __init__: __init__ is called with an instance as its first argument, and it doesn't return anything; its responsibility is to initialize the instance.

There are situations where a new instance is created without calling __init__ (for example when the instance is loaded from a pickle). There is no way to create a new instance without calling __new__ (although in some cases you can get away with calling a base class's __new__).

Regarding what you wish to achieve, there also in same doc info about Singleton pattern

class Singleton(object):
        def __new__(cls, *args, **kwds):
            it = cls.__dict__.get("__it__")
            if it is not None:
                return it
            cls.__it__ = it = object.__new__(cls)
            it.init(*args, **kwds)
            return it
        def init(self, *args, **kwds):
            pass

you may also use this implementation from PEP 318, using a decorator

def singleton(cls):
    instances = {}
    def getinstance():
        if cls not in instances:
            instances[cls] = cls()
        return instances[cls]
    return getinstance

@singleton
class MyClass:
...
1
  • 2
    Calling a bastardized form of init from __new__ sounds really hacky. This is what metaclasses are for. Jul 18, 2017 at 19:24
3

Now I've got the same problem, and for some reasons I decided to avoid decorators, factories and metaclasses. I did it like this:

Main file

def _alt(func):
    import functools
    @functools.wraps(func)
    def init(self, *p, **k):
        if hasattr(self, "parent_initialized"):
            return
        else:
            self.parent_initialized = True
            func(self, *p, **k)

    return init


class Parent:
    # Empty dictionary, shouldn't ever be filled with anything else
    parent_cache = {}

    def __new__(cls, n, *args, **kwargs):

        # Checks if object with this ID (n) has been created
        if n in cls.parent_cache:

            # It was, return it
            return cls.parent_cache[n]

        else:

            # Check if it was modified by this function
            if not hasattr(cls, "parent_modified"):
                # Add the attribute
                cls.parent_modified = True
                cls.parent_cache = {}

                # Apply it
                cls.__init__ = _alt(cls.__init__)

            # Get the instance
            obj = super().__new__(cls)

            # Push it to cache
            cls.parent_cache[n] = obj

            # Return it
            return obj

Example classes

class A(Parent):

    def __init__(self, n):
        print("A.__init__", n)


class B(Parent):

    def __init__(self, n):
        print("B.__init__", n)

In use

>>> A(1)
A.__init__ 1  # First A(1) initialized 
<__main__.A object at 0x000001A73A4A2E48>
>>> A(1)      # Returned previous A(1)
<__main__.A object at 0x000001A73A4A2E48>
>>> A(2)
A.__init__ 2  # First A(2) initialized
<__main__.A object at 0x000001A7395D9C88>
>>> B(2)
B.__init__ 2  # B class doesn't collide with A, thanks to separate cache
<__main__.B object at 0x000001A73951B080>
  • Warning: You shouldn't initialize Parent, it will collide with other classes - unless you defined separate cache in each of the children, that's not what we want.
  • Warning: It seems a class with Parent as grandparent behaves weird. [Unverified]

Try it online!

2

The __init__ is called after __new__ so that when you override it in a subclass, your added code will still get called.

If you are trying to subclass a class that already has a __new__, someone unaware of this might start by adapting the __init__ and forwarding the call down to the subclass __init__. This convention of calling __init__ after __new__ helps that work as expected.

The __init__ still needs to allow for any parameters the superclass __new__ needed, but failing to do so will usually create a clear runtime error. And the __new__ should probably explicitly allow for *args and '**kw', to make it clear that extension is OK.

It is generally bad form to have both __new__ and __init__ in the same class at the same level of inheritance, because of the behavior the original poster described.

1

However, I'm a bit confused as to why __init__ is always called after __new__.

Not much of a reason other than that it just is done that way. __new__ doesn't have the responsibility of initializing the class, some other method does (__call__, possibly-- I don't know for sure).

I wasn't expecting this. Can anyone tell me why this is happening and how I implement this functionality otherwise? (apart from putting the implementation into the __new__ which feels quite hacky).

You could have __init__ do nothing if it's already been initialized, or you could write a new metaclass with a new __call__ that only calls __init__ on new instances, and otherwise just returns __new__(...).

1

The simple reason is that the new is used for creating an instance, while init is used for initializing the instance. Before initializing, the instance should be created first. That's why new should be called before init.

1

When instantiating a class, first, __new__() is called to create the instance of a class, then __init__() is called to initialize the instance.

__new__():

Called to create a new instance of class cls. ...

If __new__() is invoked during object construction and it returns an instance of cls, then the new instance’s __init__() method will be invoked like __init__(self[, ...]), ...

__init__():

Called after the instance has been created (by __new__()), ...

Because __new__() and __init__() work together in constructing objects (__new__() to create it, and __init__() to customize it), ...

For example, when instantiating Teacher class, first, __new__() is called to create the instance of Teacher class, then __init__() is called to initialize the instance as shown below:

class Teacher:
    def __init__(self, name):
        self.name = name
        
class Student:
    def __init__(self, name):
        self.name = name

obj = Teacher("John") # Instantiation

print(obj.name)

This is the output:

<class '__main__.Teacher'>
John

And, using __new__() of the instance of Teacher class, we can create the instance of Student class as shown below:

# ...

obj = Teacher("John")
print(type(obj))
print(obj.name)

obj = obj.__new__(Student) # Creates the instance of "Student" class
print(type(obj))

Now, the instance of Student class is created as shown below:

<class '__main__.Teacher'>
<__main__.Teacher object at 0x7f4e3950bf10>
<class '__main__.Student'> # Here

Next, if we try to get the value of name variable from **the instance of Student class as shown below:

obj = Teacher("John")
print(type(obj))
print(obj.name)

obj = obj.__new__(Student)
print(type(obj))
print(obj.name) # Tries to get the value of "name" variable

The error below occurs because the instance of Student class has not been initialized by __init__() yet:

AttributeError: 'Student' object has no attribute 'name'

So, we initialize the instance of Student class as shown below:

obj = Teacher("John") 
print(type(obj))
print(obj.name)

obj = obj.__new__(Student)
print(type(obj))
obj.__init__("Tom") # Initializes the instance of "Student" class
print(obj.name)

Then, we can get the value of name variable from the instance of Student class as shown below:

<class '__main__.Teacher'>
John
<class '__main__.Student'>
Tom # Here
0

People have already detailed the question and answer both use some examples like singleton etc. See the code below:

__instance = None

def __new__(cls):
    if cls.__instance is None:
        cls.__instance = object.__new__(cls)
    return cls.__instance

Execution of above code

I got the above code from this link, it has detailed overview of new vs init. Worth reading!

Not the answer you're looking for? Browse other questions tagged or ask your own question.