References and Copies

This lecture covers the following contents:

• Objects and references
• Copies

Objects and references

When I defined the term "object", I offered a few different definitions:

1. An instance of a class

2. A thing with both state and behavior (e.g., an instance of a class that has both attributes and methods)

3. Any value (or, more rigorously, anything that stores a value)

Clearly, these definitions are not all identical, but they're all valid depending on the context. In the Python language specification, the third definition is used: an object is simply any value, regardless of the type of that value. This means that, from the perspective of the Python language specification, even integer values such as 5 and -100 are objects.

But what's perhaps even more interesting is how Python defines and deals with variables. In many programming languages, a variable is a named location in the computer's memory in which a value of a certain type can be stored. In such languages, mechanisms that serve to modify the values of variables, such as assignment operators, simply tell the computer to go to the location represented by the variable and modify the value contained within it. But the story is very different in Python. In Python, each and every variable is actually a named reference to an object (under the aforementioned definition of "object"; i.e., a value). And in Python, variables do not have fixed locations—objects do. Mechanisms such as the assignment operator, then, do not modify the value (object) stored at a fixed location. Rather, they modify the variable (reference) to refer to a new location.

Formally, a reference is just something that refers, or "points", to a certain object. But technically, references usually work by way of memory addresses. As a program executes, it needs a place to store its data. In most cases, that place is the computer's memory (usually random access memory, or RAM for short). A memory address is just a number that represents a certain place in the computer's memory. A reference is essentially a memory address: it refers to a certain location in the computer's memory, at which a certain object is stored.

Consider house addresses as an analogy. A house address is a string of text that represents the location of a house. If you were to go to the that location, you'd (hopefully) find the house itself. Similarly, a memory address is a string of bits (i.e., a number) that represents the location of an object. If you were to go to that location in the computer's memory, you'd find the object itself.

Variables are named references, and references are essentially memory addresses. y = x does not copy an object—it copies a reference, meaning a memory address. The result is that y and x are two separate copies of the same memory address. If you were to go to the location in memory referenced by x, that's the exact same location as the one referenced by y. Hence, they both "refer" to the same object.

Take the following program, for example:

x = 5
print(x)
x = 6
print(x)
x -= 4
print(x)
y = x
print(y)

We might describe line 1 as creating a variable x and storing the value 5 inside it. But in Python, that's not how things actually work. Rather, line 1 creates an integer object to store the value 5, and then it creates a variable x that refers to that object (i.e., stores the memory address of that object). Similarly, line 2 does not print the value stored within x. Rather, it prints the value of the object that the variable x currently refers to.

I'll explain why this matters in a moment. For now, let's just trace the rest of the above code to make sure we understand what's going on. Line 3 does not modify the value stored within x. Rather, it changes x to refer to a new, completely separate object: 6. Line 4 then prints the value of that object. Line 5 is a bit more interesting. Recall that x -= 4 is equivalent to x = x - 4. What line 5 does, then, is 1) retrieve the value of the object that x currently refers to (i.e., 6); 2) subtract 4 from that value, computing the result (2) and storing it in a new object; and 3) change x to refer to that new object. Line 6, again, simply prints the value of that object.

Line 7 is particularly interesting, and it illustrates the reason that all this matters: because no new values are written nor computed in line 7, it does not create any new objects whatsoever. Rather, all it actually does is create a new variable y and set it up to refer to the same object that x currently refers to. That's to say, by the time the program reaches line 8, you might think that there are two integers that both store the value 2, but you'd be wrong. Rather, there is a single integer object storing the value 2, and there are two separate variables—x and y—that both refer to that single integer.

I can prove that this is the case. In a Python program, every object that's ever created has a unique identifier, and an object's identifier can be retrieved via the id() function. It accepts a single argument, being the object whose identifier you'd like to retrieve, and it returns the identifier of that object. If you pass a variable as the argument, it returns the value of the object that that variable refers to.

Object identifiers are just large integers that uniquely identify objects, so, like other integers, they can be printed to the terminal. Let's update the above code, making it print the identifiers of the integer objects rather than the values of the integer objects:

main.py

def main() -> None:
    x = 5
    print(id(x))
    x = 6
    print(id(x))
    x -= 4
    print(id(x))
    y = x
    print(id(y))

if __name__ == '__main__':
    main()

Here's one possible example run of the above program:

(env) $ python main.py 
140459534906288
140459534906320
140459534906192
140459534906192

There are a couple important things to notice here. For one, whenever the value of x changes, its identifier changes as well. Indeed, when its value is 5, its identifier is 140459534906288, but when its value is 6, its identifier is 140459534906320 (and its identifier changes again when its value changes to 2). This is because, as I explained, x = 6 does not change the value "stored within x". Rather, it creates a new object to store the value 6, and it changes x to refer to that object (instead of referring to the object storing the value 5). After that point, id(x) retrieves the identifier of the object storing the value 6 instead of the object storing the value 5. These are two different objects, so they have two different identifiers. (And, again, this happens a second time due to x -= 4).

Second, notice that the last two printed identifiers are the same. The second to last identifier is that of x when its value is 2, and the last identifier is that of y after assigning y = x. This is because, as I explained, y = x does not store the value of x inside y. Rather, it creates y and tells it to refer to the object that x currently refers to. Hence, x and y refer to the same object, so id(x) and id(y) return the same identifiers.

An important conclusion can be drawn from all this: all primitive-type objects are immutable (constant) in Python. Indeed, the values of primitive objects cannot possibly be changed. x = 6, for example, does not change the value of the underlying object from 5 to 6, nor does x -= 4 change the value of the underlying object from 6 to 2. Rather, each of these operations create new objects to store the newly computed values, and then they update the variable on the left (x) to refer to those new objects.

(Note for the curious reader: The Python interpreter is actually allowed to "reuse" existing objects for primitive-type expressions that resolve to the same value as said object. For example, if, at the end of the program, I wrote z = 2 followed by print(id(z)), it, too, would print the same identifier as print(id(x)) and print(id(y)). That's to say, the Python interpreter is allowed to reuse the exact same object for z as it does for x and y even though z was not directly assigned the value of x nor y. It's generally only allowed to do this for primitive-type objects, though, because such objects are immutable).

Copies

The exact distinction between "changing the value stored in a variable" versus "changing a variable to refer to a different object" might not seem very important in the context of primitives such as integers, but it's critical when we're dealing with more complex types such as classes (even POD types). Consider the following program:

main.py

# A Person POD type. It should probably be in a separate person.py,
# but this is just a demonstration
class Person:
    name: str
    age: int

def main() -> None:
    x = Person()
    x.name = 'Joe'
    x.age = 42

    y = x
    y.name = 'Sally'

    print(x.name) # Question: What does this print?

if __name__ == '__main__':
    main()

Note the important question in the comment on Line 15 ("What does this print?"). What do you think?

Before you can answer this question, you need to know how the dot operator fits into all of this: when the dot operator appears to the right of a variable name, it reaches inside the object that the variable refers to, accessing the attribute or method whose name is on the right of the dot operator. The important point here is that the dot operator does not reach inside variables, but rather objects. This means that attributes and methods belong to objects—not the variables that reference those objects. Again, this might seem pedantic, but the difference matters.

Now let's answer the question. Line 8 creates a new object to store the newly created Person instance, and then it creates the variable x, setting it up to refer to that object. Line 9 creates a new string object to store the value 'Joe'. Then—and this is where things get interesting—it reaches inside the object that x refers to (i.e., the Person instance created on Line 8) to access its name attribute, and it sets up that attribute (variable) to refer to the aforementioned string object. Something similar happens on Line 10; it creates an integer object to store the value 10, and then it reaches inside the object that x refers to, accessing its age attribute and setting it up to refer to said integer object.

Line 12 creates a new variable, y, and sets it up to refer to the same object that x currently refers to—the Person instance created on Line 8. Line 13 is where things get interesting: similar to line 9, it creates a new string object to store the value 'Sally', and then it reaches inside the object that y refers to, accessing its name attribute and changing it to refer to said newly created string object.

Finally, Line 15 reaches inside the object that x refers to, accessing its name attribute and printing the value of the object that it refers to.

The key here is in recognizing two facts: 1) x and y are two separate variables, but they refer to the same underlying object (e.g., notice how I described Line 12 above); and 2) the dot operator simply reaches inside the object (specifically class instance) that the variable on the left refers to (e.g., y.name reaches inside the object that y refers to, accessing its name attribute), accessing its attributes and / or methods. Since x and y refer to the same object, when Line 13 modifies the name attribute of the object that y refers to, that also modifies the name attribute of the object that x refers to (because x and y refer to the exact same object).

This means that modifying y.name = 'Sally' is equivalent to modifying x.name = 'Sally'. Line 15, then, prints 'Sally':

(env) $ python main.py 
Sally

This might surprise you. In the past, you might have heard (and I might have even told you) that y = x copies the value stored in x into the newly created variable y. But that isn't actually the case. If y = x truly copied the value (object) stored in x, then you'd have two independent values: one stored in x, and the other stored in y (created as a copy of the original). Modifying y.name, then, would not modify x.name, because these would be two separate attributes of two separate objects. No, y = x does not copy a value. Rather, it copies a reference. You then have two references, x and y, that refer to the same value (object), so x.name and y.name are the same attribute of the same object.

A similar thing happens with lists:

x = [1, 2, 3]
y = x
y[0] = 100
print(x) # Prints [100, 2, 3]

This is for the same reason as before. y = x does not create a new list. It just sets up y to refer to the same list (object) as the one that x currently refers to. y[0] = 100 then reaches inside that list (object), modifying its first element to refer to the a new integer object storing the value 100.

Suppose you want to copy an object, then. How do you do that, given that y = x only actually copies a reference?

Well, there are at least two functions provided by the Python standard library that can be used to copy objects: copy.copy(), and copy.deepcopy(). Using either of them requires importing the copy package (or importing the respective function from the copy package, such as from copy import copy, or from copy import deepcopy). To call one of these functions, simply pass in an object (or a variable that refers to an object) as an argument, and it will construct and return a copy of that object. When I say "a copy of that object", I mean that it will return an entirely new object that stores an "equivalent" value of that of the object being copied.

In the context of this program, either one will work. But you usually want to use deepcopy, so let's do that:

main.py

from copy import deepcopy

# A Person POD type. It should probably be in a separate person.py,
# but this is just a demonstration
class Person:
    name: str
    age: int

def main() -> None:
    x = Person()
    x.name = 'Joe'
    x.age = 42

    y = deepcopy(x)
    y.name = 'Sally'

    print(x.name) # Question: What does this print?

if __name__ == '__main__':
    main()

Now, y is defined to refer to a copy of the object that x refers to. This means that x and y refer to separate objects with their own attributes, so modifying y.name does not modify x.name (or vice-versa). Line 17, then, prints 'Joe':

(env) $ python main.py 
Joe

In this case, copy() and deepcopy() essentially do the same thing, but that's not always the case. The difference between these two functions is that copy.copy() produces a shallow copy, whereas copy.deepcopy() produces a deep copy. To create a shallow copy of an object means to create a new object whose attributes refer to the exact same sub-objects as the attributes of the original object. In contrast, to create a deep copy of an object means to create a new object whose attributes in turn refer to deep copies of the sub-objects that the original object's attributes refer to (this, this is a recursive definition).

As an example, suppose x refers to an object with an attribute named y, which in turn refers to an object with an attribute named z. Then suppose you write a = copy(x). a will then refer to a new object that's a shallow copy of the object that x refers to. This means that a and x refer to two different objects. However, because it's only a shallow copy, a.y and x.y refer to the same underlying object. Similarly, a.y.z and x.y.z refer to the same underlying object. If you then proceeded to modify a.y.z, that would also modify x.y.z because a.y and x.y refer to the same object (so modifying the attributes of one also modifies the attributes of the other). However, if you modified a.y directly (e.g., a.y = something_else), that would not modify x.y because a and x refer to separate objects (each with their own separate attributes).

Now let's alter that example to consider deep copies instead: if you write a = deepcopy(x) instead of a = copy(x), then a will refer to a new object that's a deep copy of the object that x refers to. Again, this means that a and x refer to separate objects. However, the case of a deep copy, a.y and x.y would also refer to separate objects. Similarly, a.y.z and x.y.z would also refer to separate objects. And so on. Modifying a, or any of its attributes, or any of its attributes' attributes, or so one, will never result in any modifications to x or its attributes (or its attributes' attributes, and so on).

Put simply, a deep copy is a complete, independent copy of the original object and everything inside it. In contrast, a shallow copy is a copy of the original object, but the things inside it are not copied. Rather, the copy's attributes refer to the same sub-objects as the attributes of the original, copied object (e.g., a.y and x.y refer to the same object, even though a and x refer to separate objects).

This is a complicated concept, so here's a more complete example illustrating shallow copies:

shallowcopy.py

from copy import copy # Used for shallow copies

class Person:
    name: str

class House:
    owner: Person

def main() -> None:
    joe = Person()
    joe.name = 'Joe'

    house = House()
    house.owner = joe

    # house2 is a SHALLOW copy of house
    house2 = copy(house)

    # house and house2 refer to separate objects (these print different
    # identifiers)
    print(id(house))
    print(id(house2))
    print()

    # However, house.owner and house2.owner refer to the SAME
    # object (these print the same identifier):
    print(id(house.owner))
    print(id(house2.owner))
    print()

    # This means that modifying house2.owner.name will ALSO modify
    # house.owner.name (because house2.owner refers to the same
    # object as house.owner)
    house2.owner.name = 'Sally'
    print(house2.owner.name) # Prints Sally
    print(house.owner.name) # Prints Sally

    # Also, house.owner was defined to refer to the same object
    # as the variable 'joe', modifying house.owner.name ALSO
    # modifies joe.name:
    print(joe.name) # Prints Sally
    print()

    # But remember: house and house2 refer to different objects. So
    # modifying house2.owner does NOT modify house.owner
    amanda = Person()
    amanda.name = 'Amanda'
    house2.owner = amanda

    print(house2.owner.name) # Prints Amanda
    print(house.owner.name) # Prints Sally
    print()

    # Now that we have modified house2.owner, it now refers
    # to a different object from house.owner (these print different
    # identifiers):
    print(id(house2.owner))
    print(id(house.owner))
    print()

    # So NOW, modifications to house2.owner.name no longer modify
    # house.owner.name (because house.owner and house2.owner refer
    # to different objects at this point):
    house2.owner.name = 'Mahatma'
    print(house2.owner.name) # Prints Mahatma
    print(house.owner.name) # Prints Sally


if __name__ == '__main__':
    main()

Make sure to trace the above comments carefully and verify that you understand what's going on here.

In contrast, here's an example illustrating deep copies:

deepcopy.py

from copy import deepcopy # Used for deep copies

class Person:
    name: str

class House:
    owner: Person

def main() -> None:
    joe = Person()
    joe.name = 'Joe'

    house = House()
    house.owner = joe

    # house2 is a DEEP copy of house
    house2 = deepcopy(house)

    # house and house2 refer to separate objects (these print different
    # identifiers)
    print(id(house))
    print(id(house2))
    print()

    # house.owner and house2.owner ALSO refer to different objects
    # (these print different identifiers):
    print(id(house.owner))
    print(id(house2.owner))
    print()

    # No modifications to house2 will result in any modifications to
    # house:
    house2.owner.name = 'Sally'
    print(house2.owner.name) # Prints Sally
    print(house.owner.name) # Prints Joe



if __name__ == '__main__':
    main()

If you're just copying primitives (e.g., integers), or if you're copying objects whose attributes all refer to primitives (e.g., a Person with a string attribute name and integer attribute age, but no class-type attributes), then it technically doesn't matter whether you use shallow copies or deep copies. In such cases, the two kinds of copies are equivalent. But if you want to copy an object that has other objects inside it that, in turn, have other objects inside them (and so on), then there's a huge difference between a shallow copy and a deep copy. And in such cases, you usually want a deep copy, or else certain modifications to the copy could result in corresponding modifications to the original.

The only advantage of shallow copies is that, when copying extremely complicated objects with deep object graphs, shallow copies are more efficient than deep copies. This is because shallow copies only copy the "outer" object, whereas deep copies traverse the entire object graph and copy everything. But again, shallow copies can lead to issues if you want the two objects (the original and the copy) to serve as completely independent objects from one another, enabling you to modify the deeper contents of one without it affecting the corresponding contents of the other.

Lastly: everything that I've told you about the assignment operator—about how it modifies the variable on the left to refer to a new object as specified on the right—also applies to function arguments and parameters. When a function is called, the function's parameters (which are variables and therefore references) are set up to refer to the objects specified by the arguments. Indeed, this means that a given function parameter and its corresponding argument are two separate variables, but they refer to the same underlying object in the context of the function call. If the function then reaches inside that object (e.g., via the dot operator), that will also reach inside the object that the argument refers to. To avoid issues in such cases, you may need to use deepcopy() to copy the object before passing it as an argument:

functioncall.py

from copy import deepcopy

class Person:
    name: str

def change_person(p: Person) -> None:
    p.name = 'Sally'

def main() -> None:
    joe = Person()
    joe.name = 'Joe'

    # The parameter, p, refers to the same object that the argument,
    # joe, refers to. Line 5, then, reaches inside the object that
    # p refers to (which is also the object that joe refers to),
    # modifying its name attribute to refer to a new string object
    # storing the value 'Sally'. This means that joe.name is ALSO
    # modified
    change_person(joe)

    print(joe.name) # Prints Sally
    
    # Let's change joe's name back to 'Joe'
    joe.name = 'Joe'

    # This time, to avoid the issue, we can use copy() or deepcopy()
    # to create a copy of the OBJECT that joe refers to, and then pass
    # THAT as the argument. p will then refer to THAT object, rather
    # than the object that joe refers to. In this case, copy() and
    # deepcopy() will both work. But again, we usually want a deep copy
    change_person(deepcopy(joe))

    print(joe.name) # Prints Joe

if __name__ == '__main__':
    main()

Remember when I told that lists behave a bit strange when used as parameters? How, when a function modifies the elements within a list that's received as a parameter, the corresponding elements are also modified within the list that was passed as an argument? This is precisely the reason: the parameter is not a copy of the argument's value, but rather a reference to the argument's value. If you want any sort of object-copying to take place, you have to do it yourself by calling deepcopy().

(Technically, the same principles apply to return values as well, but that rarely matters in practice).