Python Tips & Tricks

A magician pulling a python out of a hat

This is a collection of more or less exotic Python tips and tricks I’ve gathered over the years.

If you’ve got some of your own that you’d like to share hit me up on Twitter!

Functions

Positional arguments can be used as named arguments, i.e. you can use the same a="b" syntax as you would for a keyword argument. Here’s an example:

def foo(a, b):
    return [a, b]

foo(b="B", a="A")
# ["A", "B"]

Useful for calling functions where it’s not obvious which positional argument is which.

Comprehensions

There are a variety of comprehensions in Python:

List: [x for x in iterable]
Dict: [k: v for k, v in iterable]
Set: {x for x in iterable}
Generator: (x for x in iterable)

Comprehensions can contain multiple for clauses. The following yields the “cross product” of list1 and list2:

numbers = [1, 2]
letters = ["a", "b"]
[(n, l) for n in numbers for l in letters]
# [(1, "a"), (2, "a"),
#  (1, "b"), (2, "b")]

The output of one for clause can also be used as input to another for clause. To flatten a nested list, for example, you can do this:

nested = [(1, 2), (3, 4), (5, 6)]
[b for a in nested for b in a]
# [1, 2, 3, 4, 5, 6]

(Mnemonic: The comprehension goes from left-to-right, outside-in)

Generator expressions

Generator expressions are like list comprehensions, but return a generator instead of a list.

Cool trick: Just like with tuples, parentheses can be omitted from generator comprehensions if the expression is already parenthesised:

# These two are equivalent:
sum((x ** 2 for x in range(3)))
sum(x ** 2 for x in range(3))

Generators

Yield from

Since Python 3.3 it’s possible to use yield from to yield directly from another generator.

# These two are equivalent:

def foo():
    for i in some_iterable:
        yield i

def foo():
    yield from some_iterable

Creating generators with iter

iter(iterable) creates a generator from an iterable.

iter(callable, sentinel) creates a generator that returns calls from the (zero-argument) callable until it returns the sentinel value. This means you can easily create an infinite generator by doing something like iter(int, None) (since int() returns 0).

Dictionaries

Have you ever wanted your Python dictionaries to automatically create keys when you try to access them? For example:

d = {}
# This throws a KeyError since the key "foo" is not set:
d["foo"]["bar"] = "baz"

Now you can!

from collections import defaultdict

def makehash():
    return defaultdict(makehash)

d = makehash()
d["foo"]["bar"] = "baz"

(For the record I think this is a terrible idea, but there you go)

Checking if a dict a subset of another dict

Since dict.items acts as a set you can use the overloaded > operator to check if a dict is a “subset” of another dict:

{"a": 1, "b": 2}.items() > {"b": 2}.items()
# True

Dates and times

If you’re working with human (i.e. weird) dates and times, use the calendar module to maintain your sanity.

Durations

Instead of using magic numbers for durations of time, e.g.

# 12 hours
ttl_in_seconds = 60 * 60 * 12

use the built-in timedelta functions and types:

ttl = datetime.timedelta(hours=12)
expires_in = datetime.datetime.now() + ttl

This gives you safety at both the type and unit level, e.g. there’s no risk of confusing seconds and milliseconds.

Enums

If some input should be constrained to a set of values, use the enum module, e.g.

from enum import Enum

class Color(Enum):
    RED = 1
    GREEN = 2
    BLUE = 3

Enum values can be accessed either by dot notation or subscripts, e.g:

Color.RED == Color['RED']

To convert a primitive value to an enum instance, use the Enum‘s constructor:

Color("RED")

Note that an enum instance is not equal to its corresponding primitive value!

Color.RED != "RED"

Remember that enums are regular classes (well, for the most part); this can be used e.g. to provide utility methods for working with enum values.

class Color(Enum):
    RED = 1
    GREEN = 2
    BLUE = 3

    def hex_color(self):
        hex_values = {
            "RED": "#ff0000",
            "GREEN": "#00ff00",
            "BLUE": "#0000ff",
        }
        return hex_values[self.name]

Numbers

If you need a really big number, why settle for anything less than infinity?

float('Inf')
float('-Inf')

(This is actually part of the IEEE 754 spec)

Decimal

If you need precise arithmetic when dealing with fractions, use the decimal module. Your accountant will thank you.

A neat trick when working with numbers in JSON is that they’re represented as arbitrary-precision decimal strings. This means they can be losslessly converted to and from Decimal instances, as in this example from the JSON module documentation:

json.loads('1.1', parse_float=decimal.Decimal)

Tip: simplejson comes with out-of-the-box support for using decimal for parsing and serialization.

Metaprogramming

You can use the inspect module to get information about live objects, e.g. the argument names of a function. Useful, for example, when you want to decorate a function and tap into one of its arguments.

You can also use inspect.getdoc to get the docstring of a function at runtime.

Shell Utilites

Python has some cool convenience utilities:

Start a HTTP server that serves files from the current directory:

python3 -m http.server

Pretty-print JSON:

python3 -m json.tool < foo.json

You can use pydoc or pydoc3 to read Python documentation on the command line.

Modules and packages

Referencing imports

Whenever you try to use a Python mocking tool you may be surprised by the following behaviour: If you use from X import Y then Y will be added to the importing module’s namespace.

Let’s say a module foo imports bar.quux the following way:

from bar import quux

Then the way to reference the quux module at runtime becomes foo.quux and not bar.quux as one might expect.

Packages

The presence of an __init__.py file (which may be empty) indicates that a directory is a Python package.

The file may be empty, but can also contain executable code. One common trick to consolidate multiple files into what appears to be one module is to do this in __init__.py:

# In file: foo/__init__.py
from foo import Foo
from bar import Bar

# Now you can import Foo and Bar like so:
from foo import Foo, Bar

Gotcha: Remember that every parent directory of the package path must also contain an __init__.py file! Forgetting to include this file in some directory can break imports and static analysis tools.

Code style

I used to have a lot of opinions about how to best format Python code, but now I have only one: Use black.

Misc

Python is one of the few languages that allow chained use of the lesser- / greater-than operators:

if 0 < foo < 10:
    do_something()

Gotchas

Default keyword arguments

Default keyword argument values are not re-instantiated every time a method is called call, e.g. if you have a kwarg foo={} then every call to that function will refer to the same dictionary. Oops!

def foo(a=[]):
    return a.append("test")
foo() # Returns ["test"]
foo() # Returns ["test", "test"]

To avoid surprising behaviour, don’t use any of Python’s mutable types as default arguments; use None instead.