Python Tips & Tricks

I’ve been working with Python on and off for more than ten years now. Along the way I’ve gathered a few small tips and tricks for Python programming. Most of them didn’t feel like they warranted their own blog post, so I’ve compiled them into a single entry instead.

Functions

Named arguments

Positional arguments can be used as named arguments, i.e. you can use the same a="b" syntax as you would for a keyword argument. Here’s an example:

def foo(a, b):
    return [a, b]

foo(b="B", a="A")
# ["A", "B"]

Useful for calling functions where it’s not obvious which positional argument is which.

Identity

Some of the built-in constructor functions, e.g. str, int, float etc, will return an identity value when called without arguments.

For example, str() will return the empty string, int() will return 0 and so on. This isn’t terribly useful by itself, but can come in handy when you need a function that will return a “default” value; we’ll see some examples of this further down.

Omitting parentheses

You can omit parentheses from certain expressions (like tuples). For example, parentheses can be omitted from generator comprehensions if the comprehension is already parenthesised:

# These are equivalent:
sum((x ** 2 for x in range(3)))
sum(x ** 2 for x in range(3))

Generators

Yield from

Since Python 3.3 it’s possible to use yield from to yield directly from another generator.

# These two are equivalent:

def foo():
    for i in some_iterable:
        yield i

def foo():
    yield from some_iterable

Creating generators with iter()

iter(iterable) creates a generator from an iterable.

iter(callable, sentinel) creates a generator that returns calls from the (zero-argument) callable until it returns the sentinel value. This means you can easily create an infinite generator by doing something like iter(int, None) This generator will yield an infinite number of zeroes, as int() returns 0 (see above) which will never be equal to None

Dictionaries

defaultdict

A defaultdict is a dictionary that calls a “factory function” to supply a default value for missing keys. This is very useful when e.g. grouping or aggregating values:

from collections import defaultdict

# Create a defaultdict where the default value is an empty list:
d = defaultdict(list)

# Aggregate words based on their first letter
words = [
    "alpaca", "capybara", "crocodile", "bonobo",
]
for w in words:
    d[w[0]].append(w)

dict(d)
# {
#   'a': ['alpaca'],
#   'c': ['capybara', 'crocodile'],
#   'b': ['bonobo']
# }

Have you ever wanted your Python dictionaries to automatically create keys when you try to access them? For example:

d = {}
# This throws a KeyError since "foo" is not set:
d["foo"]["bar"] = "baz"

Now you can!

from collections import defaultdict

def dict_factory():
    return defaultdict(dict_factory)

d = dict_factory()
d["foo"]["bar"] = "baz"

Absolutely cursed.

Checking if a dict a subset of another dict

Since dict.items acts as a set you can use the overloaded > operator to check if a dict is a “subset” of another dict:

{"a": 1, "b": 2}.items() > {"b": 2}.items()
# True

Dates and times

If you’re working with human (i.e. weird) dates and times, use the calendar module to maintain your sanity.

Durations

Instead of using magic numbers for durations of time, e.g.

# 12 hours
ttl_in_seconds = 60 * 60 * 12

use the built-in timedelta functions and types:

ttl = datetime.timedelta(hours=12)
expires_in = datetime.datetime.now() + ttl

This gives you safety at both the type and unit level, e.g. there’s no risk of confusing seconds and milliseconds.

Enums

If some input should be constrained to a set of values, use the enum module, e.g.

from enum import Enum

class Color(Enum):
    RED = 1
    GREEN = 2
    BLUE = 3

Enum values can be accessed either by dot notation or subscripts, e.g:

Color.RED == Color['RED']

To convert a primitive value to an enum instance, use the Enum’s constructor:

Color("RED")

Note that an enum instance is not equal to its corresponding primitive value!

Color.RED != "RED"

Remember that enums are regular classes; this can be used e.g. to provide utility methods for working with enum values.

class Color(Enum):
    RED = 1
    GREEN = 2
    BLUE = 3

    def hex_color(self):
        hex_values = {
            "RED": "#ff0000",
            "GREEN": "#00ff00",
            "BLUE": "#0000ff",
        }
        return hex_values[self.name]

Numbers

If you need a really big number, why settle for anything less than infinity?

float('Inf')
float('-Inf')

(This is actually part of the IEEE 754 floating-point spec!)

Decimal

If you need precise arithmetic when dealing with fractions, use the [decimal] module. Your accountant will thank you.

A neat trick when working with numbers in JSON is that they’re represented as arbitrary-precision decimal strings. This means they can be losslessly converted to and from Decimal instances, as in this example from the JSON module documentation:

json.loads('1.1', parse_float=decimal.Decimal)

Metaprogramming

You can use the inspect module to get information about live objects, e.g. the argument names of a function. Useful, for example, when you want to decorate a function and tap into one of its arguments.

You can also use inspect.getdoc to get the docstring of a function at runtime.

Shell Utilites

Python has some cool convenience utilities:

Start a HTTP server that serves files from the current directory:

python3 -m http.server

Pretty-print JSON:

python3 -m json.tool < foo.json

You can use pydoc or pydoc3 to read Python documentation on the command line.

Modules and packages

Referencing imports

Whenever you try to use a Python mocking tool you may be surprised by the following behaviour: If you use from X import Y then Y will be added to the importing module’s namespace.

Let’s say a module foo imports bar.quux the following way:

from bar import quux

Then the way to reference the quux module at runtime becomes foo.quux and not bar.quux as one might expect.

Packages

The presence of an __init__.py file (which may be empty) indicates that a directory is a Python package.

The file may be empty, but can also contain executable code. One common trick to consolidate multiple files into what appears to be one module is to do this in __init__.py

# In file: foo/__init__.py
from foo import Foo
from bar import Bar

# Now you can import Foo and Bar like so:
from foo import Foo, Bar

Gotcha: Remember that every parent directory of the package path must also contain an __init__.py file! Forgetting to include this file in some directory can break imports and static analysis tools.

Code style

I used to have a lot of opinions about how to best format Python code, but now I have only one: Use black.

Misc

Python is one of the few languages that allow chained use of the lesser- / greater-than operators:

if 0 < foo < 10:
    do_something()

Gotchas

Default keyword arguments

Default keyword argument values are not re-instantiated every time a method is called call, e.g. if you have a kwarg foo={} then every call to that function will refer to the same dictionary. Oops!

def foo(a=[]):
    return a.append("test")
foo() # Returns ["test"]
foo() # Returns ["test", "test"]

To avoid surprising behaviour, don’t use a mutable type as default arguments; use None instead.