Latest posts of series Things I learnt in November 2022

Python: typing.overload

typing.overload makes it easier to type functions with behaviour that depends on input types. Functions marked with @overload are ignored by Python and only used by the type checker:

@overload
def process(response: None) -> None:
    ...
@overload
def process(response: int) -> tuple[int, str]:
    ...
@overload
def process(response: bytes) -> str:
    ...
def process(response):
    # <actual implementation>

Python's multiprocessing and deadlocks

Python's multiprocessing is prone to deadlocks in a number of conditions. In my case, the running program was a standard single-process, non-threaded script, but it used complex native libraries which might have been the triggers for the deadlocks.

The suggested workaround is using set_start_method("spawn"), but when we tried it we hit serious performance penalties.

Lesson learnt: multiprocessing is good for prototypes, and may end up being too hacky for production.

In my case, I was already generating small python scripts corresponding to worker tasks, which were useful for reproducing and debugging Magics issues, so I switched to running those as the actual workers. In the future, this may come in handy for dispatching work to HPC nodes, too.

Here's a parallel execution scheduler based on asyncio that I wrote to run them, which may always come in handy on other projects.

Debian:

  • You can Build-Depend on debhelper-compat (=version) and get rid of debhelper as a build-dependency, and of debian/compat (details)
  • You can Build-Depend on dh-sequence-foo and get rid of the corresponding dh-foo build-dependency, and of the need to add --with foo in debian/rules (details)
  • You can (and should) get rid of dh-buildinfo, which is now handled automatically
  • In salsa.debian.org there is a default CI pipeline for Debian packages that works beautifully without needing to add any .gitlab-ci.yml to a repository
  • Add Testsuite: autopkgtest-pkg-python to debian/control, and you get a free autopkgtest that verifies that your packaged Python module can be imported. The default CI pipeline in salsa will automatically run the tests. (specification, details)

Python:

  • From Python 3.8, you can use = in format strings to make it easier to debug variables and expressions (details):
>>> name="test"
>>> print(f"{name=}")
name='test'
>>> print(f"{3*8=}")
3*8=24

Leaflet:

  • [abc].tile.openstreetmap.org links need to be replaced with tile.openstreetmap.org (details)