Unspeakable Python Metaprogramming
A short guide on how to commit devious python metaprogramming crimes and get away with it.
There are typically three reasons why you might write python code that can change its execution context while it is running. The first reason to do so happens trivially, and it occurs because you don’t know any better.
Metaprogramming By Accident
age = input("How old are you?")
print("you will be", eval(age) + 2, "in to years time")
I see this kind of thing often when people are new to python. They want the ‘24’
that the user enters to be used interchangeably with the 24 that they would
type into their code editor. People don’t usually consider it to be
metaprogramming but I’m going to claim a technicality. The programmer may not
know what behaviors they have introduced, but they have written a program that
can alter its own structure and behavior when it runs.
Metaprogramming For Evil
The other reason to use metaprogramming is when you hate your coworkers and you would like to prevent them from reasoning about the code that they write. Metaprogramming with a heart full of malice might be fun, but I encourage you to look within and restrain your perverted impulses.
import colins_helpful_logger
import math
print(math.sqrt(9)) # Outputs: 42
bad news friend, even though you did not call ANY functionality from colin’s helpful logger, it contained the following
import sys
import types
class EvilImporter:
def find_spec(self, fullname, path, target=None):
if fullname == "math":
fake_math = types.ModuleType("math")
fake_math.sqrt = lambda x: 42
sys.modules[fullname] = fake_math
return None
sys.meta_path.insert(0, EvilImporter())
This is an obvious way to get caught, I assure you that you can think of better ways to maintain plausible deniability. The problem with evil code of this kind is that it changes your program only at runtime, you get very few hints as to where the behavior comes from and there is no concrete static analysis that gets you from the misbehaving code to its runtime override.
Metaprogramming For Good
The final reason to use metaprogramming is when you love and care for your coworkers, and you want to protect them from themselves.
Consider this injector that I have used before when working in a project that has LOTS of concurrent worker functions. I want to force any functions in a certain context to be reliably cancelable at the cost of speed and timing reliability.
I can reasonably enforce that contract by changing the code written by users of my API. I love my co-workers, and so I will automatically fix their code whenever they try and run it.
class InterleaveHookInjector(ast.NodeTransformer):
def body_sleeper_logic(
self, node: ast.While | ast.For | ast.AsyncFor | ast.AsyncFunctionDef
) -> ast.AST:
for idx, expr in enumerate([x for x in node.body]):
with contextlib.suppress(IndexError):
if ast.unparse(node.body[idx + 1]) == "await asyncio.sleep(0)":
continue
node.body.insert(
idx + idx + 1,
ast.Expr(
ast.Await(
ast.Call(
func=ast.Attribute(
value=ast.Name(id="asyncio", ctx=ast.Load()),
attr="sleep",
ctx=ast.Load(),
),
args=[
ast.Constant(0),
],
),
),
),
)
ast.fix_missing_locations(node)
self.generic_visit(node)
return node
def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> ast.AST:
for idx, dec in enumerate([x for x in node.decorator_list]):
if isinstance(dec, ast.Name) and dec.id == interleave.__name__:
node.decorator_list.pop(idx)
ast.fix_missing_locations(node)
return self.body_sleeper_logic(node)
def visit_While(self, node: ast.While) -> ast.AST:
return self.body_sleeper_logic(node)
def visit_For(self, node: ast.For) -> ast.AST:
return self.body_sleeper_logic(node)
def visit_AsyncFor(self, node: ast.AsyncFor) -> ast.AST:
return self.body_sleeper_logic(node)
This effectively takes some async function like
@interleave
async def interruptable():
start = time.monotonic()
trip = False
while time.monotonic() - start < 1:
trip = not trip
for x in range(4):
print(trip)
And will unwind the syntax tree, alter it to yield after every expression, and then wind it back up.
async def interruptable():
start = time.monotonic()
await asyncio.sleep(0)
trip = False
await asyncio.sleep(0)
while time.monotonic() - start < 1:
trip = not trip
await asyncio.sleep(0)
for x in range(4):
print(trip)
await asyncio.sleep(0)
await asyncio.sleep(0)
The @interleave decorator is responsible for this runtime-change to the code.
This has a relatively small performance overhead but gives my subsystem a much greater chance at running in the way I intend it to.
There is no practical difference between this and the evil metaprogramming, and usually it is fairly likely to end up as a paving stone on the road to hell.
With great power comes great responsibility.