Python Cookbook

Debugging Memory Problems

Credit: Will Ware

Problem

You’re developing C extensions, and you experience memory problems. You suspect mismanagement of reference counts and want to check whether your C extension code is managing reference counts correctly.

Solution

To chase these problems optimally, you need to alter Python’s sources and rebuild Python. Specifically, add this function in Objects/object.c immediately before the _Py_PrintReferences function:

void
_Py_CountReferences(FILE *fp)
{
    int n;
    PyObject *op;
    for (n = 0, op = refchain._ob_next;
         op != &refchain;
         op = op->_ob_next, n += op->ob_refcnt)
    { }
    fprintf(fp, "%d refs\n", n);
}

I place in the following macros in my C extension:

#if defined(Py_DEBUG) || defined(DEBUG)
extern void _Py_CountReferences(FILE*);
#define CURIOUS(x) { fprintf(stderr, _ _FILE_ _ ":%d ", _ _LINE_ _); x; }
#else
#define CURIOUS(x)
#endif
#define MARKER(  )        CURIOUS(fprintf(stderr, "\n"))
#define DESCRIBE(x)     CURIOUS(fprintf(stderr, "  " #x "=%d\n", x))
#define DESCRIBE_HEX(x) CURIOUS(fprintf(stderr, "  " #x "=%08x\n", x))
#define COUNTREFS(  )     CURIOUS(_Py_CountReferences(stderr))

To debug, I rebuild Python using make OPT="-DPy_DEBUG", which causes the code under Py_TRACE_REFS to be built. My own makefile for my extensions does the same trick by including these lines:

debug:
        make clean; make OPT="-g -DPy_DEBUG" all
CFLAGS = $(OPT) -fpic -O2 -I/usr/local/include -I/usr/include/python1.5

Discussion

If I’m developing C extensions and I run into memory problems, I find ...