Modern Python Standard Library Cookbook

Managing filename encoding

Working with filesystems in a reliable way is not as easy as it might seem. Our system must have a specific encoding to represent text and usually that means that everything we create is handled in that encoding, including filenames.

The problem is that there is no strong guarantee on the encoding of filenames. Suppose you attach an external hard drive, what's the encoding of filenames on that drive? Well, it will depend on the encoding the system had at the time the files were created.

Usually, to cope with this problem, software tries the system encoding and if it fails, it prints some placeholders (have you ever seen a filename full of ? just because your system couldn't understand the name of the file?), which ...