Forem: Inada Naoki

Python: Use the UTF-8 mode on Windows!

Inada Naoki — Thu, 12 Dec 2019 06:59:46 +0000

Summary: Set the PYTHONUTF8=1 environment variable.

On macOS and Linux, UTF-8 is the standard encoding already.
But Windows still uses legacy encoding (e.g. cp1252, cp932, etc...) as system encoding.

Python works very well about file names and console IO (e.g. use ~W APIs). But the legacy system encoding is used for the default encoding of text files and pipes.

It is a very common mistake that omits the encoding="utf-8" option. Developers who use macOS or Linux doesn't have any trouble by the mistake.

For example, even the packaging tutorial in the packaging.python.org uses this code snippet:

with open("README.md", "r") as fh:
    long_description = fh.read()

When README.md contains some non-ASCII characters (e.g. Unicode emoji), setup.py will fail on Windows. Windows users can not install the package when wheel is not provided. (I sent a pull request to fix this example code already.)

I believe many Python programmers are suffered by this default text file encoding on Windows because:

The default encoding of Python source code is UTF-8
UTF-8 is the standard encoding of the Web
Modern text editors like VS Code use UTF-8 by default. And even the notepad.exe chose UTF-8 for the default encoding!

But it is difficult to change the default encoding of text files because it is backward incompatible change. It will break some legacy applications which relying on the legacy encoding.

But there is good news: Python 3.7 introduced the "UTF-8 mode". (thanks to Victor Stinner!!)

When UTF-8 mode is enabled, Python uses UTF-8 as default encoding for text files instead of system encoding.
You can live in the world "UTF-8 is the default. Other legacy encodings are used only when explicitly specified." like macOS and Linux.

To enable UTF-8 mode:

Set the the environment variable PYTHONUTF8=1, or
Use -Xutf8 command line option.

Use pip3 to install awscli

Inada Naoki — Tue, 25 Jun 2019 15:19:42 +0000

NOTE: AWS provides bundled installer. It is much better than pip install awscli, especially when you are installing awscli on EC2.

As you know, Python 2 will be EOL at 2020-01-01.

But more than 50% of downloads from PyPI are still from Python 2.7. This graph is copied from PyPI Stats.

This huge number of download from Python 2 put pressure on library authors to keep supporting Python 2. I want to reduce this pressure by Python 2 EOL.

Who uses Python 2 to pip install so much? This is a hint.

Most downloads are from Linux, although most Python users are using macOS or Windows. We can assume most pip install executions are for CI/CD/server deployment.

Then which packages are most downloaded? This is ranking from PyPI Stats:

All packages in the top 20 are pip, awscli, and those dependencies!

And 80% of pip install awscli are using Python 2!

If you know some document or template uses Python 2 to install awscli, please suggest using Python 3 to the author.

May Python 2 rest in peace.

Don't omit init.py

Inada Naoki — Fri, 17 May 2019 05:50:54 +0000

I saw some people think Python 3 allow package without __init__.py on Stackoverflow or mailing lists.

This is misunderstanding. Package directory without __init__.py is "namespace package", not a regular package. See PEP 420 for reference.

Namespace package is special package for people creating complex "distribution package" ("package" which can be registered on PyPI). There is guide for namespace package in "Python Packaging User guide".

There are some difference between namespace package and regular package.

For example, unittest module in standard library doesn't search into directory without __init__.py. If they do, they may take dozen seconds to search tests in node_modules which contains millions of files and subdirectories.

You should not omit __init__.py in your project unless you are 100% sure about what "namespace package" is. Use "regular package" with __init__.py instead.

How to speed up Python application startup time

Inada Naoki — Fri, 19 Jan 2018 08:48:47 +0000

I hear pipenv 9.0.2 is released with major startup time improvement.

// Detect dark theme var iframe = document.getElementById('tweet-953294094029262849-987'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=953294094029262849&theme=dark" }

I tried it soon and I didn't feel it's fast. So I investigated it with Python 3.7's new feature.

In this article, I introduce the feature and how to use it.

Startup time ≒ import time

For example, execution time of pipenv -h is much larger than time to show help message.

Generally speaking, when application starts, there are some startup process like loading environment variables or config files.

In case of Python application, importing module takes most of startup time. For example, pipenv --version took about 800ms and import pipenv took 700ms.

$ time local/py37/bin/python3 -c 'import pipenv'

real    0m0.696s
user    0m0.648s
sys     0m0.048s

$ time local/py37/bin/pipenv --version
pipenv, version 9.0.3

real    0m0.812s
user    0m0.761s
sys     0m0.052s

Show import time for modules

Python 3.7 has new feature to show time for importing modules.

This feature is enabled with -X importtime option or PYTHONPROFILEIMPORTTIME environment variable.

For example, you can profile import time of pipenv by

python3.7 -X importtime -c 'import pipenv' 2> pipenv-imports

PYTHONPROFILEIMPORTTIME=1 pipenv --version 2>pipenv-imports

Here is example output of pipenv --version

Investigating import time

At last of the output, you can see these lines:

import time: self [us] | cumulative | imported package
...
import time:      3246 |     578972 |   pipenv.cli
import time:       507 |     579479 | pipenv

In last line, 579479 means import pipenv took 579 479us.

While importing pipenv, many other modules are imported. From above example, you can see pipenv imports pipenv.cli. Subimports are indented with 2 spaces.

See last line again. 507 means only 507us was took when running pipenv module. 579 479 - 507 = 578 972us is used for subimports.

Finding slow part

Let's find slow subtree from the output. I selected some lines.

import time: self [us] | cumulative | imported package
...
import time:     86500 |     179327 | pkg_resources
...
import time:       385 |     236655 |             IPython
import time:        22 |     236677 |           IPython.core
import time:        26 |     236703 |         IPython.core.magic
import time:       978 |     237680 |       dotenv.ipython
import time:       199 |     239032 |     dotenv
...
...
import time:      3246 |     578972 |   pipenv.cli
import time:       507 |     579479 | pipenv

pkg_resources

As you can see, importing pkg_resources is slow.
And surprisingly, pkg_resources is not indented; it's not subimport of pipenv.

It means pkg_resources is imported by pipenv script, not module.

$ cat local/py37/bin/pipenv
#!/home/inada-n/local/py37/bin/python3.7
# EASY-INSTALL-ENTRY-SCRIPT: 'pipenv==9.0.3','console_scripts','pipenv'
__requires__ = 'pipenv==9.0.3'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('pipenv==9.0.3', 'console_scripts', 'pipenv')()
    )

Bad news: Importing pkg_resources is slow. It's known issue and it's difficult to fix without breaking backward compatibility.

Good news: you can avoid importing pkg_resources!

$ local/py37/bin/pip install wheel
$ local/py37/bin/pip install -U --force-reinstall pipenv
$ time local/py37/bin/pipenv --version
pipenv, version 9.0.3

real    0m0.704s
user    0m0.653s
sys     0m0.052s

When wheel is installed, pip builds wheel and install from it.

Installing from wheel (.whl) and from source package (.tar.gz) are different process.
When installing from wheel, pkg_resources is not used in script:

$ cat local/py37/bin/pipenv
#!/home/inada-n/local/py37/bin/python3.7

# -*- coding: utf-8 -*-
import re
import sys

from pipenv import cli

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(cli())

IPython

See next part.

import time:       385 |     236655 |             IPython
import time:        22 |     236677 |           IPython.core
import time:        26 |     236703 |         IPython.core.magic
import time:       978 |     237680 |       dotenv.ipython
import time:       199 |     239032 |     dotenv

pipenv imports dotenv, dotenv imports dotenv.ipython, and it imports IPython.

That's why pipenv starts slow on my environment; I have IPython installed.

But why IPython is imported? I read dotenv source and I found it's for IPython's extension.

Surely, pipenv and many dotenv users don't use IPython extension.
I made pull request to dotenv which make importing IPython on demand.

And since pipenv has own copy of dotenv, I made pull request to pipenv which removes dotenv.ipython completely.

Conclusion

I can reduce time for pipenv --version from 800ms to 500ms.

$ time local/py37/bin/pipenv --version
pipenv, version 9.0.3

real    0m0.503s
user    0m0.463s
sys     0m0.040s

Import time profiling is very nice way to investigating and optimize application startup time.