Summary of Python Performance Optimization Techniques

1.Using measurement tools, quantifying performance is the only way to improve performance. Commonly used are timeit and memory_profiler, in addition to profile, cProfile, hotshot, etc. memory_profiler uses psutil, so it cannot track CPython extensions;

2.To solve time-consuming processing with C, C is synonymous with efficiency, and it is also the main way Python uses to solve efficiency issues, even sometimes I feel that Python is the perfect partner of C. Cython is commonly used, which directly converts Python code into C and can still be used like a Python package; the next is ctypes, the most efficient existence, and finally, there are also CPython and cffi, all of which are excellent;

3.Optimize algorithms, a common problem in all languages. I think that improving algorithms is the most important of all improvements, but it is also the most difficult. Fortunately, most commonly used algorithms have been packaged. Unless you dig a pit for yourself, it is very important to understand how the standard library's data structures and commonly used APIs are implemented.

4.2there are some more efficient packages that have been implemented to replace some common implementations in Python. If the bottleneck is on StringIO, pickle, profile, and similar, consider replacing them with the C version;

5.Use tuples as much as possible, especially when the data volume is large. If you really can't, you can use lists. Try not to use class, and if you have to use it, you can add slot. If the efficiency is still not enough, you can combine2to accelerate it;

6.Lazy loading, import does not have to be written at the beginning of a page, it can be anywhere. The more fragmented, the better you can delay the loading of packages even if they are not loaded;

7.Use multiprocessing to implement multithreading, which can break the GIL limit;

8.python to handle loops is terrible, and that's the way it is for interpreted languages. Compared to other compiled languages, it is like a snail. Therefore, reducing the number of loops and nesting can significantly improve performance. Of course, using pypy doesn't have this problem;

9.Using accelerators, I really like the way psyco is used. If you use2.7-If the version is the same as yours, it is not a bad choice for a lazy person, but it is no longer maintained, and the founder has gone to pypy. pypy is a Python implemented in Python, and the underlying language is converted to a platform-dependent intermediate language such as C, .net, and Java, which is very smart and loved. However, the disadvantage is that the library support is not perfect. My project can basically support it, and solving a few small problems can solve it. If the performance bottleneck is in loops and memory, you can try it. The biggest advantage is that you don't need to change a single line of code and make any other settings, and it is not intrusive at all.

References:

Python Code Performance Optimization Techniques: https://www.ibm.com/developerworks/cn/linux/l-cn-python-optim/

Python Performance Optimization Techniques: http://kuanghy.github.io/2016/09/26/python-optimize