Profiling is a critical step when we try to optimize our code. There are a lot of profiling tools in the market. However, most of them share a common shortcoming — They provide too much information.
Don’t get me wrong, I’m not saying information is a bad thing. Sometimes we do need details to figure out why our code is slow, but for larger production level project, we are often more interested at “transaction level” profiling.
So, what is “transaction level” profiling? We should discuss “function level” profiling first, which is a more familiar concept for programmers. Take Python’s built in profiler — cProfile as an example. The profiling unit is “function”. cProfile will generate call count/time consumed information of each function. This is useful when we try to locate the bottleneck of a CPU-bound script. …
You might be familiar with the chrome trace profiling tool if you are a front-end engineer and often work with Chrome. In fact, you can open chrome://tracing and record your browser’s behavior. The result will look like the image below:
It is a great tool for both performance analysis and debugging, because it shows every single thing happened in all the threads in chronological order. You can zoom out the get a general idea of the whole picture, or zoom in to check the details of a single function.
Have you ever thought about “why is my program doing this thing” when you are developing? That happens to me quite a lot when I’m working with Python, and becomes the reason I built VizTracer.
VizTracer is helpful in many aspects, debugging, logging or profiling. The fundamental idea behind VizTracer is to help you understand what your program is doing.
Unlike traditional debuggers, which normally pin you at a specific time and allows you to check on variables, VizTracer focuses on the call stack over time. Basically, VizTracer visualizes how functions calls each other while your program is running.
This is a very simple example, the program is doing a recursive fibonacci number…
Debugging is an inevitable and often painful process when any programmer developing their programs. However, proper tools could make this process easier and help you to reach your destination faster and with less pain.
In this article, we will talk about a lot of python debugging tools that are still being maintained and could help you with your debugging process.
print() is still the number one choice for a lot of python programmers. I won’t spend too much time here because any python programmer should know how to use
print() to debug.
print() has many advantages, it’s built-in, it’s insanely easy to use and it’s very flexible. However, when your project gets larger and more complicated,
print() is never good enough to finish the job efficiently. …
I’ve shared the package VizTracer that I’ve been developing recently. I presented it as a profiler. However, it was more than a profiler. There are more aspects of it.
So today, I’ll take another angle and talk about it as a logger.
First of all I want to introduce a little bit about VizTracer to those who have never heard of it. It’s a python tool that can record data while your python program is running, and generate an HTML report of your execution like this:
The most important data VizTracer records is function entry/exit(or, FEE). It records every single call in your program and display them in timeline, which is very helpful for you to understand what your program is doing. …
My boss once told us, never optimize without profiling. That is very true in almost all cases. Without proper profiling, you may spend hours in optimizing and end up not improving your performance at all.
So, the first thing to optimize your code, is to find a good profiler.
There are a lot of options in the market, even when we only talk about python.
cProfiler, which is probably the most common profiler python programmers are using, is a built-in package that provides deterministic profiling. …