* Add profiling module documentation structure
PEP 799 introduces a new `profiling` package that reorganizes Python's
profiling tools under a unified namespace. This commit adds the documentation
structure to match: a main entry point (profiling.rst) that helps users choose
between profilers, detailed docs for the tracing profiler (profiling-tracing.rst),
and separated pstats documentation.
The tracing profiler docs note that cProfile remains as a backward-compatible
alias, so existing code continues to work. The pstats module gets its own page
since it's used by both profiler types and deserves focused documentation.
* Add profiling.sampling documentation
The sampling profiler is new in Python 3.15 and works fundamentally differently
from the tracing profiler. It observes programs from outside by periodically
capturing stack snapshots, which means zero overhead on the profiled code. This
makes it practical for production use where you can attach to live servers.
The docs explain the key concepts (statistical vs deterministic profiling),
provide quick examples upfront, document all output formats (pstats, flamegraph,
gecko, heatmap), and cover the live TUI mode. The defaults table helps users
understand what happens without any flags.
* Wire profiling docs into the documentation tree
Add the new profiling module pages to the Debugging and Profiling toctree.
The order places the main profiling.rst entry point first, followed by the
two profiler implementations, then pstats, and finally the deprecated profile
module last.
* Convert profile.rst to deprecation stub
The pure Python profile module is deprecated in 3.15 and scheduled for removal
in 3.17. Users should migrate to profiling.tracing (or use the cProfile alias
which continues to work).
The page now focuses on helping existing users migrate: it shows the old vs new
import style, keeps the shared API reference since both modules have the same
interface, and preserves the calibration docs for anyone still using the pure
Python implementation during the transition period.
* Update CLI module references for profiling restructure
Point cProfile to profiling.tracing docs and add profiling.sampling to the
list of modules with CLI interfaces. The old profile-cli label no longer
exists after the documentation restructure.
* Update whatsnew to link to profiling module docs
Enable cross-references to the new profiling module documentation and update
the CLI examples to use the current syntax with the attach subcommand. Also
reference profiling.tracing instead of cProfile since that's the new canonical
name.
- Introduce a new field in the GC state to store the frame that initiated garbage collection.
- Update RemoteUnwinder to include options for including "<native>" and "<GC>" frames in the stack trace.
- Modify the sampling profiler to accept parameters for controlling the inclusion of native and GC frames.
- Enhance the stack collector to properly format and append these frames during profiling.
- Add tests to verify the correct behavior of the profiler with respect to native and GC frames, including options to exclude them.
Co-authored-by: Pablo Galindo Salgado <pablogsal@gmail.com>
Implement a statistical sampling profiler that can profile external
Python processes by PID. Uses the _remote_debugging module and converts
the results to pstats-compatible format for analysis.
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
* expand on What's New entry for PEP 667 (including porting notes)
* define 'optimized scope' as a glossary term
* cover comprehensions and generator expressions in locals() docs
* review all mentions of "locals" in documentation (updating if needed)
* review all mentions of "f_locals" in documentation (updating if needed)
Example needed to be indented. Was trying to call a context manger `pr` (from ` with cProfile.Profile() as pr:`) wot perform ` pr.print_stats()` once it had already exited.
Automerge-Triggered-By: GH:AlexWaygood
Replace old names when they refer to actual versions of macOS.
Keep historical names in references to older versions.
Co-authored-by: Patrick Reader <_@pxeger.com>
pstats is really useful or profiling and printing the output of the execution of some block of code, but I've found on multiple occasions when I'd like to access this output directly in an easily usable dictionary on which I can further analyze or manipulate.
The proposal is to add a function called get_profile_dict inside of pstats that'll automatically return this data the data in an easily accessible dict.
The output of the following script:
```
import cProfile, pstats
import pprint
from pstats import func_std_string, f8
def fib(n):
if n == 0:
return 0
if n == 1:
return 1
return fib(n-1) + fib(n-2)
pr = cProfile.Profile()
pr.enable()
fib(5)
pr.create_stats()
ps = pstats.Stats(pr).sort_stats('tottime', 'cumtime')
def get_profile_dict(self, keys_filter=None):
"""
Returns a dict where the key is a function name and the value is a dict
with the following keys:
- ncalls
- tottime
- percall_tottime
- cumtime
- percall_cumtime
- file_name
- line_number
keys_filter can be optionally set to limit the key-value pairs in the
retrieved dict.
"""
pstats_dict = {}
func_list = self.fcn_list[:] if self.fcn_list else list(self.stats.keys())
if not func_list:
return pstats_dict
pstats_dict["total_tt"] = float(f8(self.total_tt))
for func in func_list:
cc, nc, tt, ct, callers = self.stats[func]
file, line, func_name = func
ncalls = str(nc) if nc == cc else (str(nc) + '/' + str(cc))
tottime = float(f8(tt))
percall_tottime = -1 if nc == 0 else float(f8(tt/nc))
cumtime = float(f8(ct))
percall_cumtime = -1 if cc == 0 else float(f8(ct/cc))
func_dict = {
"ncalls": ncalls,
"tottime": tottime, # time spent in this function alone
"percall_tottime": percall_tottime,
"cumtime": cumtime, # time spent in the function plus all functions that this function called,
"percall_cumtime": percall_cumtime,
"file_name": file,
"line_number": line
}
func_dict_filtered = func_dict if not keys_filter else { key: func_dict[key] for key in keys_filter }
pstats_dict[func_name] = func_dict_filtered
return pstats_dict
pp = pprint.PrettyPrinter(depth=6)
pp.pprint(get_profile_dict(ps))
```
will produce:
```
{"<method 'disable' of '_lsprof.Profiler' objects>": {'cumtime': 0.0,
'file_name': '~',
'line_number': 0,
'ncalls': '1',
'percall_cumtime': 0.0,
'percall_tottime': 0.0,
'tottime': 0.0},
'create_stats': {'cumtime': 0.0,
'file_name': '/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/cProfile.py',
'line_number': 50,
'ncalls': '1',
'percall_cumtime': 0.0,
'percall_tottime': 0.0,
'tottime': 0.0},
'fib': {'cumtime': 0.0,
'file_name': 'get_profile_dict.py',
'line_number': 5,
'ncalls': '15/1',
'percall_cumtime': 0.0,
'percall_tottime': 0.0,
'tottime': 0.0},
'total_tt': 0.0}
```
As an example, this can be used to generate a stacked column chart using various visualization tools which will assist in easily identifying program bottlenecks.
https://bugs.python.org/issue37958
Automerge-Triggered-By: @gpshead
The new option in the CLI of the profile module allow to profile
executable modules. This change follows the same implementation as the
one already present in `cProfile`.
As the argument is now present on both modules, move the tests to the
common test case to be run with profile as well.
bpo-31803: time.clock() and time.get_clock_info('clock') now emit a
DeprecationWarning warning.
Replace time.clock() with time.perf_counter() in tests and demos.
Remove also hasattr(time, 'monotonic') in test_time since time.monotonic()
is now always available since Python 3.5.
