mirror of
https://github.com/python/cpython.git
synced 2025-10-30 21:21:22 +00:00
218 lines
7.9 KiB
ReStructuredText
218 lines
7.9 KiB
ReStructuredText
:mod:`packaging.pypi.simple` --- Crawler using the PyPI "simple" interface
|
|
==========================================================================
|
|
|
|
.. module:: packaging.pypi.simple
|
|
:synopsis: Crawler using the screen-scraping "simple" interface to fetch info
|
|
and distributions.
|
|
|
|
|
|
The class provided by :mod:`packaging.pypi.simple` can access project indexes
|
|
and provide useful information about distributions. PyPI, other indexes and
|
|
local indexes are supported.
|
|
|
|
You should use this module to search distributions by name and versions, process
|
|
index external pages and download distributions. It is not suited for things
|
|
that will end up in too long index processing (like "finding all distributions
|
|
with a specific version, no matter the name"); use :mod:`packaging.pypi.xmlrpc`
|
|
for that.
|
|
|
|
|
|
API
|
|
---
|
|
|
|
.. class:: Crawler(index_url=DEFAULT_SIMPLE_INDEX_URL, \
|
|
prefer_final=False, prefer_source=True, \
|
|
hosts=('*',), follow_externals=False, \
|
|
mirrors_url=None, mirrors=None, timeout=15, \
|
|
mirrors_max_tries=0)
|
|
|
|
*index_url* is the address of the index to use for requests.
|
|
|
|
The first two parameters control the query results. *prefer_final*
|
|
indicates whether a final version (not alpha, beta or candidate) is to be
|
|
preferred over a newer but non-final version (for example, whether to pick
|
|
up 1.0 over 2.0a3). It is used only for queries that don't give a version
|
|
argument. Likewise, *prefer_source* tells whether to prefer a source
|
|
distribution over a binary one, if no distribution argument was prodived.
|
|
|
|
Other parameters are related to external links (that is links that go
|
|
outside the simple index): *hosts* is a list of hosts allowed to be
|
|
processed if *follow_externals* is true (default behavior is to follow all
|
|
hosts), *follow_externals* enables or disables following external links
|
|
(default is false, meaning disabled).
|
|
|
|
The remaining parameters are related to the mirroring infrastructure
|
|
defined in :PEP:`381`. *mirrors_url* gives a URL to look on for DNS
|
|
records giving mirror adresses; *mirrors* is a list of mirror URLs (see
|
|
the PEP). If both *mirrors* and *mirrors_url* are given, *mirrors_url*
|
|
will only be used if *mirrors* is set to ``None``. *timeout* is the time
|
|
(in seconds) to wait before considering a URL has timed out;
|
|
*mirrors_max_tries"* is the number of times to try requesting informations
|
|
on mirrors before switching.
|
|
|
|
The following methods are defined:
|
|
|
|
.. method:: get_distributions(project_name, version)
|
|
|
|
Return the distributions found in the index for the given release.
|
|
|
|
.. method:: get_metadata(project_name, version)
|
|
|
|
Return the metadata found on the index for this project name and
|
|
version. Currently downloads and unpacks a distribution to read the
|
|
PKG-INFO file.
|
|
|
|
.. method:: get_release(requirements, prefer_final=None)
|
|
|
|
Return one release that fulfills the given requirements.
|
|
|
|
.. method:: get_releases(requirements, prefer_final=None, force_update=False)
|
|
|
|
Search for releases and return a
|
|
:class:`~packaging.pypi.dist.ReleasesList` object containing the
|
|
results.
|
|
|
|
.. method:: search_projects(name=None)
|
|
|
|
Search the index for projects containing the given name and return a
|
|
list of matching names.
|
|
|
|
See also the base class :class:`packaging.pypi.base.BaseClient` for inherited
|
|
methods.
|
|
|
|
|
|
.. data:: DEFAULT_SIMPLE_INDEX_URL
|
|
|
|
The address used by default by the crawler class. It is currently
|
|
``'http://a.pypi.python.org/simple/'``, the main PyPI installation.
|
|
|
|
|
|
|
|
|
|
Usage Examples
|
|
---------------
|
|
|
|
To help you understand how using the `Crawler` class, here are some basic
|
|
usages.
|
|
|
|
Request the simple index to get a specific distribution
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Supposing you want to scan an index to get a list of distributions for
|
|
the "foobar" project. You can use the "get_releases" method for that.
|
|
The get_releases method will browse the project page, and return
|
|
:class:`ReleaseInfo` objects for each found link that rely on downloads. ::
|
|
|
|
>>> from packaging.pypi.simple import Crawler
|
|
>>> crawler = Crawler()
|
|
>>> crawler.get_releases("FooBar")
|
|
[<ReleaseInfo "Foobar 1.1">, <ReleaseInfo "Foobar 1.2">]
|
|
|
|
|
|
Note that you also can request the client about specific versions, using version
|
|
specifiers (described in `PEP 345
|
|
<http://www.python.org/dev/peps/pep-0345/#version-specifiers>`_)::
|
|
|
|
>>> client.get_releases("FooBar < 1.2")
|
|
[<ReleaseInfo "FooBar 1.1">, ]
|
|
|
|
|
|
`get_releases` returns a list of :class:`ReleaseInfo`, but you also can get the
|
|
best distribution that fullfil your requirements, using "get_release"::
|
|
|
|
>>> client.get_release("FooBar < 1.2")
|
|
<ReleaseInfo "FooBar 1.1">
|
|
|
|
|
|
Download distributions
|
|
^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
As it can get the urls of distributions provided by PyPI, the `Crawler`
|
|
client also can download the distributions and put it for you in a temporary
|
|
destination::
|
|
|
|
>>> client.download("foobar")
|
|
/tmp/temp_dir/foobar-1.2.tar.gz
|
|
|
|
|
|
You also can specify the directory you want to download to::
|
|
|
|
>>> client.download("foobar", "/path/to/my/dir")
|
|
/path/to/my/dir/foobar-1.2.tar.gz
|
|
|
|
|
|
While downloading, the md5 of the archive will be checked, if not matches, it
|
|
will try another time, then if fails again, raise `MD5HashDoesNotMatchError`.
|
|
|
|
Internally, that's not the Crawler which download the distributions, but the
|
|
`DistributionInfo` class. Please refer to this documentation for more details.
|
|
|
|
|
|
Following PyPI external links
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
The default behavior for packaging is to *not* follow the links provided
|
|
by HTML pages in the "simple index", to find distributions related
|
|
downloads.
|
|
|
|
It's possible to tell the PyPIClient to follow external links by setting the
|
|
`follow_externals` attribute, on instantiation or after::
|
|
|
|
>>> client = Crawler(follow_externals=True)
|
|
|
|
or ::
|
|
|
|
>>> client = Crawler()
|
|
>>> client.follow_externals = True
|
|
|
|
|
|
Working with external indexes, and mirrors
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
The default `Crawler` behavior is to rely on the Python Package index stored
|
|
on PyPI (http://pypi.python.org/simple).
|
|
|
|
As you can need to work with a local index, or private indexes, you can specify
|
|
it using the index_url parameter::
|
|
|
|
>>> client = Crawler(index_url="file://filesystem/path/")
|
|
|
|
or ::
|
|
|
|
>>> client = Crawler(index_url="http://some.specific.url/")
|
|
|
|
|
|
You also can specify mirrors to fallback on in case the first index_url you
|
|
provided doesnt respond, or not correctly. The default behavior for
|
|
`Crawler` is to use the list provided by Python.org DNS records, as
|
|
described in the :PEP:`381` about mirroring infrastructure.
|
|
|
|
If you don't want to rely on these, you could specify the list of mirrors you
|
|
want to try by specifying the `mirrors` attribute. It's a simple iterable::
|
|
|
|
>>> mirrors = ["http://first.mirror","http://second.mirror"]
|
|
>>> client = Crawler(mirrors=mirrors)
|
|
|
|
|
|
Searching in the simple index
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
It's possible to search for projects with specific names in the package index.
|
|
Assuming you want to find all projects containing the "distutils" keyword::
|
|
|
|
>>> c.search_projects("distutils")
|
|
[<Project "collective.recipe.distutils">, <Project "Distutils">, <Project
|
|
"Packaging">, <Project "distutilscross">, <Project "lpdistutils">, <Project
|
|
"taras.recipe.distutils">, <Project "zerokspot.recipe.distutils">]
|
|
|
|
|
|
You can also search the projects starting with a specific text, or ending with
|
|
that text, using a wildcard::
|
|
|
|
>>> c.search_projects("distutils*")
|
|
[<Project "Distutils">, <Project "Packaging">, <Project "distutilscross">]
|
|
|
|
>>> c.search_projects("*distutils")
|
|
[<Project "collective.recipe.distutils">, <Project "Distutils">, <Project
|
|
"lpdistutils">, <Project "taras.recipe.distutils">, <Project
|
|
"zerokspot.recipe.distutils">]
|