mirror of
https://github.com/python/cpython.git
synced 2025-10-31 05:31:20 +00:00
474458da48
BOM_UTF32, BOM_UTF32_LE and BOM_UTF32_BE that represent the Byte Order Mark in UTF-8, UTF-16 and UTF-32 encodings for little and big endian systems. The old names BOM32_* and BOM64_* were off by a factor of 2. This closes SF bug http://www.python.org/sf/555360
446 lines
17 KiB
TeX
446 lines
17 KiB
TeX
\section{\module{codecs} ---
|
|
Codec registry and base classes}
|
|
|
|
\declaremodule{standard}{codecs}
|
|
\modulesynopsis{Encode and decode data and streams.}
|
|
\moduleauthor{Marc-Andre Lemburg}{mal@lemburg.com}
|
|
\sectionauthor{Marc-Andre Lemburg}{mal@lemburg.com}
|
|
|
|
|
|
\index{Unicode}
|
|
\index{Codecs}
|
|
\indexii{Codecs}{encode}
|
|
\indexii{Codecs}{decode}
|
|
\index{streams}
|
|
\indexii{stackable}{streams}
|
|
|
|
|
|
This module defines base classes for standard Python codecs (encoders
|
|
and decoders) and provides access to the internal Python codec
|
|
registry which manages the codec lookup process.
|
|
|
|
It defines the following functions:
|
|
|
|
\begin{funcdesc}{register}{search_function}
|
|
Register a codec search function. Search functions are expected to
|
|
take one argument, the encoding name in all lower case letters, and
|
|
return a tuple of functions \code{(\var{encoder}, \var{decoder}, \var{stream_reader},
|
|
\var{stream_writer})} taking the following arguments:
|
|
|
|
\var{encoder} and \var{decoder}: These must be functions or methods
|
|
which have the same interface as the
|
|
\method{encode()}/\method{decode()} methods of Codec instances (see
|
|
Codec Interface). The functions/methods are expected to work in a
|
|
stateless mode.
|
|
|
|
\var{stream_reader} and \var{stream_writer}: These have to be
|
|
factory functions providing the following interface:
|
|
|
|
\code{factory(\var{stream}, \var{errors}='strict')}
|
|
|
|
The factory functions must return objects providing the interfaces
|
|
defined by the base classes \class{StreamWriter} and
|
|
\class{StreamReader}, respectively. Stream codecs can maintain
|
|
state.
|
|
|
|
Possible values for errors are \code{'strict'} (raise an exception
|
|
in case of an encoding error), \code{'replace'} (replace malformed
|
|
data with a suitable replacement marker, such as \character{?}) and
|
|
\code{'ignore'} (ignore malformed data and continue without further
|
|
notice).
|
|
|
|
In case a search function cannot find a given encoding, it should
|
|
return \code{None}.
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{lookup}{encoding}
|
|
Looks up a codec tuple in the Python codec registry and returns the
|
|
function tuple as defined above.
|
|
|
|
Encodings are first looked up in the registry's cache. If not found,
|
|
the list of registered search functions is scanned. If no codecs tuple
|
|
is found, a \exception{LookupError} is raised. Otherwise, the codecs
|
|
tuple is stored in the cache and returned to the caller.
|
|
\end{funcdesc}
|
|
|
|
To simplify access to the various codecs, the module provides these
|
|
additional functions which use \function{lookup()} for the codec
|
|
lookup:
|
|
|
|
\begin{funcdesc}{getencoder}{encoding}
|
|
Lookup up the codec for the given encoding and return its encoder
|
|
function.
|
|
|
|
Raises a \exception{LookupError} in case the encoding cannot be found.
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{getdecoder}{encoding}
|
|
Lookup up the codec for the given encoding and return its decoder
|
|
function.
|
|
|
|
Raises a \exception{LookupError} in case the encoding cannot be found.
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{getreader}{encoding}
|
|
Lookup up the codec for the given encoding and return its StreamReader
|
|
class or factory function.
|
|
|
|
Raises a \exception{LookupError} in case the encoding cannot be found.
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{getwriter}{encoding}
|
|
Lookup up the codec for the given encoding and return its StreamWriter
|
|
class or factory function.
|
|
|
|
Raises a \exception{LookupError} in case the encoding cannot be found.
|
|
\end{funcdesc}
|
|
|
|
To simplify working with encoded files or stream, the module
|
|
also defines these utility functions:
|
|
|
|
\begin{funcdesc}{open}{filename, mode\optional{, encoding\optional{,
|
|
errors\optional{, buffering}}}}
|
|
Open an encoded file using the given \var{mode} and return
|
|
a wrapped version providing transparent encoding/decoding.
|
|
|
|
\note{The wrapped version will only accept the object format
|
|
defined by the codecs, i.e.\ Unicode objects for most built-in
|
|
codecs. Output is also codec-dependent and will usually be Unicode as
|
|
well.}
|
|
|
|
\var{encoding} specifies the encoding which is to be used for the
|
|
the file.
|
|
|
|
\var{errors} may be given to define the error handling. It defaults
|
|
to \code{'strict'} which causes a \exception{ValueError} to be raised
|
|
in case an encoding error occurs.
|
|
|
|
\var{buffering} has the same meaning as for the built-in
|
|
\function{open()} function. It defaults to line buffered.
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{EncodedFile}{file, input\optional{,
|
|
output\optional{, errors}}}
|
|
Return a wrapped version of file which provides transparent
|
|
encoding translation.
|
|
|
|
Strings written to the wrapped file are interpreted according to the
|
|
given \var{input} encoding and then written to the original file as
|
|
strings using the \var{output} encoding. The intermediate encoding will
|
|
usually be Unicode but depends on the specified codecs.
|
|
|
|
If \var{output} is not given, it defaults to \var{input}.
|
|
|
|
\var{errors} may be given to define the error handling. It defaults to
|
|
\code{'strict'}, which causes \exception{ValueError} to be raised in case
|
|
an encoding error occurs.
|
|
\end{funcdesc}
|
|
|
|
The module also provides the following constants which are useful
|
|
for reading and writing to platform dependent files:
|
|
|
|
\begin{datadesc}{BOM}
|
|
\dataline{BOM_BE}
|
|
\dataline{BOM_LE}
|
|
\dataline{BOM_UTF8}
|
|
\dataline{BOM_UTF16}
|
|
\dataline{BOM_UTF16_BE}
|
|
\dataline{BOM_UTF16_LE}
|
|
\dataline{BOM_UTF32}
|
|
\dataline{BOM_UTF32_BE}
|
|
\dataline{BOM_UTF32_LE}
|
|
These constants define various encodings of the Unicode byte order mark
|
|
(BOM) used in UTF-16 and UTF-32 data streams to indicate the byte order
|
|
used in the stream or file and in UTF-8 as a Unicode signature.
|
|
\constant{BOM_UTF16} is either \constant{BOM_UTF16_BE} or
|
|
\constant{BOM_UTF16_LE} depending on the platform's native byte order,
|
|
\constant{BOM} is an alias for \constant{BOM_UTF16}, \constant{BOM_LE}
|
|
for \constant{BOM_UTF16_LE} and \constant{BOM_BE} for \constant{BOM_UTF16_BE}.
|
|
The others represent the BOM in UTF-8 and UTF-32 encodings.
|
|
\end{datadesc}
|
|
|
|
|
|
\begin{seealso}
|
|
\seeurl{http://sourceforge.net/projects/python-codecs/}{A
|
|
SourceForge project working on additional support for Asian
|
|
codecs for use with Python. They are in the early stages of
|
|
development at the time of this writing --- look in their
|
|
FTP area for downloadable files.}
|
|
\end{seealso}
|
|
|
|
|
|
\subsection{Codec Base Classes}
|
|
|
|
The \module{codecs} defines a set of base classes which define the
|
|
interface and can also be used to easily write you own codecs for use
|
|
in Python.
|
|
|
|
Each codec has to define four interfaces to make it usable as codec in
|
|
Python: stateless encoder, stateless decoder, stream reader and stream
|
|
writer. The stream reader and writers typically reuse the stateless
|
|
encoder/decoder to implement the file protocols.
|
|
|
|
The \class{Codec} class defines the interface for stateless
|
|
encoders/decoders.
|
|
|
|
To simplify and standardize error handling, the \method{encode()} and
|
|
\method{decode()} methods may implement different error handling
|
|
schemes by providing the \var{errors} string argument. The following
|
|
string values are defined and implemented by all standard Python
|
|
codecs:
|
|
|
|
\begin{tableii}{l|l}{code}{Value}{Meaning}
|
|
\lineii{'strict'}{Raise \exception{ValueError} (or a subclass);
|
|
this is the default.}
|
|
\lineii{'ignore'}{Ignore the character and continue with the next.}
|
|
\lineii{'replace'}{Replace with a suitable replacement character;
|
|
Python will use the official U+FFFD REPLACEMENT
|
|
CHARACTER for the built-in Unicode codecs.}
|
|
\end{tableii}
|
|
|
|
|
|
\subsubsection{Codec Objects \label{codec-objects}}
|
|
|
|
The \class{Codec} class defines these methods which also define the
|
|
function interfaces of the stateless encoder and decoder:
|
|
|
|
\begin{methoddesc}{encode}{input\optional{, errors}}
|
|
Encodes the object \var{input} and returns a tuple (output object,
|
|
length consumed). While codecs are not restricted to use with Unicode, in
|
|
a Unicode context, encoding converts a Unicode object to a plain string
|
|
using a particular character set encoding (e.g., \code{cp1252} or
|
|
\code{iso-8859-1}).
|
|
|
|
\var{errors} defines the error handling to apply. It defaults to
|
|
\code{'strict'} handling.
|
|
|
|
The method may not store state in the \class{Codec} instance. Use
|
|
\class{StreamCodec} for codecs which have to keep state in order to
|
|
make encoding/decoding efficient.
|
|
|
|
The encoder must be able to handle zero length input and return an
|
|
empty object of the output object type in this situation.
|
|
\end{methoddesc}
|
|
|
|
\begin{methoddesc}{decode}{input\optional{, errors}}
|
|
Decodes the object \var{input} and returns a tuple (output object,
|
|
length consumed). In a Unicode context, decoding converts a plain string
|
|
encoded using a particular character set encoding to a Unicode object.
|
|
|
|
\var{input} must be an object which provides the \code{bf_getreadbuf}
|
|
buffer slot. Python strings, buffer objects and memory mapped files
|
|
are examples of objects providing this slot.
|
|
|
|
\var{errors} defines the error handling to apply. It defaults to
|
|
\code{'strict'} handling.
|
|
|
|
The method may not store state in the \class{Codec} instance. Use
|
|
\class{StreamCodec} for codecs which have to keep state in order to
|
|
make encoding/decoding efficient.
|
|
|
|
The decoder must be able to handle zero length input and return an
|
|
empty object of the output object type in this situation.
|
|
\end{methoddesc}
|
|
|
|
The \class{StreamWriter} and \class{StreamReader} classes provide
|
|
generic working interfaces which can be used to implement new
|
|
encodings submodules very easily. See \module{encodings.utf_8} for an
|
|
example on how this is done.
|
|
|
|
|
|
\subsubsection{StreamWriter Objects \label{stream-writer-objects}}
|
|
|
|
The \class{StreamWriter} class is a subclass of \class{Codec} and
|
|
defines the following methods which every stream writer must define in
|
|
order to be compatible to the Python codec registry.
|
|
|
|
\begin{classdesc}{StreamWriter}{stream\optional{, errors}}
|
|
Constructor for a \class{StreamWriter} instance.
|
|
|
|
All stream writers must provide this constructor interface. They are
|
|
free to add additional keyword arguments, but only the ones defined
|
|
here are used by the Python codec registry.
|
|
|
|
\var{stream} must be a file-like object open for writing (binary)
|
|
data.
|
|
|
|
The \class{StreamWriter} may implement different error handling
|
|
schemes by providing the \var{errors} keyword argument. These
|
|
parameters are defined:
|
|
|
|
\begin{itemize}
|
|
\item \code{'strict'} Raise \exception{ValueError} (or a subclass);
|
|
this is the default.
|
|
\item \code{'ignore'} Ignore the character and continue with the next.
|
|
\item \code{'replace'} Replace with a suitable replacement character
|
|
\end{itemize}
|
|
\end{classdesc}
|
|
|
|
\begin{methoddesc}{write}{object}
|
|
Writes the object's contents encoded to the stream.
|
|
\end{methoddesc}
|
|
|
|
\begin{methoddesc}{writelines}{list}
|
|
Writes the concatenated list of strings to the stream (possibly by
|
|
reusing the \method{write()} method).
|
|
\end{methoddesc}
|
|
|
|
\begin{methoddesc}{reset}{}
|
|
Flushes and resets the codec buffers used for keeping state.
|
|
|
|
Calling this method should ensure that the data on the output is put
|
|
into a clean state, that allows appending of new fresh data without
|
|
having to rescan the whole stream to recover state.
|
|
\end{methoddesc}
|
|
|
|
In addition to the above methods, the \class{StreamWriter} must also
|
|
inherit all other methods and attribute from the underlying stream.
|
|
|
|
|
|
\subsubsection{StreamReader Objects \label{stream-reader-objects}}
|
|
|
|
The \class{StreamReader} class is a subclass of \class{Codec} and
|
|
defines the following methods which every stream reader must define in
|
|
order to be compatible to the Python codec registry.
|
|
|
|
\begin{classdesc}{StreamReader}{stream\optional{, errors}}
|
|
Constructor for a \class{StreamReader} instance.
|
|
|
|
All stream readers must provide this constructor interface. They are
|
|
free to add additional keyword arguments, but only the ones defined
|
|
here are used by the Python codec registry.
|
|
|
|
\var{stream} must be a file-like object open for reading (binary)
|
|
data.
|
|
|
|
The \class{StreamReader} may implement different error handling
|
|
schemes by providing the \var{errors} keyword argument. These
|
|
parameters are defined:
|
|
|
|
\begin{itemize}
|
|
\item \code{'strict'} Raise \exception{ValueError} (or a subclass);
|
|
this is the default.
|
|
\item \code{'ignore'} Ignore the character and continue with the next.
|
|
\item \code{'replace'} Replace with a suitable replacement character.
|
|
\end{itemize}
|
|
\end{classdesc}
|
|
|
|
\begin{methoddesc}{read}{\optional{size}}
|
|
Decodes data from the stream and returns the resulting object.
|
|
|
|
\var{size} indicates the approximate maximum number of bytes to read
|
|
from the stream for decoding purposes. The decoder can modify this
|
|
setting as appropriate. The default value -1 indicates to read and
|
|
decode as much as possible. \var{size} is intended to prevent having
|
|
to decode huge files in one step.
|
|
|
|
The method should use a greedy read strategy meaning that it should
|
|
read as much data as is allowed within the definition of the encoding
|
|
and the given size, e.g. if optional encoding endings or state
|
|
markers are available on the stream, these should be read too.
|
|
\end{methoddesc}
|
|
|
|
\begin{methoddesc}{readline}{[size]}
|
|
Read one line from the input stream and return the
|
|
decoded data.
|
|
|
|
Unlike the \method{readlines()} method, this method inherits
|
|
the line breaking knowledge from the underlying stream's
|
|
\method{readline()} method -- there is currently no support for line
|
|
breaking using the codec decoder due to lack of line buffering.
|
|
Sublcasses should however, if possible, try to implement this method
|
|
using their own knowledge of line breaking.
|
|
|
|
\var{size}, if given, is passed as size argument to the stream's
|
|
\method{readline()} method.
|
|
\end{methoddesc}
|
|
|
|
\begin{methoddesc}{readlines}{[sizehint]}
|
|
Read all lines available on the input stream and return them as list
|
|
of lines.
|
|
|
|
Line breaks are implemented using the codec's decoder method and are
|
|
included in the list entries.
|
|
|
|
\var{sizehint}, if given, is passed as \var{size} argument to the
|
|
stream's \method{read()} method.
|
|
\end{methoddesc}
|
|
|
|
\begin{methoddesc}{reset}{}
|
|
Resets the codec buffers used for keeping state.
|
|
|
|
Note that no stream repositioning should take place. This method is
|
|
primarily intended to be able to recover from decoding errors.
|
|
\end{methoddesc}
|
|
|
|
In addition to the above methods, the \class{StreamReader} must also
|
|
inherit all other methods and attribute from the underlying stream.
|
|
|
|
The next two base classes are included for convenience. They are not
|
|
needed by the codec registry, but may provide useful in practice.
|
|
|
|
|
|
\subsubsection{StreamReaderWriter Objects \label{stream-reader-writer}}
|
|
|
|
The \class{StreamReaderWriter} allows wrapping streams which work in
|
|
both read and write modes.
|
|
|
|
The design is such that one can use the factory functions returned by
|
|
the \function{lookup()} function to construct the instance.
|
|
|
|
\begin{classdesc}{StreamReaderWriter}{stream, Reader, Writer, errors}
|
|
Creates a \class{StreamReaderWriter} instance.
|
|
\var{stream} must be a file-like object.
|
|
\var{Reader} and \var{Writer} must be factory functions or classes
|
|
providing the \class{StreamReader} and \class{StreamWriter} interface
|
|
resp.
|
|
Error handling is done in the same way as defined for the
|
|
stream readers and writers.
|
|
\end{classdesc}
|
|
|
|
\class{StreamReaderWriter} instances define the combined interfaces of
|
|
\class{StreamReader} and \class{StreamWriter} classes. They inherit
|
|
all other methods and attribute from the underlying stream.
|
|
|
|
|
|
\subsubsection{StreamRecoder Objects \label{stream-recoder-objects}}
|
|
|
|
The \class{StreamRecoder} provide a frontend - backend view of
|
|
encoding data which is sometimes useful when dealing with different
|
|
encoding environments.
|
|
|
|
The design is such that one can use the factory functions returned by
|
|
the \function{lookup()} function to construct the instance.
|
|
|
|
\begin{classdesc}{StreamRecoder}{stream, encode, decode,
|
|
Reader, Writer, errors}
|
|
Creates a \class{StreamRecoder} instance which implements a two-way
|
|
conversion: \var{encode} and \var{decode} work on the frontend (the
|
|
input to \method{read()} and output of \method{write()}) while
|
|
\var{Reader} and \var{Writer} work on the backend (reading and
|
|
writing to the stream).
|
|
|
|
You can use these objects to do transparent direct recodings from
|
|
e.g.\ Latin-1 to UTF-8 and back.
|
|
|
|
\var{stream} must be a file-like object.
|
|
|
|
\var{encode}, \var{decode} must adhere to the \class{Codec}
|
|
interface, \var{Reader}, \var{Writer} must be factory functions or
|
|
classes providing objects of the the \class{StreamReader} and
|
|
\class{StreamWriter} interface respectively.
|
|
|
|
\var{encode} and \var{decode} are needed for the frontend
|
|
translation, \var{Reader} and \var{Writer} for the backend
|
|
translation. The intermediate format used is determined by the two
|
|
sets of codecs, e.g. the Unicode codecs will use Unicode as
|
|
intermediate encoding.
|
|
|
|
Error handling is done in the same way as defined for the
|
|
stream readers and writers.
|
|
\end{classdesc}
|
|
|
|
\class{StreamRecoder} instances define the combined interfaces of
|
|
\class{StreamReader} and \class{StreamWriter} classes. They inherit
|
|
all other methods and attribute from the underlying stream.
|
|
|