Merged revisions 69364-69365,69409-69410,69413,69417,69435,69442,69447,69495,69519-69521 via svnmerge from

svn+ssh://pythondev@svn.python.org/python/trunk

........
  r69364 | kristjan.jonsson | 2009-02-06 04:17:34 -0600 (Fri, 06 Feb 2009) | 1 line

  Fix a number of Win32ErrorTests error cases.  chmod wasn't being tested.  'access' never raises an error.
........
  r69365 | armin.rigo | 2009-02-06 05:46:26 -0600 (Fri, 06 Feb 2009) | 2 lines

  Ivan on IRC in #twisted reported this crasher.
........
  r69409 | georg.brandl | 2009-02-07 06:21:17 -0600 (Sat, 07 Feb 2009) | 1 line

  #5174: fix wrong file closing in example.
........
  r69410 | neil.schemenauer | 2009-02-07 08:53:31 -0600 (Sat, 07 Feb 2009) | 4 lines

  Fix broken test in test_hotshot.  Treating the current directory as an
  empty file is sloppy and non-portable.  Use NamedTemporaryFile to make
  an empty file.
........
  r69413 | neil.schemenauer | 2009-02-07 12:35:16 -0600 (Sat, 07 Feb 2009) | 2 lines

  Add test for issue #999042, explict global statement works.
........
  r69417 | benjamin.peterson | 2009-02-07 17:01:19 -0600 (Sat, 07 Feb 2009) | 1 line

  document individual 2to3 fixers
........
  r69435 | benjamin.peterson | 2009-02-08 08:38:13 -0600 (Sun, 08 Feb 2009) | 1 line

  document numliterals fixer
........
  r69442 | benjamin.peterson | 2009-02-08 09:14:57 -0600 (Sun, 08 Feb 2009) | 1 line

  a few edits and typos
........
  r69447 | vinay.sajip | 2009-02-08 13:06:08 -0600 (Sun, 08 Feb 2009) | 2 lines

  Issue #5170: Fixed Unicode output bug in logging and added test case. This is a regression which did not occur in 2.5.
........
  r69495 | kristjan.jonsson | 2009-02-10 07:32:24 -0600 (Tue, 10 Feb 2009) | 1 line

  Issue 4804.  Add a function to test the validity of file descriptors on Windows, and stop using global runtime settings to silence the warnings / assertions.
........
  r69519 | gregory.p.smith | 2009-02-11 17:45:25 -0600 (Wed, 11 Feb 2009) | 3 lines

  Issue #1008086: Fixes socket.inet_aton() to always return 4 bytes even
  on LP64 platforms (most 64-bit Linux, bsd, unix systems).
........
  r69520 | benjamin.peterson | 2009-02-11 21:50:00 -0600 (Wed, 11 Feb 2009) | 1 line

  os.fsync() should be used to ensure that data is written to disk
........
  r69521 | benjamin.peterson | 2009-02-11 22:17:04 -0600 (Wed, 11 Feb 2009) | 1 line

  no need for this __bases__ trick anymore
........

Doc/library/2to3.rst

@@ -14,6 +14,8 @@ adapted to custom applications in which Python code needs to be edited
 automatically.
 
 
+.. _2to3-using:
+
 Using 2to3
 ----------
 
@@ -52,10 +54,10 @@ After transformation, :file:`example.py` looks like this::
 
 Comments and exact indentation are preserved throughout the translation process.
 
-By default, 2to3 runs a set of predefined fixers.  The :option:`-l` flag lists
-all available fixers.  An explicit set of fixers to run can be given with
-:option:`-f`.  Likewise the :option:`-x` explicitly disables a fixer.  The
-following example runs only the ``imports`` and ``has_key`` fixers::
+By default, 2to3 runs a set of :ref:`predefined fixers <2to3-fixers>`.  The
+:option:`-l` flag lists all available fixers.  An explicit set of fixers to run
+can be given with :option:`-f`.  Likewise the :option:`-x` explicitly disables a
+fixer.  The following example runs only the ``imports`` and ``has_key`` fixers::
 
    $ 2to3 -f imports -f has_key example.py
 
@@ -84,12 +86,263 @@ document could also be refactored with this option.
 The :option:`-v` option enables output of more information on the translation
 process.
 
-When the :option:`-p` is passed, 2to3 treats ``print`` as a function instead of
-a statement.  This is useful when ``from __future__ import print_function`` is
-being used.  If this option is not given, the print fixer will surround print
-calls in an extra set of parentheses because it cannot differentiate between the
-print statement with parentheses (such as ``print ("a" + "b" + "c")``) and a
-true function call.
+When the :option:`-p` is passed, the :2to3fixer:`print` fixer ``print`` as a
+function instead of a statement.  This is useful when ``from __future__ import
+print_function`` is being used.  If this option is not given, the print fixer
+will surround print calls in an extra set of parentheses because it cannot
+differentiate between the print statement with parentheses (such as ``print
+("a" + "b" + "c")``) and a true function call.
+
+
+.. _2to3-fixers:
+
+Fixers
+------
+
+Each step of tranforming code is encapsulated in a fixer.  The command ``2to3
+-l`` lists them.  As :ref:`documented above <2to3-using>`, each can be turned on
+and off individually.  They are described here in more detail.
+
+
+.. 2to3fixer:: apply
+
+   Removes usage of :func:`apply`.  For example ``apply(function, *args,
+   **kwargs)`` is converted to ``function(*args, **kwargs)``.
+
+.. 2to3fixer:: basestring
+
+   Converts :class:`basestring` to :class:`str`.
+
+.. 2to3fixer:: buffer
+
+   Converts :class:`buffer` to :class:`memoryview`.  This fixer is optional
+   because the :class:`memoryview` API is similar but not exactly the same as
+   that of :class:`buffer`.
+
+.. 2to3fixer:: callable
+
+   Converts ``callable(x)`` to ``hasattr(x, "__call_")``.
+
+.. 2to3fixer:: dict
+
+   Fixes dictionary iteration methods.  :meth:`dict.iteritems` is converted to
+   :meth:`dict.items`, :meth:`dict.iterkeys` to :meth:`dict.keys`, and
+   :meth:`dict.itervalues` to :meth:`dict.values`.  It also wraps existing
+   usages of :meth:`dict.items`, :meth:`dict.keys`, and :meth:`dict.values` in a
+   call to :class:`list`.
+
+.. 2to3fixer:: except
+
+   Converts ``except X, T`` to ``except X as T``.
+
+.. 2to3fixer:: exec
+
+   Converts the :keyword:`exec` statement to the :func:`exec` function.
+
+.. 2to3fixer:: execfile
+
+   Removes usage of :func:`execfile`.  The argument to :func:`execfile` is
+   wrapped in calls to :func:`open`, :func:`compile`, and :func:`exec`.
+
+.. 2to3fixer:: filter
+
+   Wraps :func:`filter` usage in a :class:`list` call.
+
+.. 2to3fixer:: funcattrs
+
+   Fixes function attributes that have been renamed.  For example,
+   ``my_function.func_closure`` is converted to ``my_function.__closure__``.
+
+.. 2to3fixer:: future
+
+   Removes ``from __future__ import new_feature`` statements.
+
+.. 2to3fixer:: getcwdu
+
+   Renames :func:`os.getcwdu` to :func:`os.getcwd`.
+
+.. 2to3fixer:: has_key
+
+   Changes ``dict.has_key(key)`` to ``key in dict``.
+
+.. 2to3fixer:: idioms
+
+   This optional fixer preforms several transformations that make Python code
+   more idiomatic.  Type comparisions like ``type(x) is SomeClass`` and
+   ``type(x) == SomeClass`` are converted to ``isinstance(x, SomeClass)``.
+   ``while 1`` becomes ``while True``.  This fixer also tries to make use of
+   :func:`sorted` in appropiate places.  For example, this block ::
+
+       L = list(some_iterable)
+       L.sort()
+
+   is changed to ::
+
+      L = sorted(some_iterable)
+
+.. 2to3fixer:: import
+
+   Detects sibling imports and converts them to relative imports.
+
+.. 2to3fixer:: imports
+
+   Handles module renames in the standard library.
+
+.. 2to3fixer:: imports2
+
+   Handles other modules renames in the standard library.  It is separate from
+   the :2to3fixer:`imports` fixer only because of technical limitations.
+
+.. 2to3fixer:: input
+
+   Converts ``input(prompt)`` to ``eval(input(prompt))``
+
+.. 2to3fixer:: intern
+
+   Converts :func:`intern` to :func:`sys.intern`.
+
+.. 2to3fixer:: isinstance
+
+   Fixes duplicate types in the second argument of :func:`isinstance`.  For
+   example, ``isinstance(x, (int, int))`` is converted to ``isinstance(x,
+   (int))``.
+
+.. 2to3fixer:: itertools_imports
+
+   Removes imports of :func:`itertools.ifilter`, :func:`itertools.izip`, and
+   :func:`itertools.imap`.  Imports of :func:`itertools.ifilterfalse` are also
+   changed to :func:`itertools.filterfalse`.
+
+.. 2to3fixer:: itertools
+
+   Changes usage of :func:`itertools.ifilter`, :func:`itertools.izip`, and
+   :func:`itertools.imap` to their builtin equivalents.
+   :func:`itertools.ifilterfalse` is changed to :func:`itertools.filterfalse`.
+
+.. 2to3fixer:: long
+
+   Strips the ``L`` prefix on long literals and renames :class:`long` to
+   :class:`int`.
+
+.. 2to3fixer:: map
+
+   Wraps :func:`map` in a :class:`list` call.  It also changes ``map(None, x)``
+   to ``list(x)``.  Using ``from future_builtins import map`` disables this
+   fixer.
+
+.. 2to3fixer:: metaclass
+
+   Converts the old metaclass syntax (``__metaclass__ = Meta`` in the class
+   body) to the new (``class X(metaclass=Meta)``).
+
+.. 2to3fixer:: methodattrs
+
+   Fixes old method attribute names.  For example, ``meth.im_func`` is converted
+   to ``meth.__func__``.
+
+.. 2to3fixer:: ne
+
+   Converts the old not-equal syntax, ``<>``, to ``!=``.
+
+.. 2to3fixer:: next
+
+   Converts the use of iterator's :meth:`next` methods to the :func:`next`
+   function.  It also renames :meth:`next` methods to :meth:`~object.__next__`.
+
+.. 2to3fixer:: nonzero
+
+   Renames :meth:`~object.__nonzero__` to :meth:`~object.__bool__`.
+
+.. 2to3fixer:: numliterals
+
+   Converts octal literals into the new syntax.
+
+.. 2to3fixer:: paren
+
+   Add extra parenthesis where they are required in list comprehensions.  For
+   example, ``[x for x in 1, 2]`` becomes ``[x for x in (1, 2)]``.
+
+.. 2to3fixer:: print
+
+   Converts the :keyword:`print` statement to the :func:`print` function.
+
+.. 2to3fixer:: raises
+
+   Converts ``raise E, V`` to ``raise E(V)``, and ``raise E, V, T`` to ``raise
+   E(V).with_traceback(T)``.  If ``E`` is a tuple, the translation will be
+   incorrect because substituting tuples for exceptions has been removed in 3.0.
+
+.. 2to3fixer:: raw_input
+
+   Converts :func:`raw_input` to :func:`input`.
+
+.. 2to3fixer:: reduce
+
+   Handles the move of :func:`reduce` to :func:`functools.reduce`.
+
+.. 2to3fixer:: renames
+
+   Changes :data:`sys.maxint` to :data:`sys.maxsize`.
+
+.. 2to3fixer:: repr
+
+   Replaces backtick repr with the :func:`repr` function.
+
+.. 2to3fixer:: set_literal
+
+   Replaces use of the :class:`set` constructor with set literals.  This fixer
+   is optional.
+
+.. 2to3fixer:: standard_error
+
+   Renames :exc:`StandardError` to :exc:`Exception`.
+
+.. 2to3fixer:: sys_exc
+
+   Changes the deprecated :data:`sys.exc_value`, :data:`sys.exc_type`,
+   :data:`sys.exc_traceback` to use :func:`sys.exc_info`.
+
+.. 2to3fixer:: throw
+
+   Fixes the API change in generator's :meth:`throw` method.
+
+.. 2to3fixer:: tuple_params
+
+   Removes implicit tuple parameter unpacking.  This fixer inserts temporary
+   variables.
+
+.. 2to3fixer:: types
+
+   Fixes code broken from the removal of some members in the :mod:`types`
+   module.
+
+.. 2to3fixer:: unicode
+
+   Renames :class:`unicode` to :class:`str`.
+
+.. 2to3fixer:: urllib
+
+   Handles the rename of :mod:`urllib` and :mod:`urllib2` to the :mod:`urllib`
+   package.
+
+.. 2to3fixer:: ws_comma
+
+   Removes excess whitespace from comma separated items.  This fixer is
+   optional.
+
+.. 2to3fixer:: xrange
+
+   Renames :func:`xrange` to :func:`range` and wraps existing :func:`range`
+   calls with :class:`list`.
+
+.. 2to3fixer:: xreadlines
+
+   Changes ``for x in file.xreadlines()`` to ``for x in file``.
+
+.. 2to3fixer:: zip
+
+   Wraps :func:`zip` usage in a :class:`list` call.  This is disabled when
+   ``from future_builtins import zip`` appears.
 
 
 :mod:`lib2to3` - 2to3's library

Doc/library/stdtypes.rst

@@ -2061,6 +2061,11 @@ Files have the following methods:
    Flush the internal buffer, like ``stdio``'s :cfunc:`fflush`.  This may be a
    no-op on some file-like objects.
 
+   .. note::
+
+      :meth:`flush` does not necessarily write the file's data to disk.  Use
+      :meth:`flush` followed by :func:`os.fsync` to ensure this behavior.
+
 
 .. method:: file.fileno()
 

Doc/tools/sphinxext/pyspecific.py

@@ -122,3 +122,4 @@ def setup(app):
     app.add_builder(suspicious.CheckSuspiciousMarkupBuilder)
     app.add_description_unit('opcode', 'opcode', '%s (opcode)',
                              parse_opcode_signature)
+    app.add_description_unit('2to3fixer', '2to3fixer', '%s (2to3 fixer)')

Include/fileobject.h

@@ -30,6 +30,17 @@ PyAPI_FUNC(int) _Py_SetFileSystemEncoding(PyObject *);
 PyAPI_FUNC(PyObject *) PyFile_NewStdPrinter(int);
 PyAPI_DATA(PyTypeObject) PyStdPrinter_Type;
 
+#if defined _MSC_VER && _MSC_VER >= 1400
+/* A routine to check if a file descriptor is valid on Windows.  Returns 0
+ * and sets errno to EBADF if it isn't.  This is to avoid Assertions
+ * from various functions in the Windows CRT beginning with
+ * Visual Studio 2005
+ */
+int _PyVerify_fd(int fd);
+#else
+#define _PyVerify_fd(A) (1) /* dummy */
+#endif
+
 #ifdef __cplusplus
 }
 #endif

Lib/logging/__init__.py

@@ -758,17 +758,19 @@ def emit(self, record):
         """
         try:
             msg = self.format(record)
+            stream = self.stream
             fs = "%s\n"
             if not _unicode: #if no unicode support...
-                self.stream.write(fs % msg)
+                stream.write(fs % msg)
             else:
                 try:
-                    if getattr(self.stream, 'encoding', None) is not None:
-                        self.stream.write(fs % msg.encode(self.stream.encoding))
+                    if (isinstance(msg, unicode) or
+                        getattr(stream, 'encoding', None) is None):
+                        stream.write(fs % msg)
                     else:
-                        self.stream.write(fs % msg)
+                        stream.write(fs % msg.encode(stream.encoding))
                 except UnicodeError:
-                    self.stream.write(fs % msg.encode("UTF-8"))
+                    stream.write(fs % msg.encode("UTF-8"))
             self.flush()
         except (KeyboardInterrupt, SystemExit):
             raise

Lib/test/crashers/compiler_recursion.py

@@ -0,0 +1,5 @@
+"""
+The compiler (>= 2.5) recurses happily.
+"""
+
+compile('()'*9**5, '?', 'exec')

Lib/test/test_logging.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 #
-# Copyright 2001-2004 by Vinay Sajip. All Rights Reserved.
+# Copyright 2001-2009 by Vinay Sajip. All Rights Reserved.
 #
 # Permission to use, copy, modify, and distribute this software and its
 # documentation for any purpose and without fee is hereby granted,
@@ -18,13 +18,14 @@
 
 """Test harness for the logging module. Run all tests.
 
-Copyright (C) 2001-2008 Vinay Sajip. All Rights Reserved.
+Copyright (C) 2001-2009 Vinay Sajip. All Rights Reserved.
 """
 
 import logging
 import logging.handlers
 import logging.config
 
+import codecs
 import copy
 import pickle
 import io
@@ -860,6 +861,7 @@ def test_persistent_loggers(self):
             ('foo', 'DEBUG', '3'),
         ])
 
+
 class EncodingTest(BaseTest):
     def test_encoding_plain_file(self):
         # In Python 2.x, a plain file object is treated as having no encoding.
@@ -886,6 +888,27 @@ def test_encoding_plain_file(self):
             if os.path.isfile(fn):
                 os.remove(fn)
 
+    def test_encoding_cyrillic_unicode(self):
+        log = logging.getLogger("test")
+        #Get a message in Unicode: Do svidanya in Cyrillic (meaning goodbye)
+        message = '\u0434\u043e \u0441\u0432\u0438\u0434\u0430\u043d\u0438\u044f'
+        #Ensure it's written in a Cyrillic encoding
+        writer_class = codecs.getwriter('cp1251')
+        stream = io.BytesIO()
+        writer = writer_class(stream, 'strict')
+        handler = logging.StreamHandler(writer)
+        log.addHandler(handler)
+        try:
+            log.warning(message)
+        finally:
+            log.removeHandler(handler)
+            handler.close()
+        # check we wrote exactly those bytes, ignoring trailing \n etc
+        s = stream.getvalue()
+        #Compare against what the data should be when encoded in CP-1251
+        self.assertEqual(s, b'\xe4\xee \xf1\xe2\xe8\xe4\xe0\xed\xe8\xff\n')
+
+
 class WarningsTest(BaseTest):
     def test_warnings(self):
         logging.captureWarnings(True)

Lib/test/test_os.py

@@ -576,16 +576,18 @@ def test_chdir(self):
         self.assertRaises(WindowsError, os.chdir, support.TESTFN)
 
     def test_mkdir(self):
-        self.assertRaises(WindowsError, os.chdir, support.TESTFN)
+        f = open(test_support.TESTFN, "w")
+        try:
+            self.assertRaises(WindowsError, os.mkdir, support.TESTFN)
+        finally:
+            f.close()
+            os.unlink(test_support.TESTFN)
 
     def test_utime(self):
         self.assertRaises(WindowsError, os.utime, support.TESTFN, None)
 
-    def test_access(self):
-        self.assertRaises(WindowsError, os.utime, support.TESTFN, 0)
-
     def test_chmod(self):
-        self.assertRaises(WindowsError, os.utime, support.TESTFN, 0)
+        self.assertRaises(WindowsError, os.chmod, support.TESTFN, 0)
 
 class TestInvalidFD(unittest.TestCase):
     singles = ["fchdir", "dup", "fdopen", "fdatasync", "fstat",

Lib/test/test_socket.py

@@ -398,6 +398,14 @@ def testDefaultTimeout(self):
         # Check that setting it to an invalid type raises TypeError
         self.assertRaises(TypeError, socket.setdefaulttimeout, "spam")
 
+    def testIPv4_inet_aton_fourbytes(self):
+        if not hasattr(socket, 'inet_aton'):
+            return  # No inet_aton, nothing to check
+        # Test that issue1008086 and issue767150 are fixed.
+        # It must return 4 bytes.
+        self.assertEquals(b'\x00'*4, socket.inet_aton('0.0.0.0'))
+        self.assertEquals(b'\xff'*4, socket.inet_aton('255.255.255.255'))
+
     def testIPv4toString(self):
         if not hasattr(socket, 'inet_pton'):
             return # No inet_pton() on this platform

Modules/_fileio.c

@@ -146,7 +146,7 @@ check_fd(int fd)
 {
 #if defined(HAVE_FSTAT)
 	struct stat buf;
-	if (fstat(fd, &buf) < 0 && errno == EBADF) {
+	if (!_PyVerify_fd(fd) || (fstat(fd, &buf) < 0 && errno == EBADF)) {
 		PyObject *exc;
 		char *msg = strerror(EBADF);
 		exc = PyObject_CallFunction(PyExc_OSError, "(is)",

Modules/socketmodule.c

@@ -3567,8 +3567,11 @@ socket_inet_aton(PyObject *self, PyObject *args)
 #endif
 
 #if !defined(HAVE_INET_ATON) || defined(USE_INET_ATON_WEAKLINK)
+#if (SIZEOF_INT != 4)
+#error "Not sure if in_addr_t exists and int is not 32-bits."
+#endif
 	/* Have to use inet_addr() instead */
-	unsigned long packed_addr;
+	unsigned int packed_addr;
 #endif
 	char *ip_addr;
 
@@ -5090,7 +5093,10 @@ int
 inet_pton(int af, const char *src, void *dst)
 {
 	if (af == AF_INET) {
-		long packed_addr;
+#if (SIZEOF_INT != 4)
+#error "Not sure if in_addr_t exists and int is not 32-bits."
+#endif
+		unsigned int packed_addr;
 		packed_addr = inet_addr(src);
 		if (packed_addr == INADDR_NONE)
 			return 0;

Objects/exceptions.c

@@ -1776,28 +1776,6 @@ PyObject *PyExc_RecursionErrorInst = NULL;
     if (PyDict_SetItemString(bdict, # TYPE, PyExc_ ## TYPE)) \
         Py_FatalError("Module dictionary insertion problem.");
 
-#if defined _MSC_VER && _MSC_VER >= 1400 && defined(__STDC_SECURE_LIB__)
-/* crt variable checking in VisualStudio .NET 2005 */
-#include <crtdbg.h>
-
-static int	prevCrtReportMode;
-static _invalid_parameter_handler	prevCrtHandler;
-
-/* Invalid parameter handler.  Sets a ValueError exception */
-static void
-InvalidParameterHandler(
-    const wchar_t * expression,
-    const wchar_t * function,
-    const wchar_t * file,
-    unsigned int line,
-    uintptr_t pReserved)
-{
-    /* Do nothing, allow execution to continue.  Usually this
-     * means that the CRT will set errno to EINVAL
-     */
-}
-#endif
-
 
 void
 _PyExc_Init(void)
@@ -1949,13 +1927,6 @@ _PyExc_Init(void)
     }
 
     Py_DECREF(bltinmod);
-
-#if defined _MSC_VER && _MSC_VER >= 1400 && defined(__STDC_SECURE_LIB__)
-    /* Set CRT argument error handler */
-    prevCrtHandler = _set_invalid_parameter_handler(InvalidParameterHandler);
-    /* turn off assertions in debug mode */
-    prevCrtReportMode = _CrtSetReportMode(_CRT_ASSERT, 0);
-#endif
 }
 
 void
@@ -1963,9 +1934,4 @@ _PyExc_Fini(void)
 {
     Py_XDECREF(PyExc_MemoryErrorInst);
     PyExc_MemoryErrorInst = NULL;
-#if defined _MSC_VER && _MSC_VER >= 1400 && defined(__STDC_SECURE_LIB__)
-    /* reset CRT error handling */
-    _set_invalid_parameter_handler(prevCrtHandler);
-    _CrtSetReportMode(_CRT_ASSERT, prevCrtReportMode);
-#endif
 }