CloseNotifier wasn't well specified previously. This CL simplifies its
implementation, clarifies the public documentation on CloseNotifier,
clarifies internal documentation on conn, and fixes two CloseNotifier
bugs in the process.
The main change, though, is tightening the rules and expectations for using
CloseNotifier:
* the caller must consume the Request.Body first (old rule, unwritten)
* the received value is the "true" value (old rule, unwritten)
* no promises for channel sends after Handler returns (old rule, unwritten)
* a subsequent pipelined request fires the CloseNotifier (new behavior;
previously it never fired and thus effectively deadlocked as in #13165)
* advise that it should only be used without HTTP/1.1 pipelining (use HTTP/2
or non-idempotent browsers). Not that browsers actually use pipelining.
The main implementation change is that each Handler now gets its own
CloseNotifier channel value, rather than sharing one between the whole
conn. This means Handlers can't affect subsequent requests. This is
how HTTP/2's Server works too. The old docs never clarified a behavior
either way. The other side effect of each request getting its own
CloseNotifier channel is that one handler can't "poison" the
underlying conn preventing subsequent requests on the same connection
from using CloseNotifier (this is #9763).
In the old implementation, once any request on a connection used
ClosedNotifier, the conn's underlying bufio.Reader source was switched
from the TCPConn to the read side of the pipe being fed by a
never-ending copy. Since it was impossible to abort that never-ending
copy, we could never get back to a fresh state where it was possible
to return the underlying TCPConn to callers of Hijack. Now, instead of
a never-ending Copy, the background goroutine doing a Read from the
TCPConn (or *tls.Conn) only reads a single byte. That single byte
can be in the request body, a socket timeout error, io.EOF error, or
the first byte of the second body. In any case, the new *connReader
type stitches sync and async reads together like an io.MultiReader. To
clarify the flow of Read data and combat the complexity of too many
wrapper Reader types, the *connReader absorbs the io.LimitReader
previously used for bounding request header reads. The
liveSwitchReader type is removed. (an unused switchWriter type is also
removed)
Many fields on *conn are also documented more fully.
Fixes#9763 (CloseNotify + Hijack together)
Fixes#13165 (deadlock with CloseNotify + pipelined requests)
Change-Id: I40abc0a1992d05b294d627d1838c33cbccb9dd65
Reviewed-on: https://go-review.googlesource.com/17750
Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
New implementation of TimeoutHandler: buffer everything to memory.
All or nothing: either the handler finishes completely within the
timeout (in which case the wrapper writes it all), or it misses the
timeout and none of it gets written, in which case handler wrapper can
reliably print the error response without fear that some of the
wrapped Handler's code already wrote to the output.
Now the goroutine running the wrapped Handler has its own write buffer
and Header copy.
Document the limitations.
Fixes#9162
Change-Id: Ia058c1d62cefd11843e7a2fc1ae1609d75de2441
Reviewed-on: https://go-review.googlesource.com/17752
Reviewed-by: David Symonds <dsymonds@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
The Server's server goroutine was panicing (but recovering) when
cleaning up after handling a request. It was pretty harmless (it just
closed that one connection and didn't kill the whole process) but it
was distracting.
Updates #13135
Change-Id: I2a0ce9e8b52c8d364e3f4ce245e05c6f8d62df14
Reviewed-on: https://go-review.googlesource.com/16572
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Andrew Gerrand <adg@golang.org>
Errors with http.Redirect and http.StatusOk seem
to occur from time to time on the irc channel.
This change adds documentation suggesting
to use one of the 3xx codes and not StatusOk
with Redirect.
Change-Id: I6b900a8eb868265fbbb846ee6a53e426d90a727d
Reviewed-on: https://go-review.googlesource.com/15980
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
In https://golang.org/cl/15860 http2.ConfigureServer was changed to
return an error if explicit CipherSuites are listed and they're not
compliant with the HTTP/2 spec.
This is the net/http side of the change, to look at the return value
from ConfigureServer and propagate it in Server.Serve.
h2_bundle.go will be updated in a future CL. There are too many other
http2 changes pending to be worth updating it now. Instead,
h2_bundle.go is minimally updated by hand in this CL so at least the
net/http change will compile.
Updates #12895
Change-Id: I4df7a097faff2d235742c2d310c333bd3fd5c08e
Reviewed-on: https://go-review.googlesource.com/16065
Reviewed-by: Andrew Gerrand <adg@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This enables HTTP/2 by default (for https only) if the user didn't
configure anything in their NPN/ALPN map. If they're using SPDY or an
alternate http2 or a newer http2 from x/net/http2, we do nothing
and don't use the standard library's vendored copy of x/net/http2.
Upstream remains golang.org/x/net/http2.
Update #6891
Change-Id: I69a8957a021a00ac353f9d7fdb9a40a5b69f2199
Reviewed-on: https://go-review.googlesource.com/15828
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Andrew Gerrand <adg@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
The PROXY protocol is supported by several proxy servers such as haproxy
and Amazon ELB. This protocol allows services running behind a proxy to
learn the remote address of the actual client connecting to the proxy,
by including a single textual line at the beginning of the TCP
connection.
http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt
There are several Go libraries for this protocol (such as
https://github.com/armon/go-proxyproto), which operate by wrapping a
net.Conn with an implementation whose RemoteAddr method reads the
protocol line before returning. This means that RemoteAddr is a blocking
call.
Before this change, http.Serve called RemoteAddr from the main Accepting
goroutine, not from the per-connection goroutine. This meant that it
would not Accept another connection until RemoteAddr returned, which is
not appropriate if RemoteAddr needs to do a blocking read from the
socket first.
Fixes#12943.
Change-Id: I1a242169e6e4aafd118b794e7c8ac45d0d573421
Reviewed-on: https://go-review.googlesource.com/15835
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
The existing serve() method returns a zero-length response body when
it encounters an error, which results in a blank page and no visible
error in browsers.
This change sends a response body explaining the error for display in browsers.
Fixes#12745
Change-Id: I9dc3b95ad88cb92c18ced51f6b52bd3b2c1b974c
Reviewed-on: https://go-review.googlesource.com/15018
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Optimize two calls of io.Copy which cannot make use of neither
io.ReaderFrom nor io.WriterTo optimization tricks by replacing them with
io.CopyBuffer with reusable buffers.
First is fallback call to io.Copy when server misses the optimized case
of using sendfile to copy from a regular file to net.TCPConn; second is
use of io.Copy on piped reader/writer when handler implementation uses
http.CloseNotifier interface. One of the notable users of
http.CloseNotifier is httputil.ReverseProxy.
benchmark old ns/op new ns/op delta
BenchmarkCloseNotifier-4 309591 303388 -2.00%
benchmark old allocs new allocs delta
BenchmarkCloseNotifier-4 50 49 -2.00%
benchmark old bytes new bytes delta
BenchmarkCloseNotifier-4 36168 3140 -91.32%
Fixes#12455
Change-Id: I512e6aa2f1aeed2ed00246afb3350c819b65b87e
Reviewed-on: https://go-review.googlesource.com/14177
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Found in a Google program running under the race detector.
No test, but verified that this fixes the race with go run -race of:
package main
import (
"crypto/tls"
"fmt"
"net"
"net/http"
"net/http/httptest"
)
func main() {
for {
ts := httptest.NewTLSServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {}))
conf := &tls.Config{} // non-nil
a, b := net.Pipe()
go func() {
sconn := tls.Server(a, conf)
sconn.Handshake()
}()
tr := &http.Transport{
TLSClientConfig: conf,
}
req, _ := http.NewRequest("GET", ts.URL, nil)
_, err := tr.RoundTrip(req)
println(fmt.Sprint(err))
a.Close()
b.Close()
ts.Close()
}
}
Also modified cmd/vet to report the copy-of-mutex bug statically
in CL 13646, and fixed two other instances in the code found by vet.
But vet could not have told us about cloneTLSConfig vs cloneTLSClientConfig.
Confirmed that original report is also fixed by this.
Fixes#12099.
Change-Id: Iba0171549e01852a5ec3438c25a1951c98524dec
Reviewed-on: https://go-review.googlesource.com/13453
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Run-TryBot: Russ Cox <rsc@golang.org>
Introduced in https://go-review.googlesource.com/12865 (git rev c2db5f4c).
This fix doesn't add any new lock acquistions: it just moves the
existing one taken by the unreadDataSize method and moves it out
wider.
It became flaky at rev c2db5f4c, but now reliably passes again:
$ go test -v -race -run=TestTransportAndServerSharedBodyRace -count=100 net/http
Fixes#11985
Change-Id: I6956d62839fd7c37e2f7441b1d425793f4a0db30
Reviewed-on: https://go-review.googlesource.com/12909
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
HTTP servers attempt to entirely consume a request body before sending a
response. However, when doing so, it previously would ignore any errors
encountered.
Unfortunately, the errors triggered at this stage are indicative of at
least a couple problems: read timeouts and chunked encoding errors.
This means properly crafted and/or timed requests could lead to a
"smuggled" request.
The fix is to inspect the errors created by the response body Reader,
and treat anything other than io.EOF or ErrBodyReadAfterClose as
fatal to the connection.
Fixes#11930
Change-Id: I0bf18006d7d8f6537529823fc450f2e2bdb7c18e
Reviewed-on: https://go-review.googlesource.com/12865
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This makes the receiver name consistent with the rest of the methods on
type Server.
Change-Id: Ic2a007d3b5eb50bd87030e15405e9856109cf590
Reviewed-on: https://go-review.googlesource.com/13035
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
From https://github.com/golang/go/issues/11745#issuecomment-123555313
this implements option (b), having the server pause slightly after
sending the final response on a TCP connection when we're about to close
it when we know there's a request body outstanding. This biases the
client (which might not be Go) to prefer our response header over the
request body write error.
Updates #11745
Change-Id: I07cb0b74519d266c8049d9e0eb23a61304eedbf8
Reviewed-on: https://go-review.googlesource.com/12658
Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
If we receive an HTTP request with "Expect: 100-continue" and the
Handler never read to EOF, the conn is in an unknown state.
Don't reuse that connection.
Fixes#11549
Change-Id: I5be93e7a54e899d615b05f72bdcf12b25304bc60
Reviewed-on: https://go-review.googlesource.com/12262
Reviewed-by: Andrew Gerrand <adg@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
The one in misc/makerelease/makerelease.go is particularly bad and
probably warrants rotating our keys.
I didn't update old weekly notes, and reverted some changes involving
test code for now, since we're late in the Go 1.5 freeze. Otherwise,
the rest are all auto-generated changes, and all manually reviewed.
Change-Id: Ia2753576ab5d64826a167d259f48a2f50508792d
Reviewed-on: https://go-review.googlesource.com/12048
Reviewed-by: Rob Pike <r@golang.org>
Some old buggy browsers sent extra CRLF(s) after POST bodies. Skip
over them before reading subsequent requests.
Fixes#10876
Change-Id: I62eacf2b3e985caffa85aee3de39d8cd3548130b
Reviewed-on: https://go-review.googlesource.com/11491
Reviewed-by: Andrew Gerrand <adg@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
If a client sent a POST with a huge request body, calling
req.Body.Close in the handler (which is implicit at the end of a
request) would end up consuming it all.
Put a cap on that, using the same threshold used elsewhere for similar
cases.
Fixes#9662
Change-Id: I26628413aa5f623a96ef7c2609a8d03c746669e5
Reviewed-on: https://go-review.googlesource.com/11412
Reviewed-by: Andrew Gerrand <adg@golang.org>
The ListenAndServerTLS function still requires the certFile and
keyFile, but the Server.ListenAndServerTLS method doesn't need to
require the certFile and keyFile if the Server.TLSConfig.Certificates
are already populated.
Fixes#8599
Change-Id: Id2e3433732f93e2619bfd78891f775d89f1d651e
Reviewed-on: https://go-review.googlesource.com/11413
Reviewed-by: Andrew Gerrand <adg@golang.org>
The Error function is a potential XSS vector if a user can control the
error message.
For example, an http.FileServer when given a request for this path
/<script>alert("xss!")</script>
may return a response with a body like this
open <script>alert("xss!")</script>: no such file or directory
Browsers that sniff the content may interpret this as HTML and execute
the script. The nosniff header added by this CL should help, but we
should also try santizing the output entirely.
Change-Id: I447f701531329a2fc8ffee2df2f8fa69d546f893
Reviewed-on: https://go-review.googlesource.com/10640
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
We already had client support for trailers, but no way for a server to
set them short of hijacking the connection.
Fixes#7759
Change-Id: Ic83976437739ec6c1acad5f209ed45e501dbb93a
Reviewed-on: https://go-review.googlesource.com/2157
Reviewed-by: Andrew Gerrand <adg@golang.org>
