Use gobs to serialize indexes instead of encoding/binary.
Even with gobs, serialize data in slices instead of
applying gob to the entire data structure at once,
to reduce the amount of extra buffer memory needed
inside gob.
7x faster Write/Read for new BenchmarkSaveRestore
compared to old code; possibly because encoding/binary
is more expensive for int32 slice elements (interface
call to get little/big endian encoding), while gob's
encoding is fixed (unconfirmed).
new (using gobs):
suffixarray.BenchmarkSaveRestore 1 2153604000 ns/op
old (using encoding/binary):
suffixarray.BenchmarkSaveRestore 1 15118322000 ns/op
The actual serialized data is slightly larger then using
the old code for very large indices because full 32bit indices
require 5bytes using gobs instead of 4bytes (encoding/binary)
in serialized form.
R=r
CC=golang-dev
https://golang.org/cl/5087041
This implements the algorithm qsufsort using the sort package
as a sorting primitive. Its worst-case performance is O(N*log(N)), and it
uses only an additional slice of N ints of memory during creation.
Benchmarks (seconds):
old new
10k nulls 149 0.044
1M English corpus 32.0 3.6
R=gri, gri1
CC=golang-dev
https://golang.org/cl/3752044
Implementation uses fast suffixarray lookup to find
initial matches if the regular expression starts with
a suitable prefix without meta characters.
R=r, rsc
CC=golang-dev
https://golang.org/cl/3720042
This is a replacement for pending CL 2219042. It only contains
the raw suffixarray functionality with two methods:
- New create a new index from some data
- Lookup lookup occurences of a bytes slice in the data
Any other functionality (dealing with multiple data sets and
the corresponding position lists) is generic and doesn't have
to be part of this package.
Known performance bug: This implementation works fine for data sets
up to several megabytes as long as it doesn't contain very long
contiguous sequences of equal bytes. For instance, index creation for
all .go files under GOROOT (250KLOCs, approx. 9MB) takes ~50s on
2.66 GHz Intel Xeon as long as test/fixedbugs/257.go is excluded.
With that file, index creation times takes several days. 257.go contains
a string of 1M smiley faces.
There are more sophisticated suffixarray creation algorithms which
can handle very long common prefixes. The implementation can be
updated w/o the need to change the interface.
R=rsc, r, PeterGo
CC=golang-dev
https://golang.org/cl/2265041
