Commit graph

42146 commits

Author SHA1 Message Date
Michael Niedermayer
017884bdc3 avcodec/motionpixels: Mark 2 functions as always_inline
Fixes: Timeout (30sec -> 25sec)
Fixes: 17050/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MOTIONPIXELS_fuzzer-5719149803732992

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-28 18:36:12 +02:00
Michael Niedermayer
8335ba8ae9 avcodec/ituh263dec: Make the condition for the studio slice start code match between ff_h263_resync() and ff_mpeg4_decode_studio_slice_header()
If they mismatch an infinite loop can occur
Fixes: Timeout (infinite loop)
Fixes: 17043/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MPEG4_fuzzer-5695051748868096

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-28 18:35:27 +02:00
Michael Niedermayer
fbb314b6f2 avcodec/ralf: Fix integer overflow in decode_channel()
Fixes: signed integer overflow: -1094995519 * 64 cannot be represented in type 'int'
Fixes: 17030/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_RALF_fuzzer-5640695838146560

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-28 18:35:27 +02:00
Michael Niedermayer
e75e7fe160 vcodec/vc1: compute rangex/y only for P/B frames
Fixes: left shift of 1073741824 by 1 places cannot be represented in type 'int'
Fixes: 16976/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VC1_fuzzer-4847262047404032

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-28 18:34:53 +02:00
Michael Niedermayer
ced9a1cd0a avcodec/vc1_pred: Fix invalid shifts in scaleforopp()
Fixes: left shift of negative value -2
Fixes: 16964/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VC1IMAGE_fuzzer-5757853565976576

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-28 18:34:53 +02:00
Michael Niedermayer
c722a69253 avcodec/vc1_block: Fix invalid shift with rangeredfrm
Fixes: left shift of negative value -7
Fixes: 16959/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMV3_fuzzer-5200360825683968

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-28 18:34:52 +02:00
Michael Niedermayer
181e138da7 avcodec/vc1: Check for excessive resolution
Fixes: overflow in aspect ratio calculation
Fixes: signed integer overflow: 393215 * 14594 cannot be represented in type 'int'
Fixes: 15728/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMV3IMAGE_fuzzer-5661588893204480

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-28 18:34:24 +02:00
Michael Niedermayer
7f7af9e294 avcodec/vc1: check REFDIST
"9.1.1.43 P Reference Distance (REFDIST)"
"The value of REFDIST shall be less than, or equal to, 16."

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-28 18:33:20 +02:00
Michael Niedermayer
cf41da2ad2 avcodec/apedec: Allocate decoded_buffer after successful ff_get_buffer()
We need to reset samples in this case to avoid being stuck with incorrect
samples value.

Fixes: OOM
Fixes: Timeout
Fixes: 16627/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5638059583864832
Fixes: 17089/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5672188463546368

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-28 18:25:20 +02:00
Michael Niedermayer
1e95a3e8a7 avcodec/apedec: Fix several integer overflows in predictor_update_filter() and do_apply_filter()
Fixes: negation of -2147483648 cannot be represented in type 'int'; cast to an unsigned type to negate this value to itself
Fixes: signed integer overflow: -14527961 - 2147483425 cannot be represented in type 'int'
Fixes: 16380/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5645957131141120
Fixes: 16968/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5716169901735936
Fixes: 17074/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5198710497083392

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-28 17:49:00 +02:00
Michael Niedermayer
f63cd1963e avcodec/hevc_cabac: Tighten the limit on k in ff_hevc_cu_qp_delta_abs()
Values larger would fail subsequent tests.

Fixes: signed integer overflow: 5 + 2147483646 cannot be represented in type 'int'
Fixes: 16966/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_HEVC_fuzzer-5695709549953024

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-28 17:48:04 +02:00
Andreas Rheinhardt
5cf593adcd avcodec/jpeg2000dwt: Fix undefined shifts of negative numbers
Affected the vsynth*-jpeg2000 and the vsynth*-jpeg2000-97 FATE tests
(where * ranges over { 1, 2, 3, _lena }) as well as ticket #7983.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-28 17:24:32 +02:00
Andreas Rheinhardt
324487b596 avcodec/ituh263dec: Fix undefined left shift of negative number
Fixes ticket #8160.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-28 17:24:32 +02:00
Andreas Rheinhardt
b7f156e8cb avcodec/dnxhdenc: Fix undefined left shifts of negative numbers
Affected 61 FATE-tests: 60 vsynth tests and lavf-mxf_opatom.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-28 17:24:32 +02:00
Andreas Rheinhardt
f7bc0386d9 avcodec/ffv1enc: Fix out-of-bounds-array access
libavcodec/ffv1enc.c accessed an array of uint8_t [32] via array[0][j]
in order to loop over all the uint8_t in this array of arrays. Of course
this implied an out-of-bounds access for array[0] and UBSan complained
about this. So replace this with nested loops; furthermore, factor this
out into a function of its own to easily break out of the nested loops.

This affected the FATE-tests vsynth1-ffv1, vsynth1-ffv1-v3-yuv420p,
vsynth1-ffv1-v3-yuv422p10, vsynth1-ffv1-v3-yuv444p16,
vsynth1-ffv1-v3-bgr0, vsynth1-ffv1-ffv1-v3-rgb48 as well as the
corresponding vsynth2-*, vsynth3-* and the vsynth_lena-* tests.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-28 17:24:32 +02:00
Timo Rothenpieler
ab0ef1abdf avcodec/nvenc: make use of new GetLastErrorString function
Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>
2019-09-27 20:00:59 +02:00
Timo Rothenpieler
51a23343d9 avcodec/nvenc: pass CUstream to nvenc when available
Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>
2019-09-27 20:00:59 +02:00
Roman Arzumanyan
567b5e33d9 avcodec/nvenc: add multiple reference frames support
Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>
2019-09-27 20:00:59 +02:00
Timo Rothenpieler
e929b2f248 avcodec/nvenc: switch to dedicated dpb_size option
Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>
2019-09-27 20:00:59 +02:00
Jun Zhao
333109f469 lavc/mpeg4audio: add chan_config check to avoid indeterminate channels
add chan_config check to avoid indeterminate channels.

Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
Signed-off-by: James Almer <jamrial@gmail.com>
2019-09-27 10:50:18 -03:00
James Almer
75c7484fcb avcodec/mpeg4audio: fix doxy for ff_mpeg4audio_get_config_gb()
Signed-off-by: James Almer <jamrial@gmail.com>
2019-09-27 10:49:29 -03:00
James Almer
d582cc17e1 avcodec: use avpriv_mpeg4audio_get_config2()
Signed-off-by: James Almer <jamrial@gmail.com>
2019-09-27 10:37:11 -03:00
James Almer
f96a8b015f avcodec/mpeg4audio: add avpriv_mpeg4audio_get_config2()
Identical to avpriv_mpeg4audio_get_config() except taking a size argument in
bytes, and featuring a new logging context paremeter.

Schedule avpriv_mpeg4audio_get_config() for removal as soon as major is bumped
as well.

Signed-off-by: James Almer <jamrial@gmail.com>
2019-09-27 10:37:11 -03:00
Michael Niedermayer
87ddf9f1ef avcodec/4xm: Check index in decode_i_block() also in the path where its not used.
Fixes: Infinite loop
Fixes: signed integer overflow: 2147483644 + 16 cannot be represented in type 'int'
Fixes: 16169/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_FOURXM_fuzzer-5662570416963584
Fixes: 16782/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_FOURXM_fuzzer-5743163859271680
Fixes: 17641/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_FOURXM_fuzzer-5711603562971136

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-26 21:02:34 +02:00
Michael Niedermayer
c5a52eb5cd avcodec/loco: Check for end of input in the first line
Fixes: Timeout (85sec -> 0.1sec)
Fixes: 17634/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_LOCO_fuzzer-5666410809786368

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-26 21:02:34 +02:00
Michael Niedermayer
2acbbe2623 avcodec/atrac3: Check block_align
Fixes: Infinite loop
Fixes: 17620/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ATRAC3_fuzzer-5086123012915200

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-26 21:02:34 +02:00
Michael Niedermayer
581a895c5c avcodec/alsdec: Avoid dereferencing context pointer in inner interleave loop
This makes the decoder faster

Improves/Fixes: Timeout (22sec -> 20sec)
Testcase: 17619/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ALS_fuzzer-5078510820917248

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-26 21:02:34 +02:00
Michael Niedermayer
b2785cd3ac avcodec/hcom: Check that there are dictionary entries
Fixes: out of array read
Fixes: 17617/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_HCOM_fuzzer-5674970478280704

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-26 21:02:34 +02:00
Michael Niedermayer
cfa1937791 avcodec/fitsdec: Prevent division by 0 with huge data_max
Fixes: division by 0
Fixes: 15657/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_FITS_fuzzer-5738154838982656

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-26 21:02:34 +02:00
Michael Niedermayer
7dc0943d4a avcodec/dstdec: Fix integer overflow in samples_per_frame computation
Fixes: Timeout (? -> 2ms)
Fixes: 17616/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_DST_fuzzer-5198057947267072

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-26 21:02:34 +02:00
Michael Niedermayer
972a0a818f avcodec/g729_parser: Check block_size
Fixes: Infinite loop
Fixes: 17611/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ACELP_KELVIN_fuzzer-5765134928052224

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-26 21:02:34 +02:00
Michael Niedermayer
02fb6a2147 avcodec/sbcdec: Initialize number of channels
Fixes: out of array access
Fixes: 17609/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_SBC_fuzzer-5758729319874560

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Reviewed-by: Moritz Barsnick <barsnick@gmx.net>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-26 21:02:34 +02:00
Andreas Rheinhardt
8b0f949906 avcodec/exr: Fix undefined left shifts of negative numbers
Affected the FATE-tests exr-rgb-scanline-pxr24-half-uint32-13x9 and
exr-rgb-scanline-pxr24-uint32.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-26 21:02:34 +02:00
Andriy Gelman
80e1c93c87 avcodec/hevc_ps: Remove dead code in vps_id check
Since reading 4 bits always returns a value in the range [0, 15], the
check for vps_id >= HEVC_MAX_VPS_COUNT, where HEVC_MAX_VPS_COUNT = 16, is redundant.

Signed-off-by: Andriy Gelman <andriy.gelman@gmail.com>
Signed-off-by: James Almer <jamrial@gmail.com>
2019-09-26 11:20:14 -03:00
Paul B Mahol
9b611deef1 avcodec/truespeech: fix left shift of negative value 2019-09-26 16:05:15 +02:00
Andreas Rheinhardt
093b6894bf avcodec/mpeg12dec: Sanitize start codes earlier
The MPEG-1/2 decoder uses avpriv_find_start_code to search for start
codes and worked with the resulting start code before checking that it
is really a start code of a slice. In particular, if the picture is so
big that a slice_vertical_position_extension is present, it added the
slice_vertical_position_extension as if it had a slice. Then a left
shift is performed, without making sure that the value to be shifted is
nonnegative.
Afterwards the end result is checked, but even if a start code of a
non-slice has been found, it might pass these checks: If
slice_vertical_position_extension is present a start code <
SLICE_MIN_START_CODE can lead to a macroblock-row index that appears
valid. Furthermore, the left shift might make an invalid start code
appear valid by discarding the highest bit.
This has been fixed by checking directly after avpriv_find_start_code
has returned.

Fixes ticket #8162 (which is about the undefined left shifts).

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2019-09-26 14:09:14 +02:00
Andreas Rheinhardt
5886153dc3 avcodec/pcm: Cosmetics
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2019-09-26 14:06:06 +02:00
Andreas Rheinhardt
69473bec6f avcodec/pcm: Fix undefined shifts
Fixes the acodec-pcm-u16[lb]e FATE-tests.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2019-09-26 14:02:56 +02:00
Andreas Rheinhardt
84974c6fb5 avcodec/wavpackenc: Fix undefined shifts
Fixes ticket #8161 and the acodec-wavpack FATE-test.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
2019-09-26 11:11:01 +02:00
Zhong Li
525de95679 lavc/qsv: add memory type message
Signed-off-by: Zhong Li <zhong.li@intel.com>
2019-09-26 13:44:11 +08:00
Zhong Li
74007dd86a lavc/qsv: Fix MSDK initialization failure in system memory mode
MSDK does not create internal acceleration device on Linux,
So MFXVideoCORE_SetHandle() is necessary.
It has been added for ff_qsv_init_session_device().
But missed for ff_qsv_init_internal_session() due to commit
1f26a23 overwrited commit db89f45

Fix #7030

Signed-off-by: Zhong Li <zhong.li@intel.com>
2019-09-26 13:44:02 +08:00
Carl Eugen Hoyos
dbd3dbb476 Revert "lavc/tiff: correct the default value of YCbCrSubsampling to 2x2"
This reverts commit eb5d0f18ff.

Together with 89f464e9, it breaks decoding of tiff files like fate-suite/exif/image_small.tiff
2019-09-26 00:34:44 +02:00
Limin Wang
3def984a98 avcodec/dnxhdenc: return error if av_malloc failed
Signed-off-by: Limin Wang <lance.lmwang@gmail.com>
Signed-off-by: James Almer <jamrial@gmail.com>
2019-09-25 17:56:44 -03:00
Michael Niedermayer
121bf1b3b8 avcodec/decode: Check max_samples in get_buffer_internal()
This checks max_samples at the same level as max_pixels

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-25 17:21:23 +02:00
Michael Niedermayer
f8406ab4b9 avcodec: add max_samples
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-25 17:18:37 +02:00
Pavel Skakov
eb5d0f18ff lavc/tiff: correct the default value of YCbCrSubsampling to 2x2 2019-09-25 10:47:46 +02:00
Guo, Yejun
85e338ab0d libavcodec/libx265: add a flag to output ROI warnings only once.
Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
2019-09-24 17:22:15 +01:00
Guo, Yejun
104d44138b libavcodec/libx264: add a flag to output ROI warnings only once.
Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
2019-09-24 17:22:15 +01:00
Michael Niedermayer
95e5396919 avcodec/utils: Optimize ff_color_frame() using memcpy()
4650975 -> 4493240 dezicycles

This optimizes lines 2 and later. Line 1 still uses av_memcpy_backptr()
This change originally fixed ossfuzz 10790 but this is now fixed by other
optimizations already

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-24 17:53:39 +02:00
Michael Niedermayer
3dce4d03d5 avcodec/aacdec: Check if we run out of input in read_stream_mux_config()
Fixes: Infinite loop
Fixes: 16920/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_AAC_LATM_fuzzer-5653421289373696

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2019-09-24 17:53:39 +02:00