If the current layer has a file descriptor, ClamAV is passing the path
for that file to the UnRAR module, even if the RAR we want to scan is
just some small embedded bit (e.g. detected by RARSFX signature).
We need to drop the RAR portion to a new file for the UnRAR module
because it does not accept file buffers to be scanned, only file paths.
CLAM-2900
Note this commit also changes `scanners.c` to use `access()` on Windows
instead of `_access_s()`. ClamAV defines `access()` to map to a custom
`access_w32()` function on Windows. We already use it everywhere else.
Large range testing identified some files where image fuzzy hashing
produces different hashes with ClamAV 1.5 vs 1.4.
With my investigation, I found the issue is with changes in Rust library
dependencies, though it actually wasn't any change with the 'image' or
'jpeg-decoder' crates. After running a simple `cargo update` to update
all non-pinned versions.
I confirmed that this does not affect the minimum supported Rust version
(MSRV).
CLAM-2899
I am seeing missed detections since we changed to prohibit embedded
file type identification when inside an embedded file.
In particular, I'm seeing this issue with PE files that contain multiple
other MSEXE as well as a variety of false positives for PE file headers.
For example, imagine a PE with four concatenated DLL's, like so:
```
[ EXE file | DLL #1 | DLL #2 | DLL #3 | DLL #4 ]
```
And note that false positives for embedded MSEXE files are fairly common.
So there may be a few mixed in there.
Before limiting embedded file identification we might interpret the file
structure something like this:
```
MSEXE: {
embedded MSEXE #1: false positive,
embedded MSEXE #2: false positive,
embedded MSEXE #3: false positive,
embedded MSEXE #4: DLL #1: {
embedded MSEXE #1: false positive,
embedded MSEXE #2: DLL #2: {
embedded MSEXE #1: DLL #3: {
embedded MSEXE #1: false positive,
embedded MSEXE #2: false positive,
embedded MSEXE #3: false positive,
embedded MSEXE #4: false positive,
embedded MSEXE #5: DLL #4
}
embedded MSEXE #2: false positive,
embedded MSEXE #3: false positive,
embedded MSEXE #4: false positive,
embedded MSEXE #5: false positive,
embedded MSEXE #6: DLL #4
}
embedded MSEXE #3: DLL #3,
embedded MSEXE #4: false positive,
embedded MSEXE #5: false positive,
embedded MSEXE #6: false positive,
embedded MSEXE #7: false positive,
embedded MSEXE #8: DLL #4
}
}
```
This is obviously terrible, which is why why we don't allow detecting
embedded files within other embedded files.
So after we enforce that limit, the same file may be interpreted like
this instead:
```
MSEXE: {
embedded MSEXE #1: false positive,
embedded MSEXE #2: false positive,
embedded MSEXE #3: false positive,
embedded MSEXE #4: DLL #1,
embedded MSEXE #5: false positive,
embedded MSEXE #6: DLL #2,
embedded MSEXE #7: DLL #3,
embedded MSEXE #8: false positive,
embedded MSEXE #9: false positive,
embedded MSEXE #10: false positive,
embedded MSEXE #11: false positive,
embedded MSEXE #12: DLL #4
}
```
That's great! Except that we now exceed the "MAX_EMBEDDED_OBJ" limit
for embedded type matches (limit 10, but 12 found). That means we won't
see or extract the 4th DLL anymore.
My solution is to lift the limit when adding an matched MSEXE type.
We already do this for matched ZIPSFX types.
While doing this, I've significantly tidied up the limits checks to
make it more readble, and removed duplicate checks from within the
`ac_addtype()` function.
CLAM-2897
In regression testing against a large sample set, I found that strictly
disallowing any embedded file identification if any previous layer was
an embedded file resulted in missed detections.
Specifically, I found an MSEXE file which has an embedded RAR, which in
turn had another MSEXE that itself had an embedded 7ZIP containing... malware.
sha256: c3cf573fd3d1568348506bf6dd6152d99368a7dc19037d135d5903bc1958ea85
To make it so ClamAV can extract all that, we must loosen the
restriction and allow prior layers to be embedded, just not the current
layer.
I've also added some logic to prevent attempting to extract an object at
the same offset twice. The `fpt->offset`s appear in order, but if you
have multiple file type magic signatures match on the same address, like
maybe you accidentally load two different .ftm files, then you'd get
duplicates and double-extraction.
As a bonus, I found I could also skip over offsets within a valid ZIP,
ARJ, and CAB since we now have the length of those from the header check
and as I know we don't want to extract embedded contents in that way.
Previously for documents containing VBA projects, the VBA was treated
as an object within the document and not as a normalized version of
the document. I apparently switched it say that the VBA is a normalized
version of the document. This kind of makes sense in that presently
Javascript extracted from HTML is treated as a normalized version of the
HTML. But it probably shouldn't.
Normalized layers are treated as the same file as the parent.
So now those older signatures that match on VBA projects using
"Container:CL_TYPE_MSOLE2" are failing to match.
So this commit switches it back. VBA project bits written out to a temp
file for scanning will be treated as being contained within the document.
CLAM-2896
Extracted XLM macros had the same issue.
The ZIP single record search feature is used to find specific files when
parsing OOXML documents. I observed that the core properties for a
PowerPoint file were missing in a test as compared with the previous
release.
The error handling check for the unzip search returns CL_VIRUS when
there is a match, not CL_SUCCESS!
CLAM-2886
Uncompressed ZIP-based TNEF message attachments, like OOXML office
document attachments, get double-extracted because of embedded file type
recognition.
To prevent excessive scan times, disable embedded file type recognition
for TNEF files and relay on TNEF parsing to extract attachments.
CLAM-2885
If csize (and usize) are 0, like with a directory or other empty file
entry, then the functionionality to record file record information when
indexing the central directory and each associated file record will
neglect to store the `local_header_offset` or `local_header_size`.
That causes problems later after sorting the file records and then
checking for overlapping files.
CLAM-2884
The function which indexes a ZIP central directory is not advancing
to the next central directory record thus exceeding the max-files scan
limit for many ZIPs.
CLAM-2884
Scanning CL_TYPE_MSEXE that have embedded file type signature matches
for CL_TYPE_MSEXE are incorrectly passing the PE header check for the
contained file, resulting in excessive scan times.
The problem is that the `peinfo` struct needs to have the `offset` set
for the contained `CL_TYPE_MSEXE` match prior to the header check.
Without that, the header check was actually validating the PE header of
the original file, which would always pass when that's a PE, and would
always fail if it's an OLE2 file (the other type which we check for
contained PEs).
The additional code change in this commit is to make it so the `ctx`
parameter must never be NULL, and removing the `map` parameter because,
in practice, that is always from `ctx->fmap`. This is to safeguard
against future changes to the function that may accidentally use `ctx`
without a proper NULL check.
CLAM-2882
Despite using "-fips" for `EVP_MD_fetch()` with OpenSSL 3, we are seeing
this error in some FIPS-enabled environments:
LibClamAV Error: cli_scan_fmap: Error initializing md5 hash context
The fix seems to be to create a new OpenSSL context rather than passing
NULL for the default context.
See: https://docs.openssl.org/3.0/man7/fips_module/#programmatically-loading-the-fips-module-nondefault-library-context
Special thanks to Tom Judge for identifying this fix.
CLAM-2879
If the database directory has an up-to-date .cvd (not .cld) which lacks
a .sign file, then Freshclam should try to download the .cvd.sign file.
If no .sign file is available, it will debug-log it and will not
complain loudly.
Example output:
```
❯ ./install/bin/freshclam
ClamAV update process started at Fri Oct 3 17:20:04 2025
daily.cvd database is up-to-date (version: 27780, sigs: 2076928, f-level: 90, builder: tomjudge)
Time: 0.2s, ETA: 0.0s [========================>] 8.87KiB/8.87KiB
Downloaded missing CVD .sign file daily-27780.cvd.sign
main.cvd database is up-to-date (version: 62, sigs: 6647427, f-level: 90, builder: sigmgr)
Time: 0.1s, ETA: 0.0s [========================>] 8.87KiB/8.87KiB
Downloaded missing CVD .sign file main-62.cvd.sign
bytecode.cvd database is up-to-date (version: 339, sigs: 80, f-level: 90, builder: nrandolp)
Time: 0.5s, ETA: 0.0s [========================>] 8.87KiB/8.87KiB
Downloaded missing CVD .sign file bytecode-339.cvd.sign
```
`libclamav/libmspack.c`: Initialize variables before first `goto done;`
to fix unitialized variable use in an error condition.
`libclamav/others.c`: Explicitly ignore return values for calls to add
JSON values when subsequent calls don't depend on them.
If we were to add error handling here, the only thing we'd do is debug-
log it. I don't think it's worth adding the extra lines of code.
`libclamav/unarj.c`: Removed dead code.
The `status` variable is immediately set afterwards based on whether or
not any files may be extracted.
`libclamav/unzip.c`: Removed dead code.
The `ret` variable is checked immediately after being set, above. This
check after the `do`-`while()` loop is dead code.
`sigtool/sigtool.c`: Fix potential NULL deref in error handling.
This is a fix for the same issue as was fixed in a previous commit.
I somehow overlooked this one. Copy/paste bug.
`libclamav/pdfdecode.c`: Fix leaked `stream` memory when
`filter_lzwdecode()` fails.
`clamdtop/clamdtop.c`: Fix possible NULL dereference if `strchr` returns
NULL in `read_version()` and `check_stats_available()`.
`libclamav/rtf.c`: Fix memory leak in `rtf_object_process()` if
`cli_gentemp_with_prefix()` fails.
Also change empty for-loop to resolve clang-format weirdness and make it
more obvious the for-loop has no body.
`libclamav/aspack.c`: Ensure that `endoff - old` is not negative in
`build_decrypt_array()` before passing to `CLI_ISCONTAINED()` which expects
unsigned values.
`libclamav/upx.c`: Fix integer overflow checks in multiple functions.
`libclamav/vba_extract.c`: Set `entries` pointer back to NULL after free in
`word_read_macro_entry()` error condition.
`libclamav/unzip.c`: Remove logic to return `CL_EMAXFILES` from
`index_local_file_headers()`. It seems it only overwrote the status when
not `CL_SUCCESS` in which case it could be overriding a more serious failure.
Further, updates to the how the ZIP parser works has made it so this needs
to return `CL_SUCCESS` in order for the caller to at least scan the files
found so far.
Finally, the calling function has checks of its own to make sure we don't
exceeds the max-files limit.
`libclamav/unzip.c`: Fix issue where `cli_append_potentially_unwanted()` in
`index_local_file_headers()` might overwrite an error in `status` with
`CL_CLEAN`. Instead, it now checks the return value and only overwrites the
`CL_EFORMAT` status with a different value if not `CL_SUCCESS`.
`libclamav/unzip.c`: Fix a potential leak with `combined_catalogue` and
`temp_catalogue` in an error condition. We should always free them if not NULL,
not just if the function failed. And to make this safe, we must set
`combined_catalogue` to NULL when we give ownership to `*catalogue`.
`libclamav/scanners.c`: Fix a potential leak in error handling for the
`cli_ole2_tempdir_scan_vba()` function.
CLAM-2768
`clamscan/manager.c`: Fix double-free in an error condition in `scanfile()`.
`common/optparser.c`: Fix uninitialized use of the `numarg` variable when
`arg` is `NULL`.
`libclamav/cache.c`: Don't check if `ctx-fmap` is `NULL` when we've
already dereferenced it.
`libclamav/crypto.c`: The `win_exception` variable and associated logic
is Windows-specific and so needs preprocessor platform checks. Otherwise
it generates unused variable warnings.
`libclamav/crypto.c`: Check for `size_t` overflow of the `byte_read`
variable in the `cl_hash_file_fd_ex()` function.
`libclamav/crypto.c`: Fix a memory leak in the `cl_hash_file_fd_ex()`
function.
`libclamav/fmap.c`: Correctly the `name` and `path` pointer if
`fmap_duplicate()` fails. Also need to clear those variables when
duplicating the parent `map` so that on error it does not free the wrong
`name` or `path`.
`libclamav/fmap.c`: Refine error handling for `hash_string` cleanup in
`cl_fmap_get_hash()`. Coverity's complaint was that `hash_string` could
never be non-NULL if `status` is not `CL_SUCCESS`. I.e., the cleanup is
dead code. I don't think my cleanup actually "fixes" that though it is
definitely a better way to do the error handling.
The `if (NULL != hash_string) {` check is still technically dead code.
It safeguards against future changes that may `goto done` between the
allocation and transfering ownership from `hash_string` to `hash_out`.
`libclamav/others.c`: Fix possible memory leak in `cli_recursion_stack_push()`.
`libclamav/others.c`: Refactor an if/else + switch statement inside
`cli_dispatch_scan_callback()` so that the `CL_SCAN_CALLBACK_ALERT` case
is not dead-code. It's also easier to read now.
`libclamav/pdfdecode.c`: For logging, use the `%zu` to format `size_t`
instead of casting to `long long` and using `%llu`. Simiularly use the
`STDu32` format string macro for `uint32_t`.
`libclamav/pdfdecode.c`: Fix a possible double-free for the `decoded`
pointer in `filter_lzwdecode()`.
`libclamav/pdfdecode.c`: Remove the `if (capacity > UINT_MAX) {`
overflow check inside `filter_lzwdecode()`, which didn't do anything.
The `capacity` variable this point is a fixed value and so I also changed
the `avail_out` to be that fixed `INFLATE_CHUNK_SIZE` value rather than
using `capacity`. It is more straightforward and replicates how similar
logic works later in the file.
I also removed the copy-pasted `(Bytef *)` cast which didn't reaaally do
anything, and was a copypaste from a different algorihm. The lzw
implementation interface doesn't use `Bytef`.
`libclamav/readdb.c`: Fix a possible NULL-deref on the `matcher` variable
in the error handling/cleanup code if the function fails.
`libclamav/scanners.c`: Fix an issue where the return value from some of
the parsers may be lost/overridden by the call to
`cli_dispatch_scan_callback()` just after the `done:` label in
`cli_magic_scan()`.
`libclamav/scanners.c`: Silence an unused-return value warning when
calling `cli_basename()`.
`sigtool/sigtool.c` and `unit_tests/check_regex.c`:
Fix possible NULL-derefs of the `ctx.recursion_stack` pointer in the error
handling for several functions.
Also, and this isn't a Coverity thing:
`libclamav/json_api.c` and `libclamav/others.c`:
Fix support for libjson-c version 0.13 and older.
I don't think we *should* be using the old version, but some environments
such as the current OSS-Fuzz base image are older and still use it.
The issue is that `json_object_new_uint64()` was introduced in a later
libjson-c version, so we have to fallback to use `json_object_new_int64()`
with older libjson-c, provided the int were storing isn't too big.
CLAM-2768
This commit removes the XZ.DicSizeLimit heuristic. The heuristic was
triggered whenever CLI_MAX_ALLOCATION was reached or an OOM event
happened during one of the allocation wrappers. Due to the heuristic not
being indicative of the actual event(s) that lead to its triggering, and
that we have not had any valid hits for this heuristic, we are opting to
remove it.
CLAM-2834
Compiler optimization results in invalid memory access on some
systems with the PDF `pdfname_action` pointer dereference.
This fix changes the logic so that rather than assign the pointer to
the struct containing the callback, the string that would result in
the pointer assignment later on is changed to result in the same
assignment. This fixes the issue on all tested platforms.
Resolves: https://github.com/Cisco-Talos/clamav/issues/1566
CLAM-2859
When embedded file type recognition finds a possible embedded file, it
is being scanned as a new embedded file even if it turns out it was a
false positive and parsing fails. My solution is to pre-parse the file
headers as little possible to determine if it is valid. If possible,
also determine the file size based on the headers. That will make it so
we don't have to scan additional data when the embedded file is not at
the very end.
This commit adds header checks prior to embedded ZIP, ARJ, and CAB
scanning. For these types I was also able to use the header checks to
determine the object size so as to prevent excessive pattern matching.
TODO: Add the same for RAR, EGG, 7Z, NULSFT, AUTOIT, IShield, and PDF.
This commit also removes duplicate matching for embedded MSEXE.
The embedded MSEXE detection and scanning logic was accidentally
creating an extra duplicate layer in between scanning and detection
because of the logic within the `cli_scanembpe()` function.
That function was effectively doing the header check which this commit
adds for ZIP, ARJ, and CAB but minus the size check.
Note: It is unfortunately not possible to get an accurage size from PE
file headers.
The `cli_scanembpe()` function also used to dump to a temp file for no
reason since FMAPs were extended to support windows into other FMAPs.
So this commit removes the intermediate layer as well as dropping a temp
file for each embedded PE file.
Further, this commit adds configuration and DCONF safeguards around all
embedded file type scanning.
Finally, this commit adds a set of tests to validate proper extraction
of embedded ZIP, ARJ, CAB, and MSEXE files.
CLAM-2862
Co-authored-by: TheRaynMan <draynor@sourcefire.com>
Fix a possible memory leak in the overlapping files detecting logic.
The issue is because cleanup for the zip catalogue allocated by this
function only happens if the status is no CL_SUCCESS.
My fix uses a better pattern to ensure we don't override a format error
with a "clean" result when adding the heuristic alert.
Fixes: https://issues.oss-fuzz.com/issues/376723678
CLAM-2857
ClamD opens at least one socket that is then passed to server-th as
newly allocated memory. server-th then appends to this structure with
additional FDs as it handles connections. While cleaning up during
server shutdown, server-th loops through all FDs and closes them,
followed by clamd closing the FDs it opened, which have now been
previously closed by server-th.
This fix skips closing the FDs in server-th that were opened in clamd.
CLAM-2850
Fix valgrind issues regarding:
- Unclosed log file descriptor in libclamav unit test program.
Also need to disable debug logging for `iconv_cache_destroy()` for this
or else it will try to use that file descriptor after `main()` exits.
- Unclosed socket file descriptor in ClamDScan when doing `ping()`
function.
CLAM-2872
Set a C_GNU_HURD CMake variable for Hurd, matching the existing #define
in clamav-config.h; use it to set _GNU_SOURCE, which is correct for GNU
systems.
If the application never calls `cl_load()`, then the clean-cache is
never initialized. That is a legitimate mode to run in when perhaps we
just want to extract stuff, record metadata, or for fuzzing.
This commit adds a check if the ctx->engine->cache is NULL. If it is,
then we treat it as though caching is disabled.
CLAM-2856
I found that on macOS, setting the libncurses.a and libtinfo.a into the
same CURSES_LIBRARY variable does not find or link with libtinfo.a.
To fix this, this commit adds a separate TINFO_LIBRARY variable.
In the end, CURSES_LIBRARIES and the Curses::curses CMake TARGET will
still have both libraries set, if both are provided.
This fix is necessary if the ncurses was built with `--with-terminfo`.
I think we got away without it on Linux because of pkg-config.
I also found that Apple's ncurses is prioritized by PkgConfig over one
specified by using variables. To this end, we'll skip PkgConfig if
the include path was provided.
It's possible that the `token->content` variable may get freed and set
to an uninitialized value from the `decoded` variable.
This results in both
"Conditional jump or move depends on uninitialised value"
and
"Invalid free() / delete / delete[] / realloc()"
This bug appears to have been introduced during 1.5 development and does
not occur in prior versions.
To fix it, I'm initializing the `decoded` variable and adding some
callocs elsewhere to initialize a couple other things that looked iffy,
and I'm making it so it won't try to `free(token->content)` and use
`decoded` if decompression results in an empty buffer and the status
code is set to CL_BREAK.
Fixes: https://issues.oss-fuzz.com/issues/429489013
CLAM-2854
Sigtool crashes when you use the `--html-normalise` option, every time.
Simple double free bug only affecting that specific sigtool command.
Does not affect clamscan scans (thank goodness).
Fixes: https://github.com/Cisco-Talos/clamav/issues/1483
CLAM-2835
- md5 -> sha2-256 caching
- remove reliance on md5 hashes in general
- FIPS cryptographic hash limits feature to disable md5 and sha1.
- Adds related option for ClamD, ClamScan, Freshclam, Sigtool
- ClamBC: fix crashes
- signature names that start with "Weak." won't alert anymore.
- ClamScan:
- add `--hash-hint`, `--log-hash`, `--hash-alg`, `--file-type-hint`, `--log-file-type`
- accurate counts for scanned bytes and bytes read
- libclamav:
- new cl_scan*_ex() APIs
- separate temp-directory-recursion feature from keep-temps feature
- object id's for each layer scanned
- scan hash hints
- scan file type hints
- fix double-extraction for OOXML documents
- new scan callbacks, deprecate old scan callbacks
- new APIs to get access to file data and metadata from scan callbacks
- metadata.json:
- object id's
- replace "viruses" to "alerts" and add "indicators" array
- replace "FileMD5" with "sha2-256"
- json store extra hashes feature
- Related options for ClamD and ClamScan
- object id's
- related improvements
For OpenSSL 1, `EVP_get_digestbyname()` will fail with "sha2-*" algorithm names.
Must use "sha256", etc.
I made a shim that does the conversion, and I made an improvement to ignore case
when converting alg names to our hash type enumeration.
Other fixes for a few warnings.
We presently record Alerts as an array of signature names.
Instead, it should be an object with properties of its own.
We should record alerting indicators and weak indicators in a single
"Indicators", likely with the same structure as the "Alerts" objects.
When an alerting indicator is ignored (e.g. ignored by callback or if
the file is trusted by an FP signature), we can remove it from the
"Alerts" array, and for the "Indicators" array, add a "Ignored" key with
a string value that explains why it was ignored.
This eliminates the need to track and propagate the additional
"WeakIndicators" and "IgnoredAlerts" arrays.
The current name "Viruses" is incorrect both because not all malware
are viruses, but also because ClamAV may be used to classify other
data, not just to search for malware indicators.
Renaming to "Alerts" is more consistent with other language such as
the options "--alert-exceeds-max", etc.
It will match the new "IgnoredAlerts", "ContainedAlerts", and
"IgnoredContainedAlerts" JSON key names.
If the outermost layer is trusted (e.g. using an FP signature), the verdict passed
back by the `cl_scan*_ex()` functions should be CL_VERDICT_TRUSTED.
To ensure this, and other correct verdicts, I moved the logic setting the verdict
to occur when adding indicators, or trusting a given layer. Then at the end of a
scan, it will set the output verdict parameter to the top level verdict.
This commit also:
* Fixes a bug in the `ex_scan_callbacks` program where a crash would happen when
a hash was retrieved for an inner layer, but isn't for the container.
* Added debug logs whenever a hash is calculated or set, printing the hash type
and hash string.
* When a layer is trusted, in addition to removing evidence for that layer, it
will also index the metadata JSON (if that feature is enabled) and will rename
any "Viruses" to "IgnoredAlerts", and rename "ContainedIndicators" to
"IgnoredContainedIndicators".
* Fixed an issue where setting the hash algorithm with extra characters, like
setting to "sha256789" would ignore the extra characters, and report the hash
type as the same. It will now fail if the string length differs from the known
hash algorithm.
Both enum variants are 0, so it's a no-op.
`cl_error_t` should be used to determine if we keep going or stop, whether
that's because there was a detection and we're not in allmatch mode, or if
because of an error.
`cl_verdict_t` should be used to determine or say the verdict (clean is a
verdict, though I feel 'nothing found' is more accurate).
